Do you use any service to help you manage your prompts? We are creating a QA service and some of us are trying different prompts and parameters to find the best option. But keeping track of changes and providing feedback etc is hard to do. I feel that a service should exist to keep track of different versions, provide feedback, and auto-experiment (ie. given a base prompt, launch it with different parameters to see how the changes affects the result - or even automatically iterate on the prompt itself)
Potentially, the service could even analyze the prompt and give suggestions on how to improve it depending on the objective.
Does something like this already exists? How do you manage this workflow?
I don’t know of anything that does that off-hand, but it sounds like a cool idea. I’m sure you might be able to use an off-the-shelf CMS like Drupal or WP even to set it up, although something custom would probably work better.
Maybe make it so people entering prompts could share with the public or keep it private…
Come to think of it, I believe there’s a prompt market or two? I’ve never browsed or bought a prompt, though. Too much fun trying to come up with them on my own.
Thanks @PaulBellow ! The “market-ish” part could also be an option, but I’m thinking more on a private use for now: help teams to iterate, refine and keep track of prompts; also help testing when a new version of the model comes out, be able to test against different APIs… On top of these several things could be built.
Your problem is actually very common and that’s why we are building a prompt management system! We are currently in the process of onboarding beta users, you can sign up to our waitlist here: https://usegenie.app/
It is currently hard to manage and provide feedback on the prompts, despite the fact that it can be crucial for the product! We’re looking to change that.
Please don’t hesitate to contact us with any questions or feedback you may have. We look forward to hearing from you!
I have designed a proprietary tool for this to track changes/make rapid changes and monitor the outputs. Our dev has built it using flutterflow & Supabase.
It works as intended to facilitate rapid testing and refinements of prompts inc a dashboard to view all outputs from the prompt chains/process pipelines.
I found this type of setup to be necessary as the output from one prompt in the pipeline determines the inputs to all of the others with cascading effects, so it’s critical to be able to view all inputs/outputs and assess the effects of updates/refinements.
It has become obvious though that it would be futile to attempt to build a generic version of our tool due to the rapid evolution of the tech - all tools are vulnerable to becoming obsolete - rapidly.
I suggest you build your own monitoring/version history dashboard as simply and quickly as possible, for your own specific use case - to enable you to do what you need to do.
You will quickly realize how fast it needs to evolve to keep up with the evolution of your project.
Hi! It’s true that I’m not active in the forum, but the topic is still relevant for me, so the message from @tomvancamping was very useful. Thanks all!
Hi Elmstedt, my understanding is that the user and other viewers are looking for tools like this as says the headline and the comments. I think my post actually fulfills a purpose here. I am also not a big corporation but a one-man show trying to get something productive off the ground. I kindly ask you to please redo your spam mark. Would you? Thanks and best, Tom
And to revive the topic, I’d add that I keep needing a tool to collaborate on the elaboration, iteration and refinement of prompts, A/B testing, sharing versions etc. I have seen that some of the tools from months ago are no longer on development.
Last week I tested promptmetheus.com which is half done, it has most of what I’d like, although the UI is a bit cumbersome in some points.
We are also evaluating https://www.promptfoo.dev/ but I’d prefer something cloud-UI based to make collaboration as easy as possible.
