Would you like an AB test framework for LangChain projects?

While creating my startup product, I had a tough time seeing how changes to prompts made a difference.

So, I’m considering starting an open-source project for A/B testing AI prompts. It’s all about making it easier to handle prompts and few-shot examples, run tests, get feedback, and see results.

But before I dive in, I’d love to know if this tool could help others too. Does it sound handy to you?

3 Likes

I think A/B testing is a great idea for prompts.

Considering it’s open source maybe you could frame it in such a way that it can be easily modified to work with Langchain?

2 Likes

Thank you for your positive feedback @RonaldGRuckus :slight_smile: Yes, right! I’m considering a web application that works with LangChain project.

did this get going ? I am trying to figure this out myself as well.

Well considering OpenAI recently and discreetly implemented the format themselves for some of GPT-4’s responses in the web UI, I’d definitely say so.