Creating chat assistant for client, not sure what way is the best way to go about it

API
1

So I’m making a chat assistant for a client where end users can ask about thousands - millions of products. The data I’m sourcing is from an external api returned as JSON.

The two options I’ve kinda come up with are as follows:

  1. Use assistants API and function calling.

  • For example if the user asks about a certain category of products, then a function would be called to retrieve the data relevant to the requested category. Then the data would be added as a file to open ai, then attached to the assistant for retrieval.

  • For seen problems: cleaning up of unused files and token count.

  1. Using embeddings to do search and retrieve information.
  • Essentially I would load all the data onto a vector db and retrieve it as needed based off of user prompts.

-For seen problems: from what I’ve read, semantic searches aren’t great with JSON data, so I’m worried about accuracy with it.

I’m leaning towards option one and that’s probably what I’ll do if I don’t get a response back haha.

But anyways, what would you recommend in this situation. Or if theres a better way of going about it that I haven’t even thought about, pleas let me know! Thanks.

2

Welcome to the community!

So you’re the expert, the professional here! :laughing:

If you’re new to this space, I’d just recommend this when designing your system:

Think about how an expert would think about this problem: If you put a human in this role - (maybe yourself) - how would that human answer user queries?

After you’ve handled about 100 queries, reflect:

  1. did you memorize all millions of products? probably not.
  2. what did you remember? what sprung to mind on what query, and why?
  3. how did you find the correct products? what tools did you use? don’t forget: if you open a file and use ctrl+f, that’s a tool use. the small stuff might be relevant.
  4. how did you go about constructing your final answer? what did you consider?

So, think about the thought process:

  • for stuff you remembered, that belongs in an embedding DB.
  • for how tools are used, unless they’re context specific, that goes in the prompt.
  • for tools you used, maybe a SQL lookup, that goes into tools.
  • for tone and style, you might use fine tunes.

Good luck! :slight_smile:

1 Like