API and files - Assistant or classic chat completation?

Hello everyone, i have a question for you:

i need to implement chatgpt into our software, using it as an assistant who takes a request, transform it to a query (using a table schema).
I tried Assistant on the playground, giving it a tableschema file (which contains just the table data like columns and info about fields) and making the system message as detailed as possible (like "using the tableschema file, generate an sql query ecc…). It gives me a query that i execute on SQL and it is pretty good.

My problem is that costs are HUGE because of the all context.
I don’t know if Assistant is the right choice, i don’t need any context saved. I just need chatgpt to read the structure of the database e create a query, that’s all.

What do you suggest? I didn’t find anything about giving chatgpt a file through API (the only way is assistant api). Do you think the cheapest way is to use classic chat completation with a good system message? Including the table schema inside the text? Can you suggest something?

Thank you all!! :slight_smile:

1 Like

I’d avoid assistants like the plague, although I can see their value for some use cases.

How large is your schema file? Retrieval is just gonna turn it into text anyways. running your own vectordb is always an option if your schema is gigantic.

1 Like

Thanks for answering :slight_smile:
Schema is actually simple, some tables, some fields, i’m not planning to include all tables and fields but just the one i need.
How would you use a vectordb in this case?

I take it you are using Text to SQL? https://platform.openai.com/examples/default-sql-translate

I don’t understand how this could be so expensive, unless you are uploading the file(s) as knowledge.

Perhaps I’m not understanding your issue, but I thought that was what you were doing. No need to upload your data to Assistants. You only need:

  1. An API to access your data. If it is spreadsheet, for example, you could upload it to Google Sheets and use their API to access.
  2. An understanding of how to query your data through the API.
  3. A system prompt that explains how to query the API to the LLM.

With this, you could actually create a GPT that connects to your API as an Action for knowledge. So now, the LLM would simply translate the user’s query into the query language and execute that via the API.

User <------> GPT <------> Your API

This will be far, far less expensive.

That’s could be good but can custom gpt be access by API? I need it to implement into a custom software so I need everything to be doable via api. Thanks for response:)

Still confused. Why would you want to go?:

Your API -----> GPT

If you are looking to save in cost, I would think it’s more like:

GPT -----> Your API

User asks question about your data. GPT looks at your database schema and creates query language and sends that query to your API. Your API processes the request and sends back the appropriate data. GPT looks at the data and formulates a response for the user.

All this without having to upload your entire database to the GPT.

What more are you looking for?

personally, I would spin up a milvus container if necessary. or perhaps a python-faiss-fastapi container if it’s not gonna change much.

store embeddings of semantic descriptions of the tables, and return the tables.

But that would probably only make sense if you have high hundreds or thousands of fields…

That’s right, but I can’t have users to use GPT, I need integrated with my UI/Web app, that’s the problem.

Ok got it.
I have not so many tables/fields… so do you think that having it in the prompt is a good idea? Or is there a method to use files? I can’t think of any.

It really depends on how many tables!

Just try to put it into the prompt. A couple of cents extra spent on prompts is probably gonna be cheaper than dozens of hours setting up and maintaining all the ancillary crap.

And even if you do decide that it’s too expensive, you can always opt for that later.

How would you organise the prompt?
I’m sorry for all that question, I want to learn as much as possible, I tried with a speech-style prompt but it seems so long. It is normal to have a long prompt?

oh yeah.

having a structured prompt (like markdown readme/doc for example) seems to work decently well for simple cases.

Do you use function calls at all at the moment? Leaving costs and any other hard feelings about Assistants aside for a moment, unless I am getting this wrong you can achieve this with a clearly defined function call (incl. a detailed function description that sets out your SQL request parameters) plus sufficiently clear and detailed instructions so the Assistant understands how to turn user messages into SQL queries which are then executed in the background via an API connection to your database.

If you are looking for exact data points from your SQL table(s), then I would stick to using SQL queries.