Seeking Advice on Implementing Functionality for Graph Plotting in Chat Application

API
,
1

Hello everyone,

I’m currently working on a feature for a chat application and would appreciate some guidance on how to best implement it. Here’s a brief overview of the interaction flow:

  1. User Request: The user asks for records from the last three months.
  2. Assistant Response: The assistant provides the data in a text format. For example:
  • January 2023: 80%
  • April 2023: 90%
  • July 2023: 50%
  1. User Follow-Up: The user requests this data to be plotted as a graph.
  2. Expected Functionality: At this point, I will invoke a function-calling (let’s call it plotGraph(values)) that will process this data and return a graphical representation as json which I want to interpret on chat UI as chart.

The Challenge: I need a way to add a messagre as json after the call to plotGraph(values) function which I can be displayed in the chat history. The main question is:

  • Is it possible to return json from function directly into the chat flow, where the assistant’s messages are used as input for the graph plotting function?
  • If not, what would be an alternative or more efficient approach to achieve this functionality?
2

Would it make more sense to use function calling to return the values and then use code interpreter to create the plot graph or have the graphical representation sent as a hyperlink that can be formatted using your chat application?

Otherwise, if you want to work with JSON it could make more sense to just update the user’s state the same way you are managing the chat application, but instead with a separate entry that can be monitored and have the update shown on their screen. Similar to the Wanderlust demo at the presentation.

I would go with the second option. Mainly because you are limiting yourself by using code interpreter, and also by send a graphical representation.

There are a lot of powerful, lightweight, interactive graphing libraries available that you can integrate

3

Thanks @RonaldGRuckus for the reply, The reason for NOT using code interpreter is mainly because the use case is not only for graph but also for user interation as well like multiple choice questions, Do you have any suggestion on how to acheive multiple choice question during user interation? Also, if I return JSON data from the function_calling, chatgpt intepreting that and give back the text form, is there a way to override this behaviour?

Thanks in advance.

4

So, does it mean, I need to maintain separate list of messages specific to the user whoch includes all my JSON messages on the application level without relying on messageList from the Threads. Is that correct?

5

If you have an interactive graph you should build it alongside an API whichever way you determine to work best. Then separately create the Assistant with Function Calling to be able to also update the graph using the same API.

For persistence you would need to have this information saved to a user profile.

A simple database could look like:

conversations -> uid -> one -> messageObj
graphs -> uid -> one -> graphObj

With an API that the user can interact with to manipulate their respective graphs. Then finally the Assistant can access the same API.

If you can subscribe to the database then any changes would be reflected near immediately on the graph the user is looking at.

The conversations could be optional (besides the last message/run). You could just have the user save it themself to localStorage and then request it if they don’t have it. There is probably some edge cases that would have to be addressed though. (Like a user having an outdated cache that has been worked on from another device, it would be cool if we could retrieve a hash of the current thread for comparing, maybe a good idea for metadata :thinking:)

The main idea is that this graph, the questions, everything works standalone. THEN you can use the Assistant on top of it.

6

@gopichandar I am also working on similar usecase and your help would be appreciated.
Is there any possible way that i send data to openai and openai returns the graph as an image, or we would just send the data to openai get it in a particular format and then generate the graphs based on the data returned using appropriate libraries.