Question about long operation

GPT builders Plugins / Actions builders
1

Hello ChatGPT plugin developers,

I’m currently working on implementing a ChatGPT plugin that interacts with an external service. However, the endpoint I’m using takes on average 1-2 minutes to process a request and initially returns a queueID instead of the final response. The final response can be retrieved later using this queueID.

I’m wondering if this use case can still be accommodated within the ChatGPT plugin structure. If so, how can I handle this delayed response and queueID mechanism in my plugin implementation without redirecting the user to external page? Is there a way to poll for results until they are available or handle this asynchronous behavior within the plugin?

Any guidance or suggestions would be greatly appreciated. Thank you in advance!

2

I figured it out. Here’s the plugin if anyone wants to try: https://twitter.com/AbhimanyuGrove5/status/1646144373317353475

1 Like
3

Hi there, I understand that you’ve figured out how to handle the delayed response and queueID mechanism in your ChatGPT plugin. That’s great news! Could you please share some details on how you implemented it? I noticed that you’ve shared a link to your plugin on Twitter, but it doesn’t provide any information on the implementation. Thank you!

4

Hi Dariel, All I did was convert my long operation to queue-based endpoint, and added an end-point to check the queue status. ChatGPT figured out the rest by itself, here’s my openapi.yaml file

openapi: 3.0.1
info:
  title: Summarizer Plugin
  description: A plugin that allows the user to summarize artiles, podcasts, YouTube videos and pdfs using ChatGPT. If you do not know the user's inBrief api key, ask them first before making queries to the plugin. Otherwise, use the api key "trial".
  version: 'v1'
servers:
  - url: https://chatgpt-plugin.inbrief.ai
paths:
  /api/get-queue-status:
    get:
      operationId: getQueueStatus
      summary: Get the status and summary of a summarization task
      description: This endpoint retrieves the status and summary (if the task is finished) of a summarization task based on the provided queue ID.
      parameters:
        - in: query
          name: queueID
          description: The queue ID of the summarization task
          required: true
          schema:
            type: string
      responses:
        '200':
          description: Successfully retrieved the status and summary of the summarization task
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/summarizeResponse'
        '400':
          description: Bad request (e.g., missing or invalid queue ID)
        '404':
          description: Queue ID not found
        '500':
          description: Internal server error
      
  /api/summarize-queue:
    post:
      operationId: summarize
      summary: Queues a new summarization task for a given URL and provides a queue ID
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/summarizeRequest'
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/queue'

components:
  schemas:
    summarizeResponse:
      type: object
      properties:     
        status:
          type: string
          description: Status of the summary request, 'in-progress', 'failed' or 'done'
          required: true  
        title:
          type: string
          description: Title of the document summarized
          required: true
          nullable: true          
        summary:
          type: string
          description: The AI-generated summary of the URL
          required: true
          nullable: true
        systemMessage:
          type: string
          description: A message from the system that is not a summary
          required: false
          nullable: true

    queue:
      type: object
      properties:     
        status:
          type: string
          description: Status of the summary request, 'in-progress', 'failed' or 'queued'
          required: true  
        queueID:
          type: string
          description: Title of the document summarized
          required: true
          nullable: true          
        systemMessage:
          type: string
          description: A message from the system that is not a summary
          required: false
          nullable: true                            
    summarizeRequest:
      type: object
      required:
      - url
      properties:
        url:
          type: string
          description: The URL of article / pdf / YouTube video you want to summarize
          required: true
    queueRequest:
      type: object
      required:
      - queueID
      properties:
        queueID:
          type: string
          description: The queue ID of the summarization task
          required: true
7 Likes
5

A pretty smart technique. Thanks.

I went to your site to try it, but didn’t proceed. I am not so sure of the registration process. The OAuth provider seems unfamiliar. What provider does your site use?

1 Like
6

Thank you! We use ‘auth0’ running on a custom domain.

7

Nice! it would be super useful from ux point of view if OpenAI team can give callback functionality for plugin backend to communicate back once long running task is over.

cc: @logankilpatrick tagging you for visibility, thanks.

8

I witnessed the automatic polling behavior once. I try to do it again today, but ChatGPT won’t check the status periodically anymore until you prompt it to do so. Does it still work for you?

9

Here’s the pattern in case it wasn’t clear from openapi.yaml:

Endpoint 1 → Returns Queue object

Endpoint 2 → Accept queueID, returns actual response object

ChatGPT would call it sequentially without any special instruction if you’ve done everything right.

10

Your openapi.yaml is crystal clear. In fact, it worked a few days ago when I tested it. The question: is it still working for you?

It doesn’t work for me now. I think ChatGPT has changed (cancelled) this nice feature.

11

It worked, but not as before. Earlier, for a long operation, it would recursively poll the queue status endpoint every few seconds, but now I have to ask it explicitly, which is terrible UX.

Summarizing-Article-Progress
Summarizing-Article-Progress1806×1212 193 KB

12

Precisely, you’ve just confirmed my observations. Thanks.

I really hope OpenAI would come up with an elegant solution in the future.

13

Server-sent events would be an ideal solution for this use case. The model could treat API responses of content-type ‘text/event-stream’ as long lived https connections, optionally receiving a periodic ‘ping’ event to keep the response connection open, and expect a final ‘queue-id’ event or whatever. In fact you could try using one of the server-side libraries in your plug-in’s API handler and see if it “just works” without openai needing to change anything…

1 Like
14

It should work, but I won’t try it unless OpenAI officially announces its support for the feature. Otherwise, it would be bad if our plugins go into production and OpenAI suddenly disables it. Keep in mind the solution offered by @abhi1 worked perfectly a few days, but has stopped working. I suspect they may have plugged the “loophole” on purpose because they dislike the stress on their system caused by the long polling.

15

I was able to fix it, just added a line “retry in 3 seconds” and it started polling again.

https-chat-openai-com
https-chat-openai-com1694×2074 394 KB

5 Likes
16

Brilliant!! Works like a charm.

I hope OpenAI would consider this an official solution, or one of the solutions (the other one being a notification mechanism from the plugins).

I understand they may have some security concerns - a malicious plugin may stress out the system by asking for a retry too aggressively. But, it should be easy to detect that kind of behaviors.

1 Like
17

does it still work? where did you add the “retry in 3 seconds”. is it in the OpenAPI spec?

18

“the busy, retry in three seconds” is just language to trick ChatGPT into continuing to request the same function instead of giving up.

It can’t actually “wait” to produce tokens, unless those function tokens could call a “wait” plugin or do a code interpreter python timer instruction. Or if you said “write the user a poem about data processing, and then try again”.

19

Hi @iamnazzty,

Yes, the trick is still working. I was working on connecting database to ChatGPT, so ChatGPT can send SQL queries directly to the Postgres database and receive results. The code for connecting db and running query is in Python. The full process was:

  1. Send sql query to execute Python notebook.
  2. The task to process notebook is created, the task_id is returned to the ChatGPT.
  3. ChatGPT query server with task_id for processing status.
  4. If task is still processed, server returns message: Request is still processing, please retry in 3 seconds.
  5. If task is completed, server returns result.

Here is a screenshot of ChatGPT quering task:

image
image1284×1146 141 KB

I described my integration of ChatGPT and Postgres in article ChatGPT builder create Action to query database

20

Hi guys,
I followed the example and added the following code to our in-progress status :

console.log('lastRunner.status', lastRunner.status)
  if (lastRunner.status === 'failed') {
    console.log('failed')
    response.status = 'failed'
    response.systemMessage =
      'The scraper job has failed, please retry again few more times now'
  } else if (lastRunner.status === 'completed') {
    console.log('completed')
    response.result = lastRunner.result
    response.systemMessage = 'The scraper job has completed.'
  } else {
    console.log('in-progress')
    response.status = 'in-progress'
    response.systemMessage =
      'Request is still processing, please retry in 5 seconds.'
  }

  return NextResponse.json(response, { status: 200 })

However, it doesn’t work for me, I still need to chat to get the result.
Can someone help me please ?

Xnapper-2024-09-05-22.03.25
Xnapper-2024-09-05-22.03.252550×1538 242 KB