API Prompt for gpt-3.5-turbo-16k

I would really, really like to be able to use gpt-3.5-turbo-16k for more basic functions (like double checking outputs), but I just can’t seem to get good responses from it. I send it the same context documents as I send to gpt-4, in the same format, and here are the typical results:

So I’m thinking that maybe it’s the way I am constructing the prompt itself.

Essentially, it’s:

User Question: blah, blah
Context Documents:
Context Document 1: blah blah
Context Document 2: blah, blah
etc …

gpt-4, gpt-4-turbo, claude-2 have no problems reading similarly structured prompts, but gpt-3.5-turbo-16k just refuses. As does, for the record, gemini-pro.

It’s got to be the way I’m formatting the prompt. Is there anyone out there sending large amounts of text to gpt-3.5-turbo-16k in a RAG scenario and getting halfway decent responses?

I’d say your already identified the culprit: Try reformatting the return table.

It did take me a bit to grasp that 0.22 is the relevancy score (and not something else) which should be used to order the final results.
What about: Index:1, Score:0.22, Name: Jurassic 5,…

And then specify that score is the relevant value for the returned list.

Imagine handing over the list to a elementary schooler. They should be able to sort it but it needs to be more clear.


Edit: I did it like this now.

Right. But, what is a good way to format that table so that gpt=3.5-turbo-16k will recognize it as a list of documents to use as context?

This is sort of what I am sending (as json) to the model now:

Array (
    [0] => Array (
        [role] => 'system',
        [content] => 'You are a helpful assistant. Your goal is to answer each question, using the associated texts as context, as truthfully as you can. If you cannot answer the question or find the meaning or locate the documents requested, tell the user to try re-phrasing the question. Also tell the user to consider a keyword search. In addition, remind user to keep in mind that the the user access may not allow the user to view the requisite documents.'
    ),
    [1] => Array (
        [role] => 'user',
        [content] => 'Question: Are there any documents that mention Mark\'s IP address?
        Number of context documents: 2
        
        Context document 1: 
        Document Title: \'Apache Server Logs not showing Actual IP Addresses\' 
        Content: <The detailed content here...> 
        Source document summary: 
        Event Date: 
        Document Groups: D8 Development, D7 Development, Documentation, Development, D9 Development, Site Creation, D10 Development, AI, Real Estate 
        Document Taxonomy/Tags: 
        URL: https://master1and1-9.oursite.com/node/911 
        Questions that this document answers: -----

        Context document 2: 
        Document Title: \'Mark\'s New IP Address\' 
        Content: <The detailed content here...> 
        Source document summary: 
        Event Date: 
        Document Groups: Development, D9 Development 
        Document Taxonomy/Tags: 
        URL: https://master1and1-9.oursite.com/node/1018 
        Questions that this document answers: -----'
    )
)

Do you think it would recognize it better if I restructured like this?

Array (
   [0] => Array (
       [role] => 'system',
       [content] => Array (
           [instructions] => 'You are a helpful assistant. Your goal is to answer each question, using the associated texts as context, as truthfully as you can. If you cannot answer the question or find the meaning or locate the documents requested, tell the user to try re-phrasing the question. Also tell the user to consider a keyword search. In addition, remind user to keep in mind that the the user access may not allow the user to view the requisite documents.'
       )
   ),
   [1] => Array (
       [role] => 'user',
       [content] => Array (
           [questions] => Array (
               [0] => Array (
                   [text] => 'Are there any documents that mention Mark\'s IP address?',
                   [context_document_count] => 2
               )
           ),
           [context_documents] => Array (
               [0] => Array (
                   [title] => 'Apache Server Logs not showing Actual IP Addresses',
                   [content] => '<The detailed content here...>',
                   [summary] => '',
                   [event_date] => '',
                   [document_groups] => Array('D8 Development', 'D7 Development', 'Documentation', 'Development', 'D9 Development', 'Site Creation', 'D10 Development', 'AI', 'Real Estate'),
                   [taxonomy_tags] => Array(),
                   [url] => 'https://master1and1-9.oursite.com/node/911',
                   [answers_questions] => Array()
               ),
               [1] => Array (
                   [title] => 'Mark\'s New IP Address',
                   [content] => '<The detailed content here...>',
                   [summary] => '',
                   [event_date] => '',
                   [document_groups] => Array('Development', 'D9 Development'),
                   [taxonomy_tags] => Array(),
                   [url] => 'https://master1and1-9.oursite.com/node/1018',
                   [answers_questions] => Array()
               )
           )
       )
   )
)

Has anyone tried sending a prompt like this, successfully? Can gpt-3.5-turbo-16k read it like this?

Thanks for this. My format appears to be a bit more complex as it has much more metadata and the context could be several paragraphs long.

Perhaps I could try taking out the linefeeds within each document, essentially making each document structurally one line?

What do you think about my idea of structuring the entire content response in json?

1 Like

Can’t you do all of this via scripting?
Since the answer is just a single sentence + the list and the additional entries I actually believe this approach can be simplified a lot.
Then you can ask the model to simply return the prepared list you provide with a nice sentence.

1 Like

I have code that is formatting the prompt now. The problem is getting a format that works. So far, just sending all the text in the way I’ve been doing it hasn’t worked. I can format it any way I want. A way the model will always recognize, that’s the problem I face.

It’s actually pretty simple now. I send the system message, a question and a list of documents. The problem is how can the list of documents be sent in a way that gpt-3.5-16k will recognize? Especially when there are lots of documents with lots of text (all under the context limitations, of course).

This has been frustrating me for months now.

1 Like

I assume that you don’t need GPT 3.5 to fully grasp the contents of the returned list but you are really just trying to make it answer to this one specific query.

In this case you can separately prompt the model to write a sentence matching the conversation (“Since you want to know about A in the context of XYZ, here is a comprehensive list with sources and more details about XYZ”…), then append your own results and send the message to the user.

This would be a cost and time efficient solution.
Or do you really need GPT 3.5 to work through the contents of the returned document chunks and the additional metadata?

YES! That is exactly what I want it to do.

OK, I think I’ve made some progress. Here is an example of a prompt I sent, and also entered into the playground, that is answered by gpt-4 and gpt-4-turbo but consistently fails in gpt-3.5-turbo-16k:

Question: Are there any documents that mention Mark’s IP address? Number of context documents: 2
Context document 1: Document Title: ‘Apache Server Logs not showing Actual IP Addresses’ Content: httpd logs showing this for IP address: - - [07/Jan/2020:18:15:22 -0800] “GET /core/misc/icons/bebebe/chevron-disc-right.svg HTTP/1.1” 200 440 “https://d8.oursite.com/sites/d8.oursite.com/files/css/css_nKpOlPAZPrZR8…” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36” - - [07/Jan/2020:18:15:23 -0800] “GET /core/misc/icons/73b355/check.svg HTTP/1.1” 200 334 “https://d8.oursite.com/sites/d8.oursite.com/files/css/css_nKpOlPAZPrZR8…” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36” - - [07/Jan/2020:18:15:23 -0800] “GET /admin/content HTTP/1.1” 200 207697 “https://d8.oursite.com/admin/structure” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36” instead of this: **** - - [07/Jan/2020:18:18:07 -0800] “POST /quickedit/attachments?_wrapper_format=drupal_ajax HTTP/1.1” 200 9169 “https://d8.oursite.com/board” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36” **** - - [07/Jan/2020:18:18:07 -0800] “GET /sites/d8.oursite.com/files/js/js_lgWSmd1Z6uF0vN8cDr_QXgogXEk0wwwVM4Q6Rt55AiA.js HTTP/1.1” 200 1499 “https://d8.oursite.com/board” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36” **** - - [07/Jan/2020:18:18:08 -0800] “GET /user HTTP/1.1” 302 372 “https://d8.oursite.com/board” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36” **** - - [07/Jan/2020:18:18:08 -0800] “GET /user/1 HTTP/1.1” 301 378 “-” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36” **** - - [07/Jan/2020:18:18:08 -0800] “GET /user/1 HTTP/1.1” 200 109227 “-” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36” Followed this guidance to fix: Logging Actual Client IP Address Page not found | Barracuda Campus Log into the Apache server. Go to /etc/httpd/conf or /usr/local/apache2/conf path and open the file httpd.conf . Search for the string: “ Source document summary: Event Date: Document Groups: D8 Development, D7 Development, Documentation, Development, D9 Development, Site Creation, D10 Development, AI, Real Estate Document Taxonomy/Tags: URL: https://master1and1-9.oursite.com/node/911 Questions that this document answers: ----- Context document 2: Document Title: ‘Mark’s New IP Address’ Content: Mark’s new home IP address: So I got the static IP fixed. 1. Took new router out of mix, put in old router 2. Call Spectrum to confirm subnet mask and router 3. Reboot Spectrum - get static IP / 255.255.255.0 / . Can get to internet via IP and confirm a whatsmyip.com 5. Put static info into new router - reboot - Have Internet and static IP … duhh 6. Login to AWS as support@oursite.com and change all old IPs in security groups to new IP 7. Do same thing logged in to PLD account Source document summary: Event Date: Document Groups: Development, D9 Development Document Taxonomy/Tags: URL: https://master1and1-9.oursite.com/node/1018 Questions that this document answers: -----

I just tested it several times in the playground, and it fails 6 out of 7 tries.

So, I reformatted the same prompt like this:

       [questions] => Array (
            [0] => Array (
                [text] => 'Are there any documents that mention Mark\'s IP address?',
                [context_document_count] => 2
            )
        ),
        [context_documents] => Array (
            [0] => Array (
                [title] => 'Apache Server Logs not showing Actual IP Addresses',
                [content] => 'httpd logs showing this for IP address: <ip> - - [07/Jan/2020:18:15:22 -0800] "GET /core/misc/icons/bebebe/chevron-disc-right.svg HTTP/1.1" 200 440 "https://d8.oursite.com/sites/d8.oursite.com/files/css/css_nKpOlPAZPrZR8…" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" and more log entries...',
                [summary] => '',
                [event_date] => '',
                [document_groups] => Array('D8 Development', 'D7 Development', 'Documentation', 'Development', 'D9 Development', 'Site Creation', 'D10 Development', 'AI', 'Real Estate'),
                [taxonomy_tags] => Array(),
                [url] => 'https://master1and1-9.oursite.com/node/911',
                [answers_questions] => Array()
            ),
            [1] => Array (
                [title] => 'Mark\'s New IP Address',
                [content] => 'Mark\'s new home IP address: <ip>. Steps taken to fix the static IP: 1. Took new router out of mix, put in old router 2. Call Spectrum to confirm subnet mask and router 3. Reboot Spectrum - get static IP <ip> / 255.255.255.0 / <ip>. Can get to internet via IP and confirm at whatsmyip.com 5. Put static info into new router - reboot - Have Internet and static IP 6. Login to AWS as support@oursite.com and change all old IPs in security groups to new IP 7. Do same thing logged in to PLD account',
                [summary] => '',
                [event_date] => '',
                [document_groups] => Array('Development', 'D9 Development'),
                [taxonomy_tags] => Array(),
                [url] => 'https://master1and1-9.oursite.com/node/1018',
                [answers_questions] => Array()
            )
        )

And it succeeds 6 out of 7 times. So, it looks like the plan forward is clear. I need to test this in the API. If it works, it will be a revelation.

BTW, all the data above has been replace with dummy data. The original data contained Mark’s IP address, and the task was for the LLM to find and report it.

1 Like

I did a little bit of prompt tweaking. This should get you closer to 100%.

Question: Reply with the number of the context document that mentions Mark’s IP address.
Additional Information: There are 2 context documents attached. Use the following approach to answer the question: Role 1 will read the provided context documents. Role 2 will reply to the question. Like so 'The number of the context document that mentions Mark’s IP address is context document {actual document number identified by Role 1}

And when the AI can’t predict and generate the correct “number” token by simply seeing 1000+ words, you produce chain-of-thought first before final output:

Response, Part 1:

In a numbered list, from all context documents provided, print an excerpt of each line that contain a mention of Mark’s IP address.

Response, Part 2:

Then print “Total IP appearances of Mark’s IP address = nnn” with the length of the IP address list from Part 1.

Response, Part 3:

Using Part 2 total:
(output format specification, within container)

- Reminder: All parts are language to be produced in one single response.

It appears the solution to this was to send the prompt formatted as XML code:

      xmlPrompt : 
      <query>
        <questions>
          <question>
            <text>What is Mark's IP address based on the conversation?</text>
            <context_document_count>5</context_document_count>
          </question>
        </questions>
        <documents>
          <!-- Root element containing all context documents -->
          
          <!-- Context Document 1 -->
          <contextDocument id="1">
            <!-- Title of the document -->
            <title description="The title of the document.">my-queries-export.csv</title>
            
            <!-- Main content of the document -->
            <content description="Main content of the document.">
              Username,Date,IP,...
            </content>
            
            <!-- Summary of the document -->
            <summary description="Brief summary of the document."></summary>
            
            <!-- Event date of the document -->
            <eventDate description="Date of the event or document creation."></eventDate>
            
            <!-- Groups associated with the document -->
            <groups description="Groups or categories associated with the document.">Documentation, Development, D9 Development, AI</groups>
            
            <!-- Taxonomy or tags of the document -->
            <taxonomy description="Taxonomy or tags for categorization."></taxonomy>
            
            <!-- URL of the original document -->
            <url description="URL of the original document.">https://master1and1-9.schoolboard.net/system/files/my-queries-export.csv</url>
            
            <!-- Questions that this document answers -->
            <questions description="Questions that this document answers."></questions>
          </contextDocument>
          
          <!-- Context Document 2 -->
          <contextDocument id="2">
            ...
          </contextDocument>
          
          ...
          
        </documents>
      </query>

As I initially surmised, the problem, besides gpt-3.5-turbo-16k being considerably less efficient at reading text than the gpt-4 family, was the format of my submitted prompts. This one tweak has resulted in much better (though far inferior to it’s gpt-4 cousins) responses:

A step in the right direction, at the very least.

1 Like