Gpt-3.5-turbo-16k api not reading context documents

[SomebodySysop]

I have no explanation for this, but the same prompt with the same context documents given to gpt-3.5-turbo-16k today, and I think for the past couple days. It’s like it is not reading the context documents at all:

While gpt-4 with the same prompt and context documents:

It’s like gpt-3.5-turbo-16k isn’t even reading the documents at all.

This is the system message:

You are a helpful assistant. Your goal is to answer each question, using the associated texts as context, as truthfully as you can. If you cannot answer the question or find the meaning or locate the documents requested, tell the user to try re-phrasing the question. Also tell the user to consider a keyword search. In addition, remind user to keep in mind that the the user access may not allow the user to view the requisite documents.

You can see the standalone question which is the question sent along with context documents.
Same context documents in same format. One literally has the title of the question.

What is going on?

[SomebodySysop]

I have corroborated that this is indeed an issue with gpt-3.5-turbo-16k.

I went to the playground with the exact same system message and exact same context documents in the exact same format:

Array (
    [0] => Array (
        [role] => 'system',
        [content] => 'You are a helpful assistant. Your goal is to answer each question, using the associated texts as context, as truthfully as you can. If you cannot answer the question or find the meaning or locate the documents requested, tell the user to try re-phrasing the question. Also tell the user to consider a keyword search. In addition, remind user to keep in mind that the the user access may not allow the user to view the requisite documents.'
    ),
    [1] => Array (
        [role] => 'user',
        [content] => 'Question: Are there any documents that mention Mark\'s IP address?
        Number of context documents: 2
        
        Context document 1: 
        Document Title: \'Apache Server Logs not showing Actual IP Addresses\' 
        Content: <The detailed content here...> 
        Source document summary: 
        Event Date: 
        Document Groups: D8 Development, D7 Development, Documentation, Development, D9 Development, Site Creation, D10 Development, AI, Real Estate 
        Document Taxonomy/Tags: 
        URL: https://master1and1-9.schoolboard.net/node/911 
        Questions that this document answers: -----

        Context document 2: 
        Document Title: \'Mark\'s New IP Address\' 
        Content: <The detailed content here...> 
        Source document summary: 
        Event Date: 
        Document Groups: Development, D9 Development 
        Document Taxonomy/Tags: 
        URL: https://master1and1-9.schoolboard.net/node/1018 
        Questions that this document answers: -----'
    )
)

Both gpt-3.5-turbo-16k and gpt-3.5-turbo-16k-0613 both return the same response:

Based on the provided documents, there is no mention of Mark’s IP address. Please try rephrasing your question or consider a keyword search to find the information you are looking for.

However, with no change other than switching the model to gpt-4:

Yes, there is a document that mentions Mark’s IP address. In the context document 2 titled ‘Mark’s New IP Address’, it mentions that Mark’s new home IP address is…

What the hey!! This presents a huge problem that I need to get solved. HELP!!!

For context, this is the full text sent to the model (with IP addresses edited out):

Array ( 
	[0] => Array ( 
		[role] => system [
		content] => You are a helpful assistant. Your goal is to answer each question, using the associated texts as context, as truthfully as you can. If you cannot answer the question or find the meaning or locate the documents requested, tell the user to try re-phrasing the question. Also tell the user to consider a keyword search. In addition, remind user to keep in mind that the the user access may not allow the user to view the requisite documents. ) 
		
	[1] => Array ( 
		[role] => user 
		[content] => Question: Are there any documents that mention Mark's IP address? 
			Number of context documents: 2 
			Context document 1: Document Title: 'Apache Server Logs not showing Actual IP Addresses' Content: httpd logs showing this for IP address: 172.31.8.226 - - [07/Jan/2020:18:15:22 -0800] "GET /core/misc/icons/bebebe/chevron-disc-right.svg HTTP/1.1" 200 440 "https://d8.oursite.com/sites/d8.oursite.com/files/css/css_nKpOlPAZPrZR8AxT1eAKQiXbOQwLCsjSJwCkzckgNyI.css" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" 172.31.8.226 - - [07/Jan/2020:18:15:23 -0800] "GET /core/misc/icons/73b355/check.svg HTTP/1.1" 200 334 "https://d8.oursite.com/sites/d8.oursite.com/files/css/css_nKpOlPAZPrZR8AxT1eAKQiXbOQwLCsjSJwCkzckgNyI.css" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" 172.31.8.226 - - [07/Jan/2020:18:15:23 -0800] "GET /admin/content HTTP/1.1" 200 207697 "https://d8.oursite.com/admin/structure" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" instead of this: **** - - [07/Jan/2020:18:18:07 -0800] "POST /quickedit/attachments?_wrapper_format=drupal_ajax HTTP/1.1" 200 9169 "https://d8.oursite.com/board" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" **** - - [07/Jan/2020:18:18:07 -0800] "GET /sites/d8.oursite.com/files/js/js_lgWSmd1Z6uF0vN8cDr_QXgogXEk0wwwVM4Q6Rt55AiA.js HTTP/1.1" 200 1499 "https://d8.oursite.com/board" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" **** - - [07/Jan/2020:18:18:08 -0800] "GET /user HTTP/1.1" 302 372 "https://d8.oursite.com/board" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" **** - - [07/Jan/2020:18:18:08 -0800] "GET /user/1 HTTP/1.1" 301 378 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" **** - - [07/Jan/2020:18:18:08 -0800] "GET /user/1 HTTP/1.1" 200 109227 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"   Followed this guidance to fix: Logging Actual Client IP Address https://campus.barracuda.com/product/webapplicationfirewall/doc/4259905/logging-actual-client-ip-address-on-the-apache-server/ Log into the Apache server. Go to  /etc/httpd/conf  or  /usr/local/apache2/conf path  and open the file  httpd.conf . Search for the string: “ Source document summary: Event Date: Document Groups: D8 Development, D7 Development, Documentation, Development, D9 Development, Site Creation, D10 Development, AI, Real Estate Document Taxonomy/Tags: URL: https://master1and1-9.oursite.com/node/911 Questions that this document answers: ----- 
			Context document 2: Document Title: 'Mark's New IP Address' Content: Mark's new home IP address: ****   So I got the static IP fixed.    1. Took new router out of mix, put in old router 2. Call Spectrum to confirm subnet mask and router 3. Reboot Spectrum - get static IP  ****  /  255.255.255.0  /  ****. Can get to internet via IP and confirm a whatsmyip.com 5. Put static info into new router - reboot - Have Internet and static IP ..... duhh 6. Login to AWS as  support@oursite.com  and change all old IPs in security groups to new IP 7. Do same thing logged in to PLD account Source document summary: Event Date: Document Groups: Development, D9 Development Document Taxonomy/Tags: URL: https://master1and1-9.oursite.com/node/1018 Questions that this document answers: ----- ) )

[SomebodySysop]

So, I asked gpt-4 codex (my personal assistant) what it thought about this. Of course, it mentioned that it’s training cutoff was in 2021, but I asked it to speculate given the data. Here is what it came back with:

The content of the array you’ve posted is very complex and contains a lot of details. GPT-3.5-turbo-16k, or any version of the GPT model, performs best with specific, clearly articulated prompts.

If your goal is for the model to identify documents that mention Mark’s IP address, the prompt might be too lengthy and detailed for the model to pick up on the key question.

Here are a few things you could try:

1. Simplify the Prompt: Try to make the question more direct and straightforward. For example, you might start with: “In the following documents, identify any that mention Mark’s IP address.”

2. Segment the Data: If you are working with multiple documents, it might be better to send them one at a time or in smaller groups, then ask the model to identify which contain the specific information.

3. Provide Contextual Hints: If you want the model to perform a certain task, you can include instructions in the prompt. For example: “You are a research assistant tasked with identifying any documents that mention Mark’s IP address. Review the following documents: …”

4. Structure the Information: If possible, try to structure your data in a way that’s easier for the model to process. For example, you could break down the documents into sections and label them clearly.

Remember, GPT models generate responses based on the input they receive. The more clear and concise your prompt, the better the model’s output is likely to be.

So, I went back to the playground and asked and changed just this part of my prompt:

Question: in the following context documents,is there any mention of Mark’s IP address? Number of context documents:

And, it came back with this (first time ever):

Yes, there is a mention of Mark’s IP address in the provided context documents. In context document 2 titled “Mark’s New IP Address”, it states that Mark’s new home IP address is ****

I also went back to my query prompt and changed the question slightly:

I’m happy to finally get a result, but it’s pretty frustrating and alarming that simple questions like this with all the documentation needed to answer won’t even result in a partial response.

[SomebodySysop]

Not to belabor the point (well, maybe a little), but I changed the format of my context documents just to make them more clear to the LLM: How to format context documents to allow model to recognize specific fields within documents - #2 by SomebodySysop

So, I don’t think it’s a format issue. I think the model just doesn’t have the “juice” to understand those particular documents whereas gpt-4 does. But, again, how “artificial” is this “intelligence” when it looks at the same documents, but is confused by “show me the documents where mark’s ip is mentioned” but can answer “what is mark’s ip address”.

[Foxalabs]

I’m confused here, not reading the documents? How are you passing those documents to the LLM? and do they contain anything that could a) look like a data boundary marker or b) look like you attempting anything nefarious.

[SomebodySysop]

I uploaded the actual data I sent, in the format I sent it, here: Gpt-3.5-turbo-16k api totally unresponsive - #2 by SomebodySysop

This is the format I use for all the documents in question here: How to format context documents to allow model to recognize specific fields within documents - #2 by SomebodySysop

I am as baffled as anyone. In the last screenshot I posted, the format of the content did not change at all. Just the question.

I asked it three times, in different ways, what info did the docs have regarding Mark’s IP address. Each time it said it could find nothing. Then, I asked, “What is Mark’s IP address?” And it answered.

So, I guess you could make the argument that it needed a better prompt. But, I mean, come on! In two cases, the docs clearly say “Mark’s New IP Address”!!!

[SomebodySysop]

OK, community, I need some serious help here! I tried modifying the prompts:

[0] => Array ( [role] => system [content] => You are a helpful assistant. Your goal is to answer each question, using the associated texts as context, as truthfully as you can. If you cannot answer the question or find the meaning or locate the documents requested, tell the user to try re-phrasing the question. Also tell the user to consider a keyword search. In addition, remind user to keep in mind that the the user access may not allow the user to view the requisite documents. )

[1] => Array ( [role] => user [content] => Question: What information can be gathered about Mark’s IP address from the conversation? These are the context documents you are to examine for the answers: Context document 1: blah, blah ----\n Context document 2: blah, blah ----\n, etc…

But I continue to get a response from gpt-3.5-turbo-16k that seems to indicate it cannot / will not read the context documents. This is a SERIOUS difference between two models using the same prompt and context documents.

You can see from the above example, gpt-4 gives me a beautifully detailed and accurate response while gpt-3.5-turbo-16k appears not even able to read the context documents.

The submitted context documents are formatted like this:

Context Document 1: etc… ‘-----’ followed by line feed

This is the code within a foreach loop:

// Append the context document string to the prompt content|
$promptContent .= Context document  . ($index + 1) . : {$documentString}\n;|
$promptContent .= -----\n; // Delimiter to separate context documents|

I cannot for the life of me figure out why the models are responding so wildly differently, and especially why gpt-3.5-turbo-16k appears totally unable to see/understand the submitted contexts.

Help!!!

[SomebodySysop]

This is part of the conversation I had with the AI in the playground with the same user prompt and context documents.

I asked it to examine the prompt. It told me the prompt consisted of a question and two documents. I asked it for the name of the documents. It gave me the name of each. I asked it if there was any information about Mark’s IP address mentioned, and it said yes and gave me the address.

Then, I re-submitted the original user prompt, and it gave me the same response as usual: It could not find any mention…

Here is what happened afterwards:

So, it said it couldn’t find it, then upon re-examination, found it. As you can see from the following, I have tried my best to get it to tell me why it couldn’t see the information in the first place, but it won’t tell me.

I kind of feel like Captain Kirk trying to get the computer to analyze why it’s doing what it’s doing.

This is totally frustrating as I really need to be able to use gpt-3.5-turbo-16k, but it is consistently returning this message regardless of the topic.

[Foxalabs]

I got this from my API chat environment, how does that look to you? It’s with a temp of 0 so it may be a little bland

[SomebodySysop]

It is right on point. The problem is I’m not getting anything like that from the gpt-3.5-turbo-16k api or playground. Up until I asked it to re-examine the prompt, it repeatedly denied seeing any information about Mark’s IP address.

Did you use gpt-3.5-turbo-16k for that response? I know gpt-4 api and playground work without issue, but I’m TRYING to get gpt-3.5-turbo-16k to at least be able to give basic, competent responses.

[Foxalabs]

That’s all I’m using for that response.

Hmmmm

[SomebodySysop]

Would maximum length have anything to do with it? I see you have max_tokens=13000 whereas the playground has a limit of 2048.

Although, that doesn’t seem to affect gpt-4.

[Foxalabs]

Questions that this document answers: ----- ) )


------------------------------------------the above is context--------------------


Question: What information can be gathered about Mark’s IP address from the conversation?

That is the prompt I sent with no system message

[Foxalabs]

Yes the 2048 looks like it would truncate a lot of the message by the time you are at that point in your conversation.

[SomebodySysop]

OK, thank you much. I turned off the standalone question and tried your prompt (in the api), and it worked! So, at least that rules out the format of the context documents.

And this is WAY better than before. At least points me in some promising directions.

Again, Thanks!!!

[Foxalabs]

Always welcome. It can be a very frustrating place to be when everything looks like it’s just gone crazy, finding an island of stability to venture out from can be a sanity saver