Am i insane or all models are hallucinating at specific interval of day

so i have been using gpt-4 turbo prior release of gpt4o , and i switched to gpt4o as soon as it was availible. i noticed that during certain period of time for me at morning to night it just hallucinates like stupid , it is reading the small data and on that it is also hallucinating. in prompt i even suggested only reply from our dataset , and if you dont know or information lacks just notify the user that information is missing . i have added this types of different prompts at many places specifically when context window overflows i cut the message and append my system message similar to this , in order to avoid hallucinations. i have added this types of runtime prompts and suggestions in many places including specific tool call. but the thing is before i didnt had this kinds of hallucinations issues. they were there but not like this signioficanly stupid. just makes up random shit , with super confidence , i tried with different different tempature settings as welll. and i was losing hope and got super frastrated and it suddenly stops hallucinating this was night time. i was happy but very next day it just starts hallucinating all over again. and same thing happened , right now it started working.

i maybe getting insane out of frastration stress or just i am stupid and possible that i by my self is blind to issues or bugs related to my code and prompt and this time based stupudity finding is not real . but i had this type of feeling since long time. i tried all previous models and all were hallucinating yesterday.

for reader context: → i am using rag system and feeding data to model during user query. when input is larger then context window i cut it , i have set limits per user message as well that this 30k tokens are only allowed and i cut the message and add system prompt . i thought maybe cutting information was causing model to fill up gaps in data with random shit and possible that it is not followihng my system prompt , so then i tried with smaller msg which is less than 10k tokens , and still it hallucinated.

i am not putting any blame or anything, but please do educate me on this topic. i feel like suddenly model has become over energetic to provide response regardless of accuracy or correctness . like crazy hyperactive intern.

tested models :

# "model": "gpt-4-0125-preview",
    # "model": "gpt-4-turbo-2024-04-09", 
    "model": "gpt-4o-2024-05-13",
    # "model": "gpt-4o",
1 Like

right now it is in god mode , no hallucinations whats so ever. and currently it is night.

Yes, I’ve noticed the same thing . I am able to somewhat fit it by starting a new session. I’m almost ready to switch back to gpt4 Turbo.

2 Likes

Absolutely possible. Eg, there might be daemon processes reducing available compute which impacts capability

But you need to give us some specific data and time windows to go on instead of lots of words.

1 Like

i do understand that i need to provide data but i have signed non disclosure program, and also what i wrote was just full on frustration :joy::joy:. i will share the data if i get permission.

Ah, well, fwiw, I don’t think there is a loophole in NDAs that says if you just use words of frustration you’re ok.

There’s no “god mode” for GPT, and and the model’s listed are not under NDA.

I don’t know who you’ve signed that a non discourse agreement with, but it’s probably not OpenAI.

Just ran into this a few minutes ago with an Assistant. Asked it the exact same question twice from two sessions, on one it completely hallucinated the answer, and the other was bang on correct.

Its almost as if their vector storage is not returning results so the AI makes stuff up instead of giving an error. Our instructions even say to only base answers on data provided in our documents.

Remember: file_search and its vector storage is powered only by the AI’s abitlity to correctly use tools and output quality semantics as a query that will result in an embeddings similarity…

of course i dont have nda with openai , i am working on project in company where we used gpt 4 turbo , second i haven’t disclosed anything in text , basically what i said is just standard practice for llm application that you handle context overflow , rag , allowed tokens per message etc… the god mode i was referring to was analogy that it is making no hallucinations whats so ever.

I thought I was the only one …

I’ve noticed the phenom more on the interactive side than the API side - usually my interactive questions are “harder” than stuff I’d run through the API.

I definitely get the sense that GPT-4 and GPT-4o go through ‘not-as-smart-as-usual’ periods. I’d generally attribute it to resource-thresholding, but I’m not sure the phenom is tied strictly to peak-usage.

Mindful of intellectual honesty, I cannot rule out that I’m the one having the ‘not-as-smart-as-usual’ periods :sweat_smile:.

1 Like

well then you are not alone buddy😂.

Hold on. Does this strangeness happen around 4:20 in any timezone by chance? Just curious! Small smile.

1 Like

for me it was around ~11:00AM till ~8:00PM and one day it was around 1:00PM till ~11PM. what happened at 4:20?

1 Like

The time 4:20 (or 420) has become a bit of a cultural joke primarily associated with cannabis culture. The number “420” is commonly referenced in popular culture and among cannabis users as a sort of unofficial holiday on April 20th (4/20 in U.S. date notation) to celebrate and consume cannabis.

The origins of the term are a bit hazy, but it’s generally believed to trace back to a group of high school students in the 1970s, known as the “Waldos” because they hung out by a wall. They would meet at 4:20 PM to search for an abandoned cannabis crop they had heard about. They used “420” as a code word for their quest, and it eventually became a code for smoking marijuana in general.

Now, when someone mentions “4:20” or just “420,” it’s often done with a wink and a nod to this cannabis culture. It’s used both humorously and seriously, serving as a kind of in-group signal among those who participate in or are friendly to cannabis culture.

Inferring maybe the LLM gets “high” at a certain time every day.

Disregard. Trying to bring levity to the thread…

I’m pretty sure they do turn knobs/dials in the background without always telling us…

1 Like

i am definately going to use this 420 in chats.

A humorous and colorful digital artwork depicting a young man laughing as he types on his laptop in a cozy, casual setting. The screen of the laptop shows a chat window where he’s just typed ‘Happy 4:20!’ with a backdrop of a vibrant, cartoonish clock showing 4:20. The room around him is filled with playful and quirky decorations, including a small plant and a poster of a whimsical clock. This scene captures a moment of joy and amusement, emphasizing the light-hearted nature of the 4:20 meme in online chats.

I think DALLE was high here hehe :wink:

2 Likes