I have asked OpenAI ChatGPT4 for 3 issues with a gitlab CI/CD config today and in just 5 prompts it made 3 grave logical errors. I have given it a yaml config file (87 lines) and have pasted it short 16 lines of job output.
E.g. where the job ouputs starts with the ls command and prints the output immediately:
$ ls build
mathpath
There are a few lines of output in between but then the output goes:
#6 [3/3] COPY build/mathpath /app/
#6 ERROR: failed to walk /var/lib/docker/tmp/buildkit-mount3440408583/build: lstat /var/lib/docker/tmp/buildkit-mount3440408583/build: no such file or directory
The bot recognizes correctly, that a folder is not visible, but first it attempts to suggest, that the build/mathpath doesn’t exist, despite it is clearly the opposite as it follows from ls command.
Then it suggests that some artifacts may not be accessible, because the dependencies for the stage are not specified. But I have shared the yaml file with it 1-2 prompts ago, so it knows that the dependencies for the stage have been specified correctly.
For another issue in the same short conversation, it says, that some issues with repo cloning/copies may follow from artifact path clash.
" The error fatal: destination path ‘mathpath’ already exists and is not an empty directory occurs because the directory mathpath already exists when the git clone command is invoked. The mathpath directory should be deleted or moved if it’s no longer needed, or the clone should be done into a new directory to avoid the conflict.
The issue could occur in a new job for a few reasons:
Artifact Overlap: If you’re using GitLab’s CI/CD, the artifacts field in your .gitlab-ci.yml file keeps the build/mathpath directory. Ensure this doesn’t conflict with the git clone directory."
But it knows the whole yaml file, so should see, that the artifacts are not set up in a way to create a clash. When I tell it so, it apologizes again like a poor 3.5 model:
" Why are you suggesting artifact overlap if I told you the whole yaml file?
ChatGPT
My apologies for the oversight. You’re right; the complete .gitlab-ci.yml file you provided does not indicate any artifact overlap that would lead to this issue. The artifacts field specifically saves build/mathpath, which shouldn’t conflict with the git clone operation targeting the mathpath directory. Thank you for bringing it to my attention."
Such obvious logical mistakes indicate a severely downgraded level of logical reasoning and/or memory issues. Things like that have never happen to me earlier, since the time I have switched to ChatGPT4. Now it seems it is close to the 3.5 version.
What is also funny - I have just cancelled my renewal (which was also kinda tricky, as tha page didn’t want to appear a few times - maybe a subsription server DOS attack?! 
). And immediately after that, when I tried to prompt it again, it told me I hit my 24h limit for messages! I wasn’t even aware I have such! I know I have 50/4h, and I have started my day with these 5 prompts only. To me it looks like there is something malicious happening out there!
I’d guess Microsoft investment in ChatGPT has to pay back… look out for the new bing etc 
I am out!