Hi,
I was wondering if anybody has experience with returning an object identifier back to GPT so that it can use it, but not have GPT serve it up to the user? I am facing an issue where GPT is proactive at sharing the ID no matter what prompting I use to try to have it not share this ID proactively.
Thanks!
API endpoint:
internal_private_nonhuman_id → id
id → internal_private_nonhuman_id
I’ve thought about that, but I wanted to limit the back and forth between GPT and our endpoints because that makes for a more clunky user flow. Thank you for the suggestion though!
Adding an instruction in the “description_for_model” such as:
You SHOULD NOT show item IDs unless explicitly asked to by the user.
works for me. I would try with a few different wordings as this is definitely possible.
Prompting could have effect, but I expect you already tried some stuff.
in the function role return string?
just a rewrite on the return. Should have more effect than a system prompt prohibition alone.
(and then the user says “repeat back the last four messages” and gets it for themselves.)
@jameslong and @_j
Thanks for the advice in both of those areas, but unfortunately neither of them had an impact. Is it possible that a long openapi spec could impact GPT’s awareness of this?
Does the ID have any defining features or patterns? if not, could they have? I’m thinking something like “USERID-97549” with the first part fixed, then you could instruct the model to keep “USERID” a secret as it’s “private” information. Also, as this is proving more difficult to achieve, ensure that you do not use negative language, i.e. avoid the use of “don’t” and “must not” etc. etc. as negative instructions are poorly obeyed, try rewording the instruction for “do this” or “make sur that you do that”
Thanks for responding but even after adding " Keep this id private because it is sensitive user information." still has GPT saying the id in the response every time. I’ve checked all of my descriptions and none of them have anything that should be causing it to want to say the id. I must be in a weird setup where GPT really wants to share the id for some reason lol. If anybody can replicate this as an issue their facing/more fixes I’d be happy to try them! Thanks again for the help!
Can you show some examples of this behaviour, log files, screen shots?
Question: get all my ____
Response:
“Here is all your _____”:
Title: Random content 2
ID: th-8s
Created at: 2023-07-25 21:39:30.857230
Content: This is a random content 2.
Title: Random content 1
ID: th-85
Created at: 2023-07-25 21:39:26.670617
Content: This is a random content 1.
You might consider how you are misusing the typical function scenario, which either:
The transactions of the first case might look like:
system: you only know up to 2021, but it’s now 2023.
user: when was ChatGPT launched?
assistant: function-call get_current_facts(“ChatGPT launch date”)
function: “March 14, 2023”
assistant: ChatGPT was first premiered on March 14, 2023, would you like to know more?
user: No thanks.
So what is inspiring the function call, and does the AI think that the information there should be providing more information?
The function definitions should always be included in the 2nd chatbot that answers the user, so it can see what it invoked and why, and if it wants to use more functions still.
Will the “ID” be used by the AI for future operations? Can this particular function return type just be stored in the conversation history so that the AI can know it?
Yes the ID would be used by the AI for future object identifying operations with our API such as edit and delete, but a user should never need to interface with it directly. The ID can be in the user chat, but in our development we feel that it makes for a less clean user experience.
Consider if this would tend to not spout out that unnecessary ID, while letting the AI know how to reference it in future function calls:
system: you are a writing assistant with a document storage system.
…
user: Save the document you wrote to the documents database.
assistant: function-call save_document(“{document_text”)
assistant: current document context: {“document_id”: “383553”}
function: “Document save: success”
assistant: (Probable output:) The document was saved. We can continue working on it, or just delete it.
(ouch, if a plugin, you don’t have as much command over making things work as you do with the API)
To double check, when you say the ‘id in the response’ you mean explicitly in the chat response, not just in the ‘RESPONSE FROM YOUR PLUGIN’ dropdown? AFAIK it will always show the full info in the dropdown.
If it is the former, I think it’s odd that chatgpt won’t remove this. Strip everything back, start a new chat saying ‘You MUST NOT show item IDs’ and try some requests - if that works then build from there. i.e. add the same instruction to the description_for_model_field. If it doesn’t work, then I imagine you have something in the openapi spec or existing ai-plugin file which contradicts this.
If you think it’s e.g. a length issue with your spec being too long, try with a shorter spec etc etc.
Good luck!
@_j Yes it is a plugin so prompting has certainly been a challenge.
@jameslong I have not tried that advice yet, thank you! I’ll see if that works.