I assume that Custom Instructions are meant to be manually added by users when their requests are very clear and explicit. Each round of questioning merges the Custom Instructions into the Prompt.
Memory, on the other hand, is acquired through conversation, as requested by the user to remember. The use of Memory is decided by GPT in conjunction with the context, and is an agent behavior.
I’ve drawn a flowchart, but I’m not sure if it’s correct.