I am using the API playground to test different versions of a System Prompt. In the final UI users will be able to write their own user prompt (or leave it blank), so at this stage I only need to test if changes to my System Prompt are working.
Since the User Prompt is required does anyone have suggestions of what to add as a user prompt? I want to make sure I’m not influencing my results accidentally, so I have just been putting “.” in the User Prompt. Any other ideas? Is there a better tool for just testing changes to the System Prompt?
That’s a really tough question. It really depends on what you’re trying to achieve.
But I would say that if you want to go down the science route, the best option would be design a proper test array with a variety of plausible user inputs - the ‘.’ (and empty string! ‘’, oof, empty string cost one project a lot of money lol) could be a valid edge case, but it probably won’t really inform real life behavior.
Personally, I would tell you that the system prompt is probably considerably less important than you may think.
Are you generally trying to get a feel for how prompting works, or are you trying to proof your system against malicious input, or perhaps just trying to ensure you get a stable, structured output?
While the scientific approach can certainly be useful and I would absolutely not want to discourage it if you have the time and energy for it, I’d personally rather take the engineering approach: use tried methods to design and construct a prompt to get the outcomes I’m looking for.