Using Chain of thought technique with response_format

Hi,

I’m trying to use few shot learning to produce an output in json format using response_format. However, with the response_format other than text, it seems to ignore few shot learning - I guess this is because the model (gpt3.5 turbo) would not generate token to explain its steps. For example, if my prompt is:

Blockquote
You are trying to summarize a paragraph of product description. When given a paragraph, you will:
Step 1: Boil the paragraph down into keyword list.
Step 2: Summarize what impression you can get from the paragraph into keyword list.
Step 3: Merge both into one list.
Step 4: Remove duplicate or similar keywords.
Describe each step. Then, give the final list in json form.
Example 1:
User input: Our hair dryer concentrates on a new futuristic design. It has sleek body, diamond-cutted finishes, and hexagonal grill. It fits perfectly with any modern and simple rooms.
Response:
Step 1: [hair dryer, new, futuristic, sleek, diamond-cutted, hexagonal grill, modern, simple]
Step 2: [new, next-generation, luxury, high quality]
Step 3: [hair dryer, new, futuristic, sleek, diamond-cutted, hexagonal grill, modern, simple, new, next-generation, luxury, high quality]
Step 4: [hair dryer, new, modern, sleek, diamond-cutted, hexagonal grill, simple, luxury, high quality]
Json = {
“keywords” : [hair dryer, modern, sleek, diamond-cutted, hexagonal grill, simple, luxury, high quality]
}

With a default response_format, the model does its job properly. However with response_format = json, it was only returning an answer as if it only performed step 1.
Is response_format not designed to work well with few shot learning with examples?

Many thanks in advance.

You make AI output worse, because JSON requires strings in quotes.

Boil the paragraph into keyword list? We don’t immerse words in hot water.

etc.


Imagine that JSON mode ensures that there is only a single JSON that is created that would be a valid input for an API that only accepts the JSON format.

1 Like

Just a small point to add: The differentiation between steps 1 and 2 is not entirely clear, neither based on the wording nor the examples . If it’s not clear to a human, then likely the AI struggles too and this is why it might only be performing the first step.

1 Like

Makes complete sense. I will improve the prompt and try again.

I wonder if json mode will have any effect in making few-shot learning less effective by forcing the model to answer in a specific way though. I’m not sure if it works by letting the model answer normally and then find json output then part it, or if it works by making the model to come to a final json output response straight away.

Thanks for the advice. I will make each step specific or just combine them into one.

What you show is a chain-of-thought. Breaking down a big task into smaller tasks. It is not few-shot, which is giving a synthetic conversation, showing the input and desired output over and over as if it actually was a conversation or pattern before the final user input to be completed.

However, there is no self-reflection there. What the AI came up with in the first pass is output.

It’s not like you asked the AI to make a top 20 list, and then a quality score to each item, then rerank into a sorted top-10 list, then translate to an uncommon world language, then provide a justification for the appearance of the word in the list when in the destination language…

It likely excess words for “AI response is a pythonic list: perform top-10 search engine keyword extraction on text, then append top-5 topical and subjective keywords based on text.”

1 Like

Blockquote
“AI response is a pythonic list: perform top-10 search engine keyword extraction on text, then append top-5 topical and subjective keywords based on text.”

You are definitely right, I had a misunderstanding of the notion of few shot learning, I updated the thread title as well. And I think this example prompt you gave me is much simpler and will save me tokens…! Will definitely rephrase mine into something like yours, thanks.

But let’s say I have a really well-defined prompt that makes use of decent chain-of-thought technique. At the end of the prompt, I will be able to say “describe each step and then provide the final answer”, or just “provide the final answer only”. My assumption was the the former will work much better than the latter due to the nature of LLMs i.e. the model will take into account what it already said (previous steps) while predicting the next word to say.
I wonder if this assumption is correct. And if so, I wonder if response_format option will have the same effect of asking for the model to respond with the final answer only.