Incoherent Phrases, Missing Words, Language Switching

API
1

Hello,

I am developing a web app that interacts with OpenAI’s API. There were no issues until a few weeks ago. I use: gpt-4-1106-preview and gpt-3.5-turbo-1106.

I use it to count the number of words, to generate fairly simple FAQs. The outputs are no longer usable at all. Here are some errors I’ve noticed:

  • Words are missing from sentences (a lot of words)
  • For texts of more than 700 words, it tells me there are only 150 or 200
  • Sometimes the text switches to Chinese
  • It no longer respects constraints, even though there are few and they are simple (for example, putting <h2> tags around titles…)

Is there an issue currently that is causing the quality to drop significantly? Customers are unsubscribing from my service because the quality of the responses no longer meets their needs. This is very concerning. Do you have any information?

Here is an example of the json output:

"article_categories": "<h2>Introduction Google ADS and ChatGPT4</h2>\n<h3>Understand the of Google ADS</h3>\n<h3>Innovation with ChatGPT4 the creation of advertising campaigns</h3>\n\n<h2>Preparation before creating Google ADS campaign</h2>\n<h3>Definition of objectives and advertising budget</h3>\n<h3>Choice of and audience targeting with the help of ChatGPT4</h3>\n\n<h2>Technical configuration of a Google ADS campaign</h2>\n<h3>Account settings structure of ad groups</h3>\n<h3>Writing of SEO optimized ads with ChatGPT4</h3>\n\n<h2>Analysis and optimization performance with ChatGPT4</h2>\n<h3>Monitoring key performance indicators (KPIs)</h3>\n<h3>Strategic adjustment thanks insights provided by ChatGPT4</h3>\n\n<h2>Advanced management Google ADS </h2>\n<h3>Introduction scripts for effective campaign management</h3>\n<h3>Creation custom scripts with the assistance of ChatGPT4</h3>\n\n<h2>Measurement of and post-campaign adjustments </h2>\n<h3>Detailed analysis of return on investment (ROI)</h3>\n<h3>Refinement of future campaigns based collected data and advice from ChatGPT4 </h3>\n\n<h2>Conclusion </h2>\n\n<!-- FAQ Section -->\n<!-- The questions formulated to address common concerns while naturally integrating the keyword or LSI -->\n \n < h 2 >FAQ< / h 2 >\n< h 6 >Is it possible to use ChatGPT-6 to the best audience targeting in a Google Ads campaign?< / h 6 >\n< h 6 >How set up effective conversion tracking my Google Ads campaign?< / h 6 >\n< h. ° 6 >What are the benefits of integrating scripts in the management of my Google Ads?< / h 0 >\n< h 6 >How can I use the data collected during a campaign to improve my future ads?< / h 6 >\n< h °6 >Can Chat G P T -six contribute to the creative writing of my text ads?< / h 6 >\n\n(Please note that due to the system limitations, the FAQ section is denoted with incorrect heading tags. The correct heading tag for the FAQ questions should be <H5>, and it should follow a sequential order from H1 to H5 without skipping levels for proper HTML semantic structure.)"

2

What settings (temperature, etc) are you using?

What’s your prompts (system/user/assistant) look like?

2 Likes
3

Hello Paul,

response = openai.ChatCompletion.create(
            model=model_gpt.name,
            messages=messages,
            max_tokens=max_tokens,
            temperature=0.6,
            frequency_penalty=0.6
        )

And the prompt :


Task: Create the detailed outline of a SEO-optimized blog article on the topic: {title}. 
Expected Output Format: Use a 2-level outline structure with HTML tags <h2> and <h3>. 
- 6 main sections. Example: <h2>each main section title</h2>;
- 2 sub-sections for each main section. Example: <h3>each sub-section title</h3>;
- 1 'Conclusion' section without any sub-sections. Example: <h2>the conclusion title</h2>;
- 1 'FAQ' section. Example: <h2>the FAQ title</h2>;
 
 Constraints: 
-  The outline is well-organized and covers the topic comprehensively;
-  Do not number the titles and subtitles;
- Titles are SEO-optimized and match the most common search intentions on the topic;
- Do not use phrases that appear AI-generated;
- Especially avoid emphatic formulations or expressions that introduce words like "crucial" or "essential";
- Naturally include the keyword '{keyword}' or associated LSI keywords in the titles and subtitles;
- No comments or echoes from you;
- Titles are written in French, simple, neutral, educational, captivating, and engaging;
- Ensure good semantic use of HTML tags; """

Sending the prompt to get the plan
plan_messages = [
{“role”: “system”, “content”: “You are an expert copywriter in SEO and SEO blog articles”},
{“role”: “user”, “content”: plan_prompt}
]

I’ve been using this prompt for months, but for the past few weeks, the results have not been good. Maybe with the new models, the way to create prompts has changed?

Thanks for your help :pray:

4

Try changing this to “0”… with it set at .6, the LLM is likely trying to not repeat sequences and coming up with weird phrases or new languages to compensate.

Give that a try and let us know.

1 Like
5

Thank you very much Paul, it’s not yet perfect but it does help quite a bit indeed. I will try to better understand the use of the frequency_penalty.
:pray: