Giving up on Chat GPT, here is why

Hi guys, I am extremely frustrated here as I cannot seem to get a consistent quality in the response from Chat GPT in all of its versions. I have the paid subscription and I am trying to ask chat GPT to follow a prompt that helps me make superior quality descriptions for some day trips.

For the past two days I must have spent 12 hours trying to polish the prompt so that the outcome is always of a similar standard. However, I feel like giving out now as it is ridiculous. You work for hours on it, eventual get one or two answers in the structure, style and format you wanted, but then all of a sudden GPT reverts back to not following all the guidelines in the prompt. Does anyone have suggestions for this? At first I thought it was the prompt that was wrong, but I must have tried a 100 different ways of providing outlines and GPT consistently seems to forget to go through the full prompt rules after two or three successful results.

The most frustrating one I that I am specifically asking it to write two or more paragraphs, however, 50% of the time it chooses to only write one.

Here is version 500 of the prompt in case anyone has suggestions:
Create a day itinerary from the provided texts. Rules:

Begin with the trip descriptive paragraphs directly, so no titles here. Write in a realistic style, avoiding superfluous adjectives or vague language, unnecessary embellishments. Only use adjectives when necessary. Do not write corny, over-the-top phrases. Craft clear, detailed, and realistic portrayals of the day’s activities, distributed across 2 or more paragraphs. Each paragraph should focus on a different event, activity, or site visit, providing clarity and sequential flow. Incorporate appropriate context to explain each event, activity, or site. Emphasize unique aspects of each trip or activity. Ensure comprehensiveness, avoiding summaries that skip details. Avoid repeating words.

Paragraphs should follow one another without space in between.

After the descriptive paragraphs, leave one free line and then write: “Schedule:”. In the line directly under “Schedule:” detail the day’s main activities of the day. Write them as sentences, one after the other. Exclude mentions of spare time or optional activities.

After the schedule, leave a line and then write: “Approximate Trip Duration Section:”

Here is an example of how your output should look:

"Our second day exploring the Colca Canyon starts with a very early visit to La Cruz del Condor (The Condor’s Cross) viewpoint. This is an ideal spot to observe the majestic flight of Andean condors and the impressive depth of the canyon. Observing these magnificent birds, especially on bright sunny days which are common in this area, is a highlight. You can usually see between 10 and 15 condors overflying just a few meters away. However, it’s important to note that this is a weather-dependent activity, and condor sightings are less likely if it’s rainy.

After spending some time at the viewpoint, the journey continues with a drive back to your hotel. On the way back, there’s an opportunity to make short stops at the traditional villages of Pinchollo, Maca, and Achoma. These stops offer a chance to catch a glimpse of local culture and traditions. The afternoon is then yours to explore or relax as you prefer. There are various options for local hikes, such as the Uyo Uyo pre-Inca Site in Yanque or San Antonio in Coporaque, or you can choose to visit the hot springs again for a few hours of relaxation.

Schedule:

Visit La Cruz del Condor viewpoint for condor sighting and canyon views.

Drive back to the hotel with optional stops at Pinchollo, Maca, and Achoma villages.

Approximate Trip Duration:"

And with this same prompt I have received answers such as this:

"Today’s journey begins with a scenic 160 km drive from Arequipa to the Colca Canyon. As we leave the city and ascend into the mountains, you’ll enjoy beautiful views of the Misti, Chachani, and Pichu Pichu volcanoes. Then, we’ll enter the Salinas and Aguada Blanca national reserve, where you’ll have the chance to spot llamas, alpacas, and wild vicuña herds grazing on the local pastures. After reaching the “Mirador de Los Andes” viewpoint, we’ll make our way to the Colca Valley, where our hotel is located. The afternoon is free for you to relax and enjoy the beautiful countryside or take a leisurely walk along the fields and terraces.

Schedule:
- Scenic drive from Arequipa to the Colca Canyon
- Visit the Salinas and Aguada Blanca national reserve
- Stop at “Mirador de Los Andes” viewpoint
- Free time at the hotel to relax and explore the area

Approximate Trip Duration Section:"

Which is not what I asked for. Or this:

"This morning, prepare for a 4-hour drive to Pampachiri Stone Forest, one of the most unusual and impressive landscapes in Peru. The journey will take you through the breathtaking high Andean scenery as you make your way to the small village also named Pampachiri. Upon arrival, you’ll settle into your simple yet comfortable hotel, ensuring a cozy place to rest before your exploration.
Afterward, we’ll venture out from the village, driving south along a narrow and steep valley surrounded by untamed and picturesque countryside. The rugged beauty of this remote area sets the stage for a unique adventure.
Upon reaching the Stone Forest of Pampachiri, you’ll have the opportunity to witness its extraordinary geography. Over centuries, volcanic activity has given rise to peculiar cone-like formations, ranging in size from just a few centimeters to towering heights of up to 4 meters. The stark contrast between the grey and red-hued cones against the backdrop of a brilliant blue sky and the lush green vegetation that thrives naturally in this isolated region creates a truly authentic and special atmosphere.

Schedule:
4-hour drive to Pampachiri Stone Forest
Check-in at the hotel
Exploration of the unique Pampachiri Stone Forest

Approximate Trip Duration:"

Which would be correct. The frustrating thing is that both outcomes come from the same prompt and input text. Therefore I cannot rely on chat gpt to help because it’s quality is unpredictable.

Would love to read about any suggestions, help, comment about this.

Thanks in advance

4 Likes

Hi @cino.nicholas,
Are you only using this through the standard ChatGPT interface? If so, then yes you will get varying responses.
Best to use this via API and modify your temperature or top_p settings.

If APIs are too technical, then I’d suggest using the playground instead (https://platform.openai.com/playground?mode=chat)

If you set temperature to 0 then you’ll be very close to deterministic.

Temperature:
Downside of temperature to 0 is that you lose creativity in order to gain more consistent results. I use it for my use cases, but they do not involve creative writing as I’m aiming towards 100% consistency - you might find 0.1-0.6 to suit your needs.

Instruction following:
I have also found that switching from GPT 3.5 to GPT 4 gives far better results in regards to following instructions (albeit at a much higher cost).
I have found the new preview for GPT 4 Turbo is excellent though and is much cheaper than vanilla GPT4.

More information:
Here’s some related discussions (note I haven’t read it thoroughly, but hopefully it’ll help):

Hope this helps!! good luck!!

9 Likes

I’ll have a look at this thanks. I was using temperature for my tests… I brought it down to 0,2 though not to 0. I’ll try. Thanks

Hi @cino.nicholas

Have you tried creating a GPT with Instructions, Knowledge and Actions?

I’ve had pretty good success with repeatable results.

If you add an example or two of what the end output should look like I’d be happy to try creating one and share the Instructions back with you.

2 Likes

What are you using? The 1 prompt shows different messages. It’s something that general users should know. And I never knew what caused it. Or how to fix it? Until I found comment number 2. However, I’m just a normal user. My skills in using AI in writing and documentation are very poor. But I know that if the all-in-one implementation of ChatGPT isn’t what I wanted, it’s complex and requires expertise to build and use. It might be better if we create multiple characters that are specific in their function but have a relationship with our work.

I executed this prompt three times and obtained the identical response format in each instance:

You are a methodical trip planner robot who always performs the steps below, in that respective order:

Step 1) Begin with the trip descriptive paragraphs directly, so no titles here. Write in a realistic style, avoiding superfluous adjectives or vague language, unnecessary embellishments. Only use adjectives when necessary. Do not write corny, over-the-top phrases. Craft clear, detailed, and realistic portrayals of the day’s activities, distributed across 2 or more paragraphs. Each paragraph should focus on a different event, activity, or site visit, providing clarity and sequential flow. Incorporate appropriate context to explain each event, activity, or site. Emphasize unique aspects of each trip or activity. Ensure comprehensiveness, avoiding summaries that skip details. Avoid repeating words. Paragraphs should follow one another without space in between.

Step 2) After the descriptive paragraphs, leave one free line and then write: “Schedule:”. In the line directly under “Schedule:” detail the day’s main activities of the day. Write them as sentences, one after the other. Exclude mentions of spare time or optional activities.

Step 3) After the schedule, leave a line and then write: “Approximate Trip Duration Section:”

This method of explicitly requesting steps aids the model in preserving appropriate focus at every stage.

6 Likes

Hi Cass, thanks for providing more ideas!

The temperature solution did not work for me as it simply gave me back almost the exact same text I inputed but add a few extra details specific to my prompt such as the “Schedule”.

To explain a little futher what I’m looking for I have about 1,000 descriptions of activities we offer in Peru. We been building these through the years using different styles and even different writers (some not native speakers). So I know wish to standarize certain things within all these texts so that they are of better quality for their purpose (explain travelers about our activities in a realistic, clear and detailed manner).

So I am looking for this one prompt I can apply to all of them. But so far, the prompt I have, in some cases outputs the ideal description but in some other it is not even close.

For example I have two rules that are clearly stated, write 2 to 5 paragraphs and avoid certain types of adjectives we don’t like. Regardless, sometimes GTP chooses to write one paragraph descriptions or use these adjectives we do not want to use. I mean I can understand that the quality of what is written is not ideal but I do not get why it does not follow a specific command.

Here is another version of my prompt I’ve been trying:

Create a detailed and genuine day itinerary from the provided texts. You must read very carefully the provided texts and make sure you do not miss any details.

Follow these rules:

  • Never write an initial comment before giving my what I am asking for in the prompt. Always go directly to providing me with what I ask.
  • Even if I provided you with texts that cover a trip with many days. Always present me with only one at a time so that you can be more detailed in your answer.
  • Before you start writing anything ask me for the duration of the day itinerary.
  • Avoid Repetition in Multi-Day Itineraries: When creating descriptions for multiple days within the same itinerary, ensure that each day’s description is unique and distinct. Avoid repeating similar descriptions of areas or experiences. Recognize that each day offers different aspects and opportunities, and highlight these unique qualities. This approach will provide a more diverse and engaging portrayal of the overall travel experience.
  • The itinerary must have the following sections in the following order:
    1. Title
    2. Description Section
    3. Schedule Section
    4. Approximate Trip Duration
    5. Additional Sections (If Applicable)

Description Section rules:

  1. Highlight Key Sites and Activities: Highlight key sites and activities of each day in bold. This will help in emphasizing the most significant parts of the itinerary.
  2. Detailed, Engaging, and Realistic Descriptions: Ensure the description is detailed, engaging, and realistic. It should provide a vivid portrayal of the day’s activities and experiences.
  3. Length and Detail: The description should be written in 2 to 4 paragraphs, with each paragraph thoroughly covering different aspects of the day’s activities. This ensures a comprehensive and complete description.
  4. Historical and Cultural Context: Include historical or cultural context to ensure the descriptions are comprehensive and factual. This adds depth and relevance to the itinerary.
  5. Avoid Vague and Exaggerated Language: Avoid using vague, exaggerated language, embellishments, or unrealistic descriptions. Specifically, do not use these words and phrases: “seamlessly,” “grandeur,” “serene,” “intriguing,” “nestled,” “marvel,” “stunning,” “delightful,” “breathtaking,” “amidst,” “mesmerizing,” “captivating,” “enigmatic,” “extraordinary,” “vibrant,” “iconic,” “awe-inspiring,” “enriching,” “splendor,” “fascinating,” “your canvas to paint with the colors.”
  6. No Mention of Specific Hotels or Companies: Do not mention specific hotel or company names, except in the ‘Accommodation’ section.
  7. Emphasis on Detail and Accuracy: The description section must be thorough and detailed, accurately reflecting the provided information and capturing the full scope of the day’s itinerary.
  8. Highlight Unique Experiences: Explicitly highlight unique aspects of the day’s activities, such as visiting key sites at less crowded times or special features of the experience.
  9. Adherence to Provided Information: Stick closely to the provided information without adding extra details or speculating about activities or experiences not mentioned.
  10. Avoid Overly Concise Summaries: While maintaining clarity, avoid overly concise summaries that might miss out on important details or nuances of the experience.

Schedule Section rules:

  1. Detail the day’s activities, excluding free time or optional activities.
  2. Do not highlight anything bold in this section.
  3. Use a “-” before each item in this section.
  4. Never add times of phrases like am/pm or morning/afternoon/evening. Only place the activities.
  5. All items should be placed one after the other leaving no lines in between. After the last item and the next section, leave a line.

Approximate Trip Duration rules:

  1. Do not highlight anything bold in this section.
  2. Provide a concise estimate, e.g., “Approximate Trip Duration: 1 to 2 hours.”
  3. If the duration is not provided, ask for it before presenting the full description.
  4. This section must always be formatted in this manner: “Approximate Trip Duration: XXX hours.”
  5. After this section, leave a line to separate it from the next sections.

Additional Sections (If Applicable) rules:

  1. Do not highlight anything bold in this section.
  2. Format as follows: “Meals: [Details] Type of Service: [Details] Accommodation: [Details]”
  3. Omit these sections if not provided in the text.
  4. Place these three sections one after the other without leaving lines.
  5. This is the last section, do not add any comments or extra text after this section.

Example of how the result should look like:

"Day 1 - Welcome to Peru and Cusco | Sun, 14 Apr Welcome to Peru! Your journey begins with your arrival at the airport, where we will meet you and arrange your transfer to Cusco. The city of Cusco, a significant historical landmark, was the capital of the Inca Empire and now serves as a hub where history intertwines with contemporary culture. Its streets, lined with a mix of Incan and Spanish colonial architecture, offer a unique glimpse into Peru’s past and present.

After settling into your accommodation, we will conduct a brief trip briefing to answer any queries and help you prepare for the upcoming days. The rest of the afternoon is yours to relax and acclimatize to Cusco’s altitude.

For those feeling up to a bit of exploration, Cusco’s calm streets and local markets provide an ideal setting for a gentle introduction to Peruvian culture. Alternatively, you can take this time to unwind.

Schedule:

  • Airport pickup and transfer to the hotel
  • Trip briefing at the hotel
  • Free afternoon for acclimatization

Approximate Trip Duration: 1 to 2 hours

Meals: None Type of Service: Private Accommodation: Los Portales Cusco"

Here are some examples of what I expect:
Input:

Day 1 - Arrival to Lima. Reception & Briefing at Airport hotel
Welcome to Lima! Upon Arrival at Lima’s Airport, you will be greeted by our local staff and assisted in the Lima Airport hotel located within walking distance from the arrivals gate. Once checked in, we’ll have a short briefing to go over the trip and answer any questions you may have. Staying at Lima airport is a great advantage as you will be able to get more of a rest before tomorrows morning flight.

Meals: None
Type of service: Private
Accommodation: Costa del Sol Wyndham Cusco

Output:

Day 1 - Arrival to Lima. Reception & Briefing at Airport hotel
Welcome to Lima, the vibrant capital of Peru! Upon your arrival at Lima’s Airport, our local staff will greet you and assist with your transfer to the nearby Lima Airport hotel. This convenient location, just a short walk from the arrivals gate, allows for a smooth start to your Peruvian adventure.

Once you’re settled in at the Costa del Sol Wyndham Cusco, we will conduct a short briefing. This is an opportunity to go over your trip details, answer any questions, and provide useful insights to enhance your experience in Peru. Staying right at the airport hotel not only saves travel time but also ensures you can maximize your rest, especially beneficial before your morning flight tomorrow.

This first day in Peru is designed to be relaxed and stress-free, setting a comfortable pace for the exciting journey ahead.

Schedule:
Greeting and assistance at Lima’s Airport
Check-in and briefing at Costa del Sol Wyndham Cusco

Approximate Trip Duration: 1 hour

Meals: None
Type of Service: Private
Accommodation: Costa del Sol Wyndham Cusco

Input:

Day 2 - Journey to the Sacred Valley: Maras & Moray
We’ll be heading into the Andes today. A morning flight leads us to Cusco. If possible, try to book a seat on the left side of the aircraft, as it offers the best views on bright days. Upon arriving, we’ll be expecting you at the airport, ready to take you to the Sacred Valley Incas. Because of its lower elevation (2, 870 m or 9, 415 ft) and its peaceful atmosphere, this valley is the perfect place to enjoy your first days in Cusco while you acclimatize.
Before heading to our hotel, we will make some visits on route. First, Moray, one of the most incredible Inca sites consisting of circular terraces built upon a natural depression.
After, we will stop at Unu, a restaurant with beautiful views of the mountains, and located in the Tiobamba community. We’ll enjoy a set menu of 2 courses to try the flavors of the local cuisine of Cusco.
Then, we will head to Maras, a salt-extraction complex composed of thousands of small ponds hanging on a mountain slope. If you have the energy and time, we can take a scenic downhill walk from Maras Terraces to the base of the Sacred Valley. Otherwise, we’ll head directly to your hotel in your private transfer.

Meals: B & L
Type of service: Private

Output:

Day 2 - Journey to the Sacred Valley: Maras & Moray
Embark on an exciting journey into the Andes with a morning flight to Cusco. For an enhanced experience, consider booking a left-side seat on the aircraft, offering the best views on clear days. Upon arrival in Cusco, our team will be ready at the airport to welcome you and escort you to the Sacred Valley of the Incas. At an elevation of 2,870 meters (9,415 feet), the valley’s serene environment makes it an ideal place to acclimatize and enjoy your first days in the Cusco region.

The day’s adventure begins with a visit to Moray, an astonishing Inca site characterized by its unique circular terraces set within a natural depression. This site is not just a historical wonder but also an engineering marvel, showcasing the advanced agricultural techniques of the Incas.

Following the exploration of Moray, enjoy a lunch break at Unu, a restaurant in the Tiobamba community, known for its delightful views of the mountains. Here, you’ll savor a two-course menu featuring the local cuisine of Cusco, giving you a taste of the region’s culinary traditions.

Your next stop is Maras, renowned for its intricate network of salt extraction ponds cascading down a mountainside. Depending on your energy levels and time, you have the option of taking a scenic downhill walk from the Maras terraces to the base of the Sacred Valley. Alternatively, a direct transfer to your hotel in the valley is available for a more relaxed end to the day.

Schedule:
Morning flight to Cusco
Visit to Moray
Lunch at Unu restaurant
Visit to Maras
Optional walk from Maras to the Sacred Valley or direct transfer to the hotel

Approximate Trip Duration: 6 to 7 hours

Meals: B & L
Type of Service: Private
Accommodation: Taypikala Valle Sagrado

In an ideal world It would be awesome if it where possible to apply this prompts directly in the google sheet where I store my texts but that isn’t possible yet right? Or work on GPT directly on a google doc where you can give it formating instructions too.

@cass you mention applying instructions? Is that something different to prompts?

2 Likes

ChatGPT is not for backend production, it is for chatting.

2900 characters of custom instruction though.

https://chat.openai.com/share/ad47697c-df54-4681-836e-10f26a8f628b

Very clear use case and examples. Building off of @natanael.wf 's advice I would use instructions that sound like deterministic programming language (see his words in bold).

The key for you is to break down the end output into chunks. Start really small (one paragraph you want ChatGPT to return reliably).

You may need to break your end output into 15-20 chunks that you process one at a time.

Once you get a small chunk working reliably try doing a slightly larger one, and ensure that works.

You will reach a point where suddenly the responses are not what you desire and that is your limit of size for the output.

Also you should create a GPT. This is a custom version of ChatGPT with these specific instructions and preloaded with the example outputs you desire .

More on that here - How to build your own custom ChatGPT with OpenAI's GPT builder

Keep plugging away. You will get there. Good luck.

5 Likes

If you’d like to keep using the web version of ChatGPT you can also try to split the complete task into several steps. And send the instructions one after the other with each one building on top of the previous result.
This will add manual input while saving you time to fiddle around with the API and it’s limitations and settings.

The best solution for you likely depends on how deep you want to get into the subject matter of working with LLMs. But at the same time you can use the same approach with the API as well.

Hope this helps!

4 Likes

One day I was struggling with something similar. I ditched a 2 paragrah very long system message and replaced it with “Your only role is to [do the thing I want stated in simple clear language]”

Went from 30-40 lines to 2-3 and it was like hitting the on switch. Big revelation for me.

My experience has been that the shorter the system messages the better.

If it was me things like “Paragraphs should follow one another without space in between.” I would rip out immediately. If you describe the goal well and it understands, it will do that anyway. It can easily grab on to a line like that more than you want with unexpected results.

I get it to write code and it took months to realise that ‘Hey! Write a script that does X’ works wonders, but trying to explicitly tell it what code was difficult and prone to failure. What really surprised me was that the less technical the better! Things like: “Yo, I got a bag of messy html, whip up a script to fix it please!” (literally) work really, really well. Lot’s of this machine learning stuff can be pretty counter-intuitive.

There is a ton of nuance to this stuff, and one person’s approach might not work for someone else. Get a feel for what works for you and go with it.

“Your only role is…” is hands down the most powerful thing I have stumbled across. I use it all over the place with great success.

“Your only role is to confirm that what the user shows you is valid Python.” - just that as a system message and you have a powerful code analyzer.

9 Likes

Great point. I’ve found the same thing. Constantly refining and simplifying the instructions really helps

1 Like

Thanks for the input, everyone. I have tried everything: long prompts, short to-the-point prompts, etc. It has improved, but it is still far from what we ideally want. I found that the best way is to do it by stages. So, first, you ask it to do segment X of your long prompt, wait for the answer, and then focus on segment Y of your prompt, and so on. There is an add-on I found which allows me to automate all of this on Google Sheets. It took a while to go through all my texts, but it worked. And the cool thing is that you can always add an extra prompt in a column if you need to add more rules for your texts.

4 Likes

Yea, breaking down the tasks always works better. What’s the addon you mention?

Also, in general, on your previous tries, if you had good results on the 1-2 first descriptions, and after that starts to fall-off it’s logic to happen if you continue on the same chat.

Thank you for pointing this out.

I’m working on discovering what will be possible once this is ready for production. Deploying things in it’s current state for anything other than chat is irresponsible. It will make a mistake at some point, that’s how it works, so as long as making mistakes won’t harm anything then your use case is good to go. Understanding what those mistakes are, where they come from and how to trap them is the key for me. The variation in these aberrations is too large to be able to really do anything about it for now, but like everything else the rate of improvement is amazing.

Already what is possible (proof of concept) is mind-boggling. Everything is getting better, more stable and reliable and the idea of actually using this stuff feels close.

The non-deterministic nature of the technology is so fun, you have to approach everything totally different, watching people using traditional approaches to software development with llms is a bit like watching soemone trying to train a cat. Hilarious. However, the systems I have do work, and amazingly well: most of the time The reasons for this are so numerous and varied, but it’s all getting better and better.

Once that’s sorted it’s game on. For me, It’s close.

1 Like

What I mean to point out there is that ChatGPT, the website where you can chat with an AI, is not where you would really make single-purpose projects where you take a user input, even one submitted by a form, and then produce an output. That instead is best done by API and code.

The example here shows that what is desired is some scripting, by the need to have an incomplete answer where the end is filled by other methods.

An example like this topic is more likely something that you’d want to put on your website and program with API, not a GPT you can share only for subscribers of ChatGPT Plus.

Indeed, and asking for data from an llm is not the same as an sql query or traditional api request.

You have to deal with things differently, and things change day to day and week to week. A whole little project I was working on has been rendered moot due to changes in how gpt4 repsonds in the last few weeks. The approach worked great for weeks and weeks, then it just stopped working. Dug into it and discovered it is a fundamental change in the way gpt4 responds and the approach is no longer valid/worth pursuing. As long as you bear all this in mind a lot can be done.

2 Likes

Hey, thank you, I find this very helpful and I will try it immediately after reading all comments here! I noticed the same thing, but wasn’t sure and this did approve my assumptions :slight_smile: I hit the wall with healthy recipes creation where I explained about 1000 times that I want the instructions in specific format with pasting the exact text, and then get the same wrong formatting every time :grin: It generates the recipe till one time, then suddenly like the chat just “gets dumb” :grin: Also noticed when it reaches that point, when I open New topic, things are better and it provides the output I want.

My prompting tips:

I think it’s best to say in plain language what you want it to output right up front. Instead of starting with “Begin with the trip descriptive paragraphs directly, so no titles here.” which has no context, and is quite impossible to interpret right off the bat, instead start with “You will be writing two paragraphs of text for me, for the purpose of…”

Tell it “The first paragraph will be {whatever1}, and the second paragraph will be {whatever2}”, then proceed to desribe what you meant by the terinologies in whatver1, and whatever2, after that. Every time you tell it some new constraint you should be using terminologies that tie back into something you’ve previously said, so it has exact context of how to interpret exactly what you’re asking.

2 Likes

Repetative answers and endup with this.

I can confirm GPT has lost its quality.

1 Like