Playground Prompt length - what is too long?

Prompting
1

Opening Context:

I am working with large complex audited annual financial statements, extremely detailed Management and Discussion & Analysis, in a very specific sector. I provided 3 large documents, all pdfs. One being a 200 page introduction to the industry sector as the RAG primer. The other 2 being the aforementioned source documents, generated by the company be analyzed, married by context as well.

My Forum Question:

I asked The GPT for a re-vamped prompt that I just had used upon my request. The original prompt itself was already very successful, but GPT has recommended a huge prompt. 370 words over 19 lines. 2,800 characters. Very explicitly detailed step by step process with Indents, headings, bullet points, etc. It takes me a long time to get a run together, and my prompts have been hasty in the past.

The recommended prompt re-written by GPT is not part of this post. I have not used it yet. Of course I guess I could just try running it but, I like the human input this forum offers. I learn more.

What I would like to ask now, Is the GPT response below a hallucination or is it the truth ? Looking for more insights from the pro’s here.

User:

Is the rephrased prompt you just generated too long ? Is length a negative factor factor in prompting ?

—>> start of GPT4 response

GPT4-turbo-preview:

The length of a prompt is not inherently a negative factor; the key is clarity and specificity. A longer prompt can be beneficial if it:

  1. Clarifies the Task: Provides a detailed description of what needs to be done, ensuring a mutual understanding of the task at hand.
  2. Outlines Expectations: Clearly states the goals, desired outcomes, and specific points to be addressed, which can lead to more accurate and relevant responses.
  3. Gives Context: Offers necessary background information that informs the analysis, enhancing the quality and depth of insights.

However, there are situations where a lengthy prompt might not be ideal:

  1. Overwhelming Information: Too much detail or unnecessary information can overwhelm or confuse the AI, potentially leading to less focused responses.
  2. Efficiency in Execution: In scenarios where quick insights or answers are needed, a lengthy prompt could slow down the process.
  3. Repetitiveness: Repetition of information within the prompt can dilute the focus, making it harder to pinpoint the primary objective.

Considering these factors, a balanced approach is recommended. If the task is complex and requires deep insights, as with a detailed analysis involving multiple documents, a comprehensive prompt like the rephrased version can be very effective, providing it remains focused and avoids unnecessary repetition. For simpler tasks or when looking for quick insights, a more concise prompt may suffice.

In essence, the length should be tailored to the complexity of the task and the clarity required to achieve the desired outcome without overburdening or confusing the instruction process.

—>> end of GPT4 output

I am making good progress, asking GPT to rewrite playground run instructions, and only now looking at getting rewrites on prompts by GPT.

1 Like
2

Overall it’s advice seems somewhat acceptable, albeit with some caveats.

This is a notable “sweet spot” when it comes to prompting. Too little, and you run the risk of making a vague request that makes it easy for the LLM to provide a generalized answer. Too much, and you run the risk of losing its attention quite quickly, and stuffing its context window with junk.

Also, the “repetitiveness” thing isn’t necessarily true. It’s also unclear as to what, precisely, “overwhelms” an LLM, if anything. The biggest thing is clarity, conciseness, details, and examples.

Explicit details, step by step processes, and clear, easy to interpret formatting are typically signs of a good prompt.

That being said, gpt-4-turbo seems to be in its own little world lately, and many users have reported that what was once single-shot prompting requires a few shots now. Its larger context window seems to have been having an unintended side effect, or so it seems. Just be careful when comparing prompts and such between different model implementations. It is a non-trivial factor.

The other thing to keep in mind is that the more logic / order of operations you want it to follow in one go, the more error-prone it is likely to be. This is probably the biggest thing that requires a lot of balancing. Of course, in a perfect world, each step in a logic flow would be its own call, but that’s of course extremely inefficient. However, packing in too much, and you run the risk of not knowing where/how it went wrong, while potentially leading to unpredictable results.

I would pay less attention to overall size of a prompt, and more about the clarity of the prompt itself. Ask yourself “can I give this to a 5th grader or new intern and they would be able to follow it without asking for help, so long as they read the instructions to a T?” If you believe so, you have a quality prompt. Honestly, I remember this one article I read where it says speaking to GPT is like “asking a genie”, and honestly, they’re kind of right. The prompts that fail are typically ones where little thought was taken as to the consequences of what they were really asking for. That wisdom can be applied here quite well.

1 Like
3

Macha,

I shortened the prompt to 1600 characters, after successive efforts to elicit an acceptable response. This was the prompt I the best result so far with:

—->>

Today’s task involves an in-depth analysis of three critical .pdf documents uploaded for your retrieval, to augment your response generation. all documents are related to Nevada Exploration Inc. Efficiency and attention to detail are paramount.

Document Review Order and Objectives:

  1. Essentials of Mineral Exploration: Begin by absorbing this document to grasp the foundational concepts, technical aspects, and best practices in mineral exploration.
  2. Consolidated Financial Statements and Notes pdf Analyze these financial documents next. Focus on understanding Nevada Exploration Inc.'s financial health, including asset management, liabilities, equity changes, and cash flow operations and share issuance or dilutions.
  3. Nevada Exploration MD&A: Finally, examine the Management Discussion and Analysis pdf. Deduce the strategic outlook, operational achievements, and potential concerns from the management’s perspective. Integrate financial data insights

Expectations for Your Analysis:

  • Depth and Insight: provide nuanced interpretations of data, strategy, and corporate narratives.
  • Contextual Integration: Leverage the mineral exploration industry context to interpret financial and strategic information meaningfully.
  • Clear and Actionable Outcomes: Conclude with actionable insights that synthesize your findings from all three documents, highlighting implications for strategic decision-making.

Execution Guidance: Take the necessary time to thoroughly digest each document, step by step, using the run instructions execution framework. Efficiency includes not just speed but also the depth of comprehension and insight.

—>>

GPT4-Turbo-Preview results:

GPT4-Turbo-preview:
Consolidated Financial Statement Analysis for Nevada Exploration Inc. Overview:

Nevada Exploration Inc.'s Consolidated Financial Statements for the years ended April 30, 2023, and April 30, 2022, revealed several key financial dynamics and strategic insights into the state of the enterprise’s health and operational strategies[1].

Financial Health Assessment:

  • Revenue and Losses: The company incurred a net loss of $3,111,283 during the year ended April 30, 2023. This represents the company’s ongoing challenges in maintaining a positive net income amidst its operations and exploration activities.

  • Assets and Equity: The Consolidated Statements of Changes in Equity provide a detailed account of the equity movements within the company. The equity change overview from May 1, 2021, to April 30, 2023, offers insights into various equity-related activities—such as private placements, share issuance costs, and share-based payments. Notably, the company’s total equity has decreased from April 30, 2022, to April 30, 2023, which highlights challenges in maintaining or increasing shareholders’ value over this period[2].

  • Cash Flow and Financing: The consolidated statements of cash flows outline the financial operations, investing activities, and financing undertakings. A significant observation is the net cash used in operating activities, investable cash gained from investing activities, and cash influx through financing activities such as private placements, options exercised, and share issuance costs. The financing activities indicate an effort to counterbalance operational losses and fund exploration activities[3].

Critical Remarks and Subsequent Events:

  • Going Concern Uncertainty: The auditors highlighted a material uncertainty that may cast significant doubt on the company’s ability to continue as a going concern, particularly due to its reliance on additional fundraising to support activities in the upcoming year[4].

  • Capital Management and Funding: The company’s management of capital draws heavily on equity funding, with an emphasis on common shares and share purchase warrants as primary instruments. The reliance on equity financings to fund further exploration is accentuated by the company’s insufficient capital to support its exploration programs without additional fundraising[5].

  • Subsequent Developments: Post the financial reporting period, several subsequent events were noted, including amendments to agreements, lease terminations within the Kelly Creek project, and the completion of a significant non-brokered private placement offering for gross proceeds of $1,430,000. These events underscore ongoing strategic adjustments and efforts to secure financing for operations and exploration activities[6].

Conclusion and Strategic Insights:
Nevada Exploration Inc. is navigating through considerable financial and operational challenges, reflected in its net losses and dependence on additional funding to pursue exploration activities. The company’s financial statements reveal a concerted effort toward equity financing to mitigate operational cash flow deficits and invest in exploration prospects. Sustaining operations and achieving strategic milestones necessitate proactive capital management, strategic financing efforts, and efficient exploration expense management. Moving forward, the company should continue diversifying its financing strategies, optimizing exploration expenditures, and exploring potential strategic partnerships or agreements to strengthen its financial position and operational prospects.
[1][2][3][4][5][6]

—>>

I have learned a lot here and now recognize the importance of these things in prompt design:

Organizing the following prompt categories into the prompt framework works well and taking my time to formulate meticously, each category is paramount. The less prompts I send in a day the better, like advising to GPT take time to think about it, so should the prompt designer do the same

Task Overview:
Document Review Order and Objectives:
Expectations for Your Analysis:
Execution Guidance:

Though more details could have been discussed in the output. I can tease that out of GPT4 in other iterations. The execution was swift considering the amount and complexity of materials input.

Thanks Macha for your comments insights, it was most helpful. Ive learned a lot, and have a nice prompt framework now to work with.

2 Likes
4

@Whistler.bc.ca

Hi Bruce,

I’ve also been experimenting with large documents and trying to figure out a way to get it to review the files and provide reliable information back. To that end, I produced a GPT to provide context whether it’s hallucinating or not. If you’d like to try it, here is the link:

https://chat.openai.com/g/g-qi7hUfKEJ-hallucination-check

I’ve also developed a more advanced Diagnostics GPT which provides even more detail around what kind of information it’s using to write the response. Here is the link to that:

https://chat.openai.com/g/g-WWpY0W3jN-diagnostics-mode-for-analysis-research

In my research, I’ve found the Hallucination Check GPT to be very useful for everyday use when I’m concerned about accuracy and want a bit more speed.

With Diagnostics Mode, I can see whether GPT is interpreting my prompts correctly and pulling the right type of data back to me as I intended. With both I’ve found them be very helpful as I experiment with different large files and prompting methods.

Hoping this helps & good luck!!

All the best,
Gary

5

That’s a very smart approach to the problem Gary Thanks very much for your input. I have been having pretty good success lately with the large prompt framework, that is reusable.

I am going to check out your links

Bruce

1 Like
6

Gary,

Hmmm I also have significantly complex run instructions, even more so than my prompts, within my API / Playground tailoring my gpt environment to a specific industrial sector. So there is no way an outside GPT is going to be able to contextualize my run environment or even get close correctly interpreting my prompt.

Additionally In some cases now, my GPT4-Turbo analysis & results there are no wrong answers, just different interpretations shaped by multiple layers of contextualization.

1 Like
7

@Whistler.bc.ca, I’m wondering if you’d benefit from prompt compression? I recently released a beta test Middle/Out which will compress your prompts. I’d be curious if we could help you. We’ve built it to retain the context of the original prompt. There is a demo on the site that you can copy and paste your prompt into and compress it. Would love to know if it helps out and any feedback.

url: middleout dot wehead dot com

8

The length of a prompt isn’t a problem by itself; it’s more about how clear and focused it is. A longer prompt can actually be helpful in some cases, especially if:

  1. Clarifies the Task: It clearly explains what you need to do.
  2. Outlines Expectations: It sets clear goals and expected outcomes.
  3. Gives Context: It provides background information that can lead to better results.

However, there are times when a long prompt can be an issue:

  1. Overwhelming Information: Too much detail can confuse the AI.
  2. Efficiency: If you need quick answers, a long prompt might slow things down.
  3. Repetition: Repeating information can make it harder to focus on the main point.

Balancing Length and Clarity

  • For complex tasks, like analyzing detailed financial statements, a longer prompt can be very useful if it stays clear and avoids unnecessary repetition.
  • For simpler tasks, a shorter, more concise prompt is usually better.

Practical Tips

  1. Be Clear and Specific: Include only necessary details.
  2. Organize Information: Use headings, bullet points, and clear structure.
  3. Test and Refine: Try out different prompt lengths and structures to see what works best.

Examples

Detailed Prompt for Complex Task:

Objective: Analyze the company's financial statements and provide insights.

Documents Provided: 
- 200-page industry primer.
- Two audited financial statements.
- Management Discussion & Analysis.

Tasks:
1. Summarize key points from the industry primer.
2. Identify and analyze key financial metrics.
3. Extract and interpret critical points from the Management Discussion & Analysis.

Expected Outcome: A comprehensive report with actionable insights.

Concise Prompt for Simple Task:

Analyze the key financial metrics from the provided statements and summarize the main insights.

In essence, GPT-4’s advice is on point. The length of the prompt should fit the complexity of the task. A longer prompt can be very effective for detailed tasks, as long as it’s well-structured and doesn’t repeat itself unnecessarily. For simpler tasks, keep it short and to the point.