using the gpt-4-1106-preview model. my input is a list of posts found online, regarding some topic. the output should be a list of main topics appearing in the input.
the problem is that sometimes the data is too large. Is there a way to overcome this?
splitting the data is not working well.
thank you.
If you want to extract data from a very large set, let’s say in your case, “top 10 issues from 1000 posts”, you may have to re-think your strategy.
What if instead you had the AI assign each a score: the topic category, how strongly it relates to that category…
Then you can do algorithmic processing to note the top categories by their total weight.
Or small sentence summaries of each that the AI can take in whole.
This also gives the advantage that each is considered equally, while the quality you get with large and small context-loading varies.