REST APIs are not designed with LLM/context sizes in mind. Often, API endpoints return too much data and cause problems like these, clogging the context window.
If you want an imminent solution, you need reduce the response size before it gets to the model (or in this case, GPT). If you can’t modify the API implementation then one way is deploy a proxy in front of the target API and modify the response there. Then, use the proxy in your GPT.
Other approach how to use APIs with large responses in GPTs is to use superface – crate a tool that trims the API response and then use the tool in your GPT.
Of course, this only works when you don’t need all the data in the GPT. If you do need all the data from the API accessible there, I’d consider RAG or sliding window but that will take a bigger effort.
DISCLAIMER: I work at superface