GPT token error - though breaking down the code into small chunks

Hello , I am trying to upload the csv file and fetching the result via OpenAI API. My file size is 25MB. I know the token won’t work for such a big file so I broke the csv into the different chunks of 10000 token each so that the program can check for the proper answer without exceeding the usage of token and pass it to the openAI api to display the result. But still I am getting an error though it is working for the small file of 3MB. Below is the code

    [HttpPost]
    [Route("AskQuestionCsv")]
    public async Task<IActionResult> AskQuestionCsv([FromBody] string question)
    {
        if (string.IsNullOrWhiteSpace(extractedCsvText))
        {
            return BadRequest(new { Message = "No CSV content available. Please upload a CSV file first." });
        }

        if (string.IsNullOrWhiteSpace(question))
        {
            return BadRequest(new { Message = "Question cannot be empty." });
        }

        try
        {
            var openai = new OpenAIAPI("API_KEY");

            // Process the CSV text to remove repetitive patterns
            var processedCsvText = ProcessCsvText(extractedCsvText);

            // Split the processed CSV text into manageable chunks
            var chunkSize = 10000; // Adjust this size based on token limits
            var chunks = SplitTextIntoChunks(processedCsvText, chunkSize);

            var responses = new List<string>();

            foreach (var chunk in chunks)
            {
                bool retry = true;
                while (retry)
                {
                    try
                    {
                        var chatRequest = new ChatRequest
                        {
                            Model = "gpt-3.5-turbo",
                            Temperature = 0.7,
                            Messages = new List<ChatMessage>
                    {
                        new ChatMessage
                        {
                            Role = ChatMessageRole.System,
                            Content = "You are a helpful assistant."
                        },
                        new ChatMessage
                        {
                            Role = ChatMessageRole.User,
                            Content = $"Based on the following text from the CSV file, answer the question.\n\nCSV Text:\n{chunk}\n\nQuestion: {question}"
                        }
                    }
                        };

                        var chatResponse = await openai.Chat.CreateChatCompletionAsync(chatRequest);
                        var answer = chatResponse.Choices.FirstOrDefault()?.Message.Content.Trim();
                        responses.Add(answer);
                        retry = false; // If the request is successful, exit the loop
                    }
                    catch (Exception ex)
                    {
                        if (ex.Message.Contains("rate limit"))
                        {
                            // Wait for the specified time before retrying
                            await Task.Delay(1000);
                        }
                        else
                        {
                            return StatusCode(500, new { Message = "ERROR: " + ex.Message });
                        }
                    }
                }
            }

            var combinedAnswer = string.Join("\n\n", responses);

            return Ok(new { Question = question, Answer = combinedAnswer });
        }
        catch (Exception ex)
        {
            return StatusCode(500, new { Message = "ERROR: " + ex.Message });
        }
    }

    private string ProcessCsvText(string text)
    {
        // Remove or summarize repetitive patterns
        // This is a simple implementation and may need to be adjusted based on the specific content of your CSV
        var lines = text.Split(new[] { '\r', '\n' }, StringSplitOptions.RemoveEmptyEntries);
        var uniqueLines = lines.Distinct().ToList();
        return string.Join("\n", uniqueLines);
    }

    private List<string> SplitTextIntoChunks(string text, int chunkSize)
    {
        var chunks = new List<string>();
        for (int i = 0; i < text.Length; i += chunkSize)
        {
            chunks.Add(text.Substring(i, Math.Min(chunkSize, text.Length - i)));
        }
        return chunks;
    }

Error : 
{
  "message": "ERROR: Error at chat/completions (https://api.openai.com/v1/chat/completions) with HTTP status code: TooManyRequests. Content: {\n    \"error\": {\n        \"message\": \"Rate limit reached for gpt-3.5-turbo in organization org-uedxqeR1FzNcdHx3MOuawI9d on tokens per min (TPM): Limit 60000, Used 57755, Requested 2557. Please try again in 312ms. Visit https://platform.openai.com/account/rate-limits to learn more.\",\n        \"type\": \"tokens\",\n        \"param\": null,\n        \"code\": \"rate_limit_exceeded\"\n    }\n}\n"
}

It’s a rate limit error, just slow down your requests a bit more.

Can you let me know how should I resolve this?

Yes.

Sloooooooooooooow dooooooooooooooown yoooooooooooouuuuuuuuuuuuuuur reeeeeeeeeeeeeeeeequuuuuuuuuuuuuuuuuueeeeeeeeeeeeeeeeeessssssssssssssssssstsssssssssssss.

1 Like

I am new to this. How should I slow down the response? If i will just be using using gpt-3.5-turbo and no embeddings , then i will have a good amount of token then also i will face an error as the token limit exceeded. I can’t trim my data so how should i slow down the response?

The requests!

Make one request.

Wait.

Make another request.

Wait.

It seems your solution is to send many requests over and over with no rate limitations, each with part of a document.

The naive way is just to add a sleep command into your loop so it only makes a few requests a second.

You end up getting many repetitious answers of varying quality. “Check for the proper answer” then seems like a dubious operation. Especially that the key to columns and the meaning is immediately lost.

Depending on the information in the CSV file, you may instead want to offer the AI a search tool that can process from the available fields imported into code as a data object. If the file has language data that can be searched on to match the user question, you can consider an embeddings-based vector database semantic search.

Thank you for the proper explanation. I applied sleep concept and can able to load the large CSV and is working fine for me.

Thank you for giving the heads-up. It worked for me.