GPT4 is unable to process (error: Unable to extract text) an uploaded file as .txt but processes the file normaly if the file extension is changed to .csv

ChatGPT Bugs
1

The file in question follows this format,
“195;Employ Aerial Weapons Not Specified Below
1951;Employ Precision-guided Aerial Munitions
1952;Employ Remotely Piloted Aerial Munitions
196;Violate Ceasefire
20;Use Unconventional Mass Violence
200;Use Unconventional Mass Violence Not Specified Below
201;Engage In Mass Expulsion
202;Engage In Mass Killings
203;Engage In Ethnic Cleansing
204;Use Weapons Of Mass Destruction Not Specified Below
2041;Use Chemical Biological Or Radiological Weapons
2042;Detonate Nuclear Weapons”.

The file is 316 lines long. I have processed it looking for invisible chrs ( ‘\u200B’, ‘\u00A0’, ‘\u200E’, ‘\u200F’, ‘\uFEFF’) etc but no luck.
Snippets from the beginning and the end of the file upload fine as .txt files.
I have tried changing the delimiter to a tab and this did not help.

Overall it seems like the general format is the issue, snippets as .txt are acceptable but the full 300 lines is seemingly confusing to gtp4. This is odd, an outline could follow a similar format of a number followed by text.

It seems like GPT is trying to interpret the file as meaningful speech and is throwing an error (Unable to extract text) over its inability to do so.

On a hunch I changed the extension to .csv and the full file uploads and processes fine. Which leads me to think that there is something weird going on in the processing of text files. There is no obvious reason not to be able to load a small .csv file as a .txt file.

2

Here is the AI trying to summarize the problem itself;

Report: Issue with Processing Structured Data in .txt Files

Overview

A user has encountered difficulties when attempting to upload structured data contained within .txt files to the platform. Despite the structured nature of the content and its small file size, the platform consistently fails to process the full .txt file, throwing an “Unable to extract text” error. Interestingly, when the file extension is changed to .csv or when only snippets of the data (e.g., the first or last ten lines) are uploaded, the platform processes the data without issue.

Details of the Issue

  • File Content: The file in question contains structured data, resembling an outline or tabulated information, using semicolons as delimiters initially. The content format is akin to key-value pairs, where each line represents a specific item or action code followed by a description (e.g., “01;Make Public Statement”).
  • Attempts to Resolve:
    • Changing the delimiter to a tab did not resolve the issue.
    • Uploading snippets of the file (first or last lines) succeeded, indicating the platform can process the content in smaller segments.
    • Renaming the file with a .csv extension allowed for successful processing, suggesting the platform’s file handling logic for .csv files accommodates the structured nature of the data more effectively.
  • Assumed Problem: The platform seems to have a limitation or a specific processing logic for .txt files that does not adequately account for structured data. This could be due to an expectation for .txt files to contain unstructured, continuous text, leading to challenges in parsing and recognizing structured outlines or lists within such files.

Potential Causes

  1. File Processing Logic: The platform may employ different processing logic for .txt files compared to .csv files, with the former not optimized for structured data.
  2. Parsing Capabilities: There might be limitations in the platform’s text extraction and parsing capabilities, particularly for recognizing and handling structured data patterns within .txt files.
3

I did some more testing and this is what I found;

Report on Structured Data Processing Issues

Introduction

This report presents an analysis aimed at diagnosing issues encountered during the upload and processing of structured data within .txt files on the platform. Detailed attention has been given to understanding how the choice of delimiter and the volume of data influence processing success.

Investigation Overview

The investigation was sparked by processing failures when uploading .txt files containing structured data. Initial indications pointed to semicolons (;) as a problematic delimiter, prompting a broader exploration into how different delimiters—specifically semicolons, tabs, colons (:), and pipes (|)—and the number of data lines impact processing.

Delimiter-Specific Findings

  1. Semicolons (;):
  • Issue Identification: It was observed that files with more than two lines of semicolon-delimited data failed to process. This issue did not present in files with two or fewer lines.
  • Further Testing: Incremental testing revealed that while one or two lines of semicolon-delimited data processed without issue, introducing a third line consistently triggered processing failures.
  1. Tab Characters:
  • Initial Success: Testing with tab characters as delimiters showed promising results, with the platform successfully processing files containing up to 59 lines of tab-delimited data.
  • Identified Threshold: A precise threshold was discovered whereby files containing 60 lines of tab-delimited data failed to process, regardless of the content. Even duplicating a successfully processed line to create the 60th line resulted in failure, underscoring that the issue was tied to the number of lines rather than specific data syntax or content.
  1. Colons (:) and Pipes (|):
  • Unproblematic Processing: Throughout the testing, no processing problems were observed with files using colons or pipes as delimiters. These delimiters consistently facilitated successful data processing across various file sizes, indicating that the issue was specific to semicolons and, under certain conditions, tab characters.

Hypotheses and Observations

  • The issue with semicolons might be attributed to their significance in programming and scripting languages, where they are often used to terminate statements. This could potentially trigger parsing or security mechanisms.
  • The failure at the specific threshold of 60 lines for tab-delimited files suggests the platform may employ conditional logic in processing, possibly involving resource allocation limits or security protocols activated based on data volume.
  • The absence of issues with colons and pipes suggests these characters do not conflict with the platform’s parsing logic or security measures, highlighting their reliability as delimiters for structured data.

Recommendations for Platform Operators

  1. In-Depth Review of Delimiter Handling: Conduct a thorough examination of how the platform’s parsing logic and security measures handle different delimiters, with particular focus on semicolons and tabs. Understanding the underlying cause of the observed issues is crucial for developing a resolution.
  2. Clarification in Documentation: Update platform documentation to explicitly state the findings regarding delimiter usage—specifically, the reliability of colons and pipes and the identified issues with semicolons and tabs. Providing clear guidelines on delimiter selection can help users avoid processing failures.
  3. Enhancement of Processing Logic: Consider enhancing the platform’s data processing logic to accommodate a broader range of delimiters without triggering failures. This could involve adjusting parsing rules or security measures that currently interpret semicolons or large volumes of tab-delimited data as problematic.
4

try replace ‘;’ to ’ '(space). that will work.