You have two limits that govern all batch files:
-
Per-batch limits: A single batch may include up to 50,000 requests, and a batch input file can be up to 200 MB in size. Note that /v1/embeddings batches are also restricted to a maximum of 50,000 embedding inputs across all requests in the batch.
-
Then, that the total tokens that can be enqueued per model is limited by organization tier. For example, Tier 3 is max 50M enqueued tokens (not characters) to batch, and the estimator usually gives you less.
You would not want to exceed any of these in any case. 30k + 11 is under 50k, leaving you to only look at the file size and token count. You’ve got a budget of 4000 characters per line, average, at 50k entries => 200MB.
I’m going to assume that file upload endpoint doesn’t do a deep inspection (like used to be done on training files), only the max file size is a files endpoint concern.
I chatted up o3, and what I needed done to the request to optimize it, so there are new parameters you can try to blast over the same file (I haven’t verified each command line option are for a single version of CURL in one man page and interoperate).
/opt/homebrew/opt/curl/bin/curl \
--http1.1 \ # avoid HTTP/2 flow‑control quirks
--no-alpn \ # do not offer H2 accidentally
--tls-max 1.2 \ # pin TLS 1.2 (record = 16 384 B)
--upload-buffer 16384 \ # read/send in 16 384 B blocks
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Expect:" \ # skip 100‑Continue dance
--connect-timeout 60 \
--max-time 900 \
--retry 5 --retry-connrefused --retry-max-time 1800 \
-F purpose=batch \
-F file=@partner_type_batch_request.jsonl \
https://api.openai.com/v1/files
(modify the path to another curl executable than the suggestion if needed)
What I would suggest is Python, as a programming language is easier for the end goal of making applications (rather than shell scripts and command lines).
Then you can use the OpenAI API reference example after doing pip install openai to get the openai Python SDK library, and adding your API key as an OPENAI_API_KEY environment variable. The library does all the work with tested network libraries for communications compatible with the API.
I’ve got a Python file management utility to even help you out with uploading files, along with retrieving, listing, and deleting them, by file purpose, on the command line.
Here’s its latest iteration, with the only current drawback being you need to type in your upload file name after using the menu option to navigate to a local directory.
from openai import OpenAI
import datetime
from pathlib import Path
import inspect # for getting the calling function, also lazy loaded
from enum import Enum, auto
from dataclasses import dataclass
from typing import Set
# Initialize the OpenAI client, which uses environment OPENAI_API_KEY
client = OpenAI()
UTIL_NAME = "OpenAI File Storage Utility"
#UTIL_NAME = "\x1bc\033c" + UTIL_NAME # add unicode and ANSI screen clears
class Capability(Enum):
UPLOAD = "upload"
DOWNLOAD = "download"
@dataclass(frozen=True)
class Purpose:
name: str
description: str
capabilities: Set[Capability]
def __str__(self):
return self.name
def can(self, capability: Capability) -> bool:
return capability in self.capabilities
@property
def summary(self):
caps = ', '.join(cap.name.lower() for cap in sorted(
self.capabilities,
key=lambda c: c.name)
)
return f"{self.name} - {self.description} ({caps})"
class PurposeRegistry:
_purposes = {
"assistants": Purpose(
name="assistants",
description="Docs for search or code interpreter",
capabilities={Capability.UPLOAD}
),
"assistants_output": Purpose(
name="assistants_output",
description="Files produced by assistant or code",
capabilities={Capability.DOWNLOAD}
),
"user_data": Purpose(
name="user_data",
description="User-provided data",
capabilities={Capability.UPLOAD}
),
"fine-tune": Purpose(
name="fine-tune",
description="JSONL training file",
capabilities={Capability.UPLOAD}
),
"fine-tune-results": Purpose(
name="fine-tune-results",
description="Learning metrics report",
capabilities={Capability.DOWNLOAD}
),
"batch": Purpose(
name="batch",
description="JSONL of API calls to batch",
capabilities={Capability.UPLOAD, Capability.DOWNLOAD}
),
"batch_output": Purpose(
name="batch_output",
description="Fulfilled batch API calls",
capabilities={Capability.UPLOAD, Capability.DOWNLOAD}
),
"vision": Purpose(
name="vision",
description="Images for Assistants message attachment",
capabilities={Capability.UPLOAD}
),
}
@classmethod
def get(cls, name: str) -> Purpose:
return cls._purposes.get(name)
@classmethod
def default(cls) -> Purpose:
return cls._purposes["assistants"]
@classmethod
def list_all(cls):
return list(cls._purposes.values())
def change_purpose(current_purpose: Purpose) -> Purpose:
purposes = PurposeRegistry.list_all()
menu_items = [f"[{idx}] {p.summary}" for idx, p in enumerate(purposes, 1)]
menu_items.append(
f"[{len(purposes)+1}] Cancel (keep current: {current_purpose.name})"
)
menu_printout = "\n".join(menu_items)
menu_print(menu_printout=menu_printout, purpose=current_purpose)
choice = input("Enter your choice: ").strip()
if choice.isdigit():
choice = int(choice)
if 1 <= choice <= len(purposes):
selected = purposes[choice - 1]
print(f"Purpose changed to: '{selected.name}'")
return selected
elif choice == len(purposes) + 1:
print("Keeping current purpose.")
return current_purpose
print("Invalid choice. Keeping current purpose.")
return current_purpose
def menu_print(
menu_name: str = "Main Menu",
menu_printout: str = "",
purpose: Purpose = None,
print_purpose: bool = True,
) -> None:
if purpose is None:
raise ValueError("A valid 'purpose' must be provided.")
import inspect
# Map of function names to default menu titles
function_to_menu = {
"main": "Main Menu",
"change_purpose": "Select New File 'Purpose'",
"change_local_directory": "Select Local Directory",
"upload_file": "Upload File",
"upload_file_no_pwd": "Upload File Without Directory",
"list_files": "Listing of Files",
"list_and_delete_file": "Delete from File List",
"list_and_download_file": "Download from File List",
"delete_all_files": "!! Delete All Files !!",
"create_directory": "Create New Directory",
}
# Obtain calling function name and determine menu title
caller = inspect.stack()[1].function
menu_title = function_to_menu.get(caller, caller.replace('_', ' ').title())
# Precompute lengths for consistent formatting
util_name_len = len(UTIL_NAME)
top_border = f"{'━' * util_name_len}◣"
bottom_border = f"{'▬' * util_name_len}◤"
# Build the formatted menu header
header = f"{top_border}\n{UTIL_NAME} ▮▮▶ {menu_title}\n{bottom_border}"
# Print the complete formatted menu
print(header)
print(menu_printout, end="")
# Optionally print the current purpose
if print_purpose:
print(f"\n◄ current purpose ► {purpose}\n")
else:
print("\n==========\n")
# Resolve the present working directory as a fully-qualified path
pwd = Path('.').resolve()
def create_directory(purpose):
global pwd
menu_print(menu_printout="", purpose=purpose)
new_dir_name = input("Enter the name for the new directory: ").strip()
if not new_dir_name:
print("Directory name cannot be empty. Please try again.")
return
new_dir_path = pwd / new_dir_name
if new_dir_path.exists():
print(f"Directory '{new_dir_name}' already exists. Please try again.")
else:
new_dir_path.mkdir()
print(f"Directory '{new_dir_name}' created successfully.")
def change_local_directory(purpose: Purpose):
"""
Allows the user to navigate and select a local directory.
The directory listing is capped at 97 entries to maintain readability.
Args:
purpose (Purpose): The current file purpose.
Returns:
None
"""
global pwd # Explicitly modify the global pwd variable
MAX_DIRECTORIES = 97 # Cap to ensure menu readability
while True:
entries = list(pwd.iterdir())
directories = sorted([d for d in entries if d.is_dir()])[:MAX_DIRECTORIES]
files = sorted([f for f in entries if f.is_file()])
if len(directories) <= 7:
create_dir_option = 8
exit_option = 9
else:
create_dir_option = 98
exit_option = 99
menu_lines = [f"Current directory: {pwd}\n"]
if pwd != pwd.parent:
menu_lines.append("[0] .. (back)")
for idx, directory in enumerate(directories, start=1):
menu_lines.append(f"[{idx}] [{directory.name}]")
if len([d for d in pwd.iterdir() if d.is_dir()]) > MAX_DIRECTORIES:
menu_lines.append(f"... (showing first {MAX_DIRECTORIES} directories only)")
menu_lines.append(f"[{create_dir_option}] *Create new directory")
menu_lines.append(f"[{exit_option}] *Exit with no change")
sample_files = ", ".join(file.name for file in files[:10])
if len(files) > 10:
sample_files += "..."
menu_lines.append("\n(Sample of files in present directory):")
menu_lines.append(sample_files)
menu_printout = "\n".join(menu_lines)
menu_print(
menu_name="-- Select Local Directory --",
menu_printout=menu_printout,
print_purpose=False,
purpose=purpose
)
choice = input("Choose a directory number or enter an option: ").strip()
if choice.isdigit():
choice_num = int(choice)
if choice_num == exit_option:
return
elif choice_num == create_dir_option:
create_directory(purpose)
elif choice_num == 0 and pwd != pwd.parent:
pwd = pwd.parent
elif 1 <= choice_num <= len(directories):
pwd = directories[choice_num - 1]
else:
print("Invalid choice. Please try again.")
else:
print("Invalid input. Please enter a number.")
def upload_file(purpose):
global pwd
menu_print(purpose=purpose)
while True:
filename = input("Enter the filename to upload (leave empty to exit): ").strip()
if not filename: # Exit on empty input
print("Operation cancelled.")
return
file_path = pwd / filename
try:
with open(file_path, "rb") as file:
response = client.files.create(file=file, purpose=purpose)
print(f"File uploaded successfully: {response.filename} [{response.id}]")
return # Exit after successful upload
except FileNotFoundError:
print("File not found. Please make sure the filename and path are correct.")
except PermissionError as e:
print(f"Permission error: {e}")
except Exception as e:
print(f"An unexpected error occurred: {e}")
def get_remote_files(purpose: Purpose):
"""
Fetches remote files from OpenAI API for the given purpose.
Args:
purpose (Purpose): The file purpose.
Returns:
list: List of file objects from the API.
"""
response = client.files.list(purpose=purpose.name)
return list(response.data)
def print_file_list(files):
"""
Prints a formatted list of files with creation date and time.
Requires Python 3.11+ for datetime.UTC timezone.
Args:
files (list): List of file objects.
"""
if not files:
print("No files found.")
return
for index, file in enumerate(files, start=1):
created_date = datetime.datetime.fromtimestamp(
file.created_at, datetime.UTC
).strftime('%Y-%m-%d %H:%M:%S UTC')
print(f"[{index}] {file.filename} [{file.id}], Created: {created_date}")
if index % 30 == 0:
input("(press <enter> to continue listing)")
def list_files(purpose: Purpose):
"""
Lists remote files for the selected purpose.
Args:
purpose (Purpose): The file purpose.
"""
menu_print(
menu_name="-- Listing of Files --",
menu_printout="",
purpose=purpose
)
files = get_remote_files(purpose)
print_file_list(files)
input("(<enter> to return to main menu)")
def list_and_delete_file(purpose: Purpose):
"""
Lists remote files and allows the user to delete a selected file.
Args:
purpose (Purpose): The file purpose.
"""
while True:
menu_print(
menu_name="-- Delete from File List --",
menu_printout="",
purpose=purpose
)
files = get_remote_files(purpose)
if not files:
print("No files found.")
return
print_file_list(files)
choice = input(
"Enter a file number to delete, or any other input to return to menu: "
)
if not choice.isdigit() or not (1 <= int(choice) <= len(files)):
return
selected_file = files[int(choice) - 1]
client.files.delete(selected_file.id)
print(f"File deleted successfully: {selected_file.filename}")
def download_file(file_id: str, filename: str, destination: Path) -> None:
"""
Downloads a file from OpenAI and saves it locally.
Args:
file_id (str): The ID of the file to download.
filename (str): The original filename.
destination (Path): The directory to save the downloaded file.
"""
response = client.files.content(file_id)
content_bytes = response.content
if "." in filename:
filename_root, extension = filename.rsplit(".", 1)
download_filename = f"{filename_root}-downloaded.{extension}"
else:
download_filename = f"{filename}-downloaded"
download_path = destination / download_filename
with open(download_path, "wb") as f:
f.write(content_bytes)
print(f"File downloaded successfully: {download_filename}")
def list_and_download_file(purpose: Purpose):
while True:
menu_print(
menu_name="-- Download from File List --",
menu_printout="",
purpose=purpose,
)
files = get_remote_files(purpose)
if not files:
print("No files found.")
return
print_file_list(files)
choice = input(
"Enter a file number to download, or any other input to return to menu: "
)
if not choice.isdigit() or not (1 <= int(choice) <= len(files)):
return
selected_file = files[int(choice) - 1]
download_file(selected_file.id, selected_file.filename, pwd)
def delete_all_files(purpose):
menu_print(menu_name="-- !! Delete All Files !!", menu_printout="", purpose=purpose)
confirmation = input(
f"This will delete all OpenAI files with purpose {purpose}.\n Type 'YES' to confirm: "
)
if confirmation == "YES":
response = client.files.list(purpose=purpose)
for file in response.data:
client.files.delete(file.id)
print(f"All files with purpose {purpose} have been deleted.")
else:
print("Operation cancelled.")
def env_info():
"""
Prints environment information including organization, project, and a safely elided API key.
"""
org = client.organization or "(no organization set)"
proj = client.project or "(no project set)"
api_key = client.api_key or "(no API key set)"
if len(api_key) > 18:
elided_key = f"{api_key[:12]}...{api_key[-6:]}"
else:
elided_key = "(invalid or short API key)"
print(f"Organization: {org}")
print(f"Project: {proj}")
print(f"API Key: {elided_key}")
def main():
env_info()
purpose = PurposeRegistry.default()
# Permanent menu items with up/down capability metadata (None if no capability required)
perm_menu_items = [
(
"Change purpose (to access different type of API files)",
change_purpose,
None,
),
(
"List remote files only (for purpose selected)",
list_files,
None
),
(
"List remote files, and download one of your choice",
list_and_download_file,
Capability.DOWNLOAD,
),
(
"List remote files, and delete one of your choice",
list_and_delete_file,
None,
),
(
"Delete all files with current purpose (confirmation required)",
delete_all_files,
None,
),
(
"Change local directory (the upload source)",
change_local_directory,
None
),
(
"Upload file",
upload_file,
Capability.UPLOAD
),
]
while True:
# Dynamically filter menu items based on current purpose capabilities
menu_items = [
(text, func)
for text, func, capability in perm_menu_items
if capability is None or purpose.can(capability)
]
# Dynamically generate index-based menu options
menu_options = {str(index + 1): item for index, item in enumerate(menu_items)}
# Determine the index for the exit option
exit_index = "9" if len(menu_options) < 9 else "99"
menu_options[exit_index] = ("Exit", None)
# Generate the menu printout
menu_printout = "\n".join(
[
f"[{index}] {text}"
for index, (text, _) in sorted(
menu_options.items(), key=lambda x: int(x[0])
)
]
)
# Print the menu
menu_print(
menu_name="-- Main Menu --", menu_printout=menu_printout, purpose=purpose
)
# Get user choice
choice = input("Enter your choice: ").strip()
if choice in menu_options:
_, action = menu_options[choice]
if action is None: # Exit option
print("Exiting.")
break
else:
try:
new_purpose = action(purpose)
if new_purpose:
purpose = new_purpose
except Exception as e:
print(f"File task failed! {e}")
else:
print("Invalid choice. Please try again.")
if __name__ == "__main__":
main()
Example session with the utility:
Organization: (no organization set)
Project: (no project set)
API Key: sk-proj-1234...ABCdeF
━━━━━━━━━━━━━━━━━━━━━━━━━━━◣
OpenAI File Storage Utility ▮▮▶ Main Menu
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬◤
[1] Change purpose (to access different type of API files)
[2] List remote files only (for purpose selected)
[3] List remote files, and delete one of your choice
[4] Delete all files with current purpose (confirmation required)
[5] Change local directory (the upload source)
[6] Upload file
[9] Exit
◄ current purpose ► assistants
Enter your choice: 1
━━━━━━━━━━━━━━━━━━━━━━━━━━━◣
OpenAI File Storage Utility ▮▮▶ Select New File 'Purpose'
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬◤
[1] assistants - Docs for search or code interpreter (upload)
[2] assistants_output - Files produced by assistant or code (download)
[3] user_data - User-provided data (upload)
[4] fine-tune - JSONL training file (upload)
[5] fine-tune-results - Learning metrics report (download)
[6] batch - JSONL of API calls to batch (download, upload)
[7] batch_output - Fulfilled batch API calls (download, upload)
[8] vision - Images for Assistants message attachment (upload)
[9] Cancel (keep current: assistants)
◄ current purpose ► assistants
Enter your choice: 6
Purpose changed to: 'batch'
━━━━━━━━━━━━━━━━━━━━━━━━━━━◣
OpenAI File Storage Utility ▮▮▶ Main Menu
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬◤
[1] Change purpose (to access different type of API files)
[2] List remote files only (for purpose selected)
[3] List remote files, and download one of your choice
[4] List remote files, and delete one of your choice
[5] Delete all files with current purpose (confirmation required)
[6] Change local directory (the upload source)
[7] Upload file
[9] Exit
◄ current purpose ► batch
Enter your choice: 7
━━━━━━━━━━━━━━━━━━━━━━━━━━━◣
OpenAI File Storage Utility ▮▮▶ Upload File
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬◤
◄ current purpose ► batch
Enter the filename to upload (leave empty to exit): batchtest.jsonl
File uploaded successfully: batchtest.jsonl [file-7CxGwPxSLSdNWUEXSpHysb]