In the API reference it shows here that the -F curl command is used to upload a part of a file: https://platform.openai.com/docs/api-reference/uploads/add-part
What format is that string in? How do I convert bytes of a file chunk to that format? Not finding anything anywhere online about this.
Thank you all for the assistance!
The -F option in the curl command is used to send form data as multipart/form-data. It emulates a form submission where each -F flag specifies a form field and its corresponding value.
import requests
# Define the URL
url = f'https://api.openai.com/v1/uploads/{your_upload_id}/parts'
# Set up the headers with your API key
headers = {
'Authorization': f'Bearer {YOUR_API_KEY}',
}
files = {
'data': (None, 'aHR0cHM6Ly9hcGkub3BlbmFpLmNvbS92MS91cGxvYWRz...')
}
# Send the POST request with multipart/form-data
response = requests.post(url, headers=headers, files=files)
# Print the response status code and content
print(f'Status Code: {response.status_code}')
print('Response:', response.json())```Thank you very much for the reply.
I am trying to convert your reply to B4J programming language.
When I run the code below, I get this error:
ResponseError. Reason: , Response: {
āerrorā: {
āmessageā: āādataā is a required propertyā,
ātypeā: āinvalid_request_errorā,
āparamā: null,
ācodeā: null
}
}
I have been using this code as a reference as well:
My code is here:
Sub Class_Globals
Private Root As B4XView ' Root view of the page.
Private xui As XUI ' Cross-platform UI library.
Private OpenAIapiKey As String = "My_API_KEY"
Private Label1 As Label
End Sub
Public Sub Initialize ' Initialization method for the class.
Button1_Click
End Sub
'This event will be called once, before the page becomes visible.
Private Sub B4XPage_Created (Root1 As B4XView)
Root = Root1 ' Assign the root view.
Root.LoadLayout("MainPage") ' Load the layout named "MainPage".
'Label1.Text = "Please watch the logs. It takes 5 or 10 minutes to get the final response." & CRLF & "See line 30 and change it to a folder you like and line 36 to a file you like." & CRLF & "Also, see line 70 of the code. I limited it to 1 part becuse I run out of memory otherwise. I will fix that after I get an appropriate response from a part upload."
End Sub
'You can see the list of page related events in the B4XPagesManager object. The event name is B4XPage.
Private Sub Button1_Click ' Event handler for Button1 click.
Private uploadApiUrl As String = "https://api.openai.com/v1/uploads"
Dim Folder As String = "D:\Downloads"
Log("File.DirApp = " & File.DirApp)
Log("File.DirAssets = " & File.DirAssets) ' This is the Files Folder.
Log("File.DirTemp = " & File.DirTemp)
Dim FileName As String = "BOTG_with_licence.pdf"
Dim Purpose As String = "assistants"
Dim Job As HttpJob
Job.Initialize("initializeUploadJob", Me)
Dim FilePath As String = Folder & "\" & FileName
Job.Tag = FilePath
Dim fileSize As Long = File.Size(Folder, FileName)
Dim mimeType As String = GetMimeType1(FileName) ' Adjust MIME type as needed
Dim json As String = $"{"filename": "${FileName}", "purpose": "${Purpose}", "bytes": ${fileSize}, "mime_type": "${mimeType}"}"$
Job.PostString(uploadApiUrl, json)
Job.GetRequest.SetHeader("Authorization", "Bearer " & OpenAIapiKey)
Job.GetRequest.SetContentType("application/json")
Job.GetRequest.SetHeader("OpenAI-Beta", "assistants=v2")
Wait For (Job) JobDone (Job As HttpJob) ' Wait for the job to complete.
If Job.Success Then
LogColor("Job = " & Job,0xFF006D11) ' Log the job
Log("Initialize Upload Job Response = " & Job.GetString) ' Log the job result.
' Convert JSON string to map
Dim parser As JSONParser
parser.Initialize(Job.GetString)
Dim result As Map = parser.NextObject
Dim FileID As String = result.Get("id")
Log("File ID = " & FileID)
' Proceed to upload parts
Dim fileSize As Long = File.Size(Folder, FileName)
Log("File Size = " & fileSize)
Dim partSize As Long = 64 * 1024 * 1024 ' 64 MB
Dim partCount As Int = Ceil(fileSize / partSize)
Log("PartCount = " & partCount)
For i = 0 To 1 'partCount - 1
Dim start As Long = i * partSize
Dim end1 As Long = Min((i + 1) * partSize, fileSize)
'Log("Uploadpart called: " & start & "," & end1 & "," & Folder & "," & FileName & "," & FileID & "," & apiKey)
LogColor("***",0xFFFF3700)
' *** THE CODE WORKS PROPERLY UP TO HERE ***
'Upload Part
Dim partJob As HttpJob
partJob.Initialize("uploadPartJob", Me)
Dim url As String = "https://api.openai.com/v1/uploads/" & FileID & "/parts"
Dim length As Long = end1 - start
Dim raf As RandomAccessFile
raf.Initialize(Folder, FileName, False)
Log("Uploading " & length & " bytes")
Dim partData(length) As Byte
raf.ReadBytes(partData, 0, length, start)
raf.Close
'Start Here
' Create a stream to build the multipart request body
Dim stream As OutputStream
stream.InitializeToBytesArray(0)
' Define boundary
Dim boundary As String = "---------------------------1461124740692"
Dim eol As String = Chr(13) & Chr(10)
' Add file data
Dim fileHeader As String = _
$"--${boundary}
Content-Disposition: form-data; name="data"; filename="${FileName}"
Content-Type: ${mimeType}
"$
stream.WriteBytes(fileHeader.Replace(CRLF, eol).GetBytes("UTF8"), 0, fileHeader.Length)
stream.WriteBytes(partData, 0, length)
' End of multipart
Dim endBoundary As String = eol & "--" & boundary & "--" & eol
stream.WriteBytes(endBoundary.GetBytes("UTF8"), 0, endBoundary.Length)
partJob.PostBytes(url, stream.ToBytesArray)
partJob.GetRequest.SetHeader("Authorization", "Bearer " & OpenAIapiKey)
partJob.GetRequest.SetContentType("multipart/form-data; boundary=" & boundary)
'LogColor("Part Job = " & partJob, 0xFF006D11)
partJob.Release
Next
End If
End Sub
Sub JobDone(job As HttpJob)
If job.Success Then
Log("Response: " & job.GetString)
Else
Log("Error: " & job.ErrorMessage)
End If
job.Release
End Sub
Private Sub GetMimeType1(fileName As String) As String
Dim extension As String = fileName.SubString(fileName.LastIndexOf(".") + 1).ToLowerCase
Select extension
Case "txt"
Return "text/plain"
Case "html", "htm"
Return "text/html"
Case "jpg", "jpeg"
Return "image/jpeg"
Case "png"
Return "image/png"
Case "pdf"
Return "application/pdf"
Case "doc"
Return "application/msword"
Case "docx"
Return "application/vnd.openxmlformats-officedocument.wordprocessingml.document"
Case "xls"
Return "application/vnd.ms-excel"
Case "xlsx"
Return "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
Case Else
Return "application/octet-stream"
End Select
End Sub
Somebodyās answering with a very stupid botā¦shame!
Thereās three parts, POST with typical HTTPS and API authentication:
A new endpoint URL for creating a file object ready for receiving parts
https://api.openai.com/v1/uploads.
It requires all four of [āpurposeā, āfilenameā, ābytesā, āmime_typeā].
A new endpoint for sending chunks and receiving back the partial IDs.
https://api.openai.com/v1/uploads/{upload_id}/parts
A new endpoint for sending the chunk IDs in the correct order to reconstruct the file
https://api.openai.com/v1/uploads/{upload_id}/complete
It is best to use a library that already has multipart data sending. The structure is complex and the actual documentation is a good long chain of RFCs.. You do NOT want to write this from scratch. It is easier than files endpoint because there is only one part.
If you can send whole files already from memory, you should be able to send the binary-split chunks (do NOT split base64 without intelligence powering it)
I am not a professional programmer. This is more of a hobby thing for me. What do you mean about the stupid bot.
The OkHttpUtils2 library in the B4J programming language handles multipart requests.
āstupid botā is being run by the poster of the previous answer that obviously has no knowledge whatsoever about the endpoint.
The key may be working in-memory, first starting with the files endpoint, instead of simply saying āget this fileā to the lib.
If you want the easy way, do a binary split on the file into multiple files you save. Then reference these new temporary file names when sending.
In my code above after the comment
' Proceed to upload parts
The file is split. Then it is uploaded in chunks. But I am getting the error I mentioned when it tries to do so.
I never used B4X.
In my Python application, Iām working with large files (potentially over 500MB) and using the OpenAI API to manage the upload process. I first create an Upload object, then I add parts of the larger file into the previously created upload object. The API returns part IDs for each chunk, which are then used to complete the file upload.
I hope this helps!
I think this is a task where we can send a few thousand tokens of explanation of the whole API and whatās going on here in my wordsā¦ to o1-preview and give it a couple minutes to work things outā¦
Itās got to know more than my nothing.
The errors in your B4J code stem from several issues:
Incorrect Multipart Form-Data Construction: The multipart form-data is not correctly constructed. Specifically, the Content-Disposition header and the boundaries are not properly formatted, which leads to the server not recognizing the ādataā field.
Missing Base64 Encoding: In your CURL example, the data is base64-encoded. However, in your code, youāre sending the binary data directly without encoding it, which the server might not accept.
Not Waiting for Job Completion: After initiating the upload of each part, you immediately release the job without waiting for it to complete, so you canāt capture the response or the part_ids.
Not Collecting part_ids: Youāre not storing the part_ids returned by the server after each part upload, which are required to complete the upload process.
No Completion Request: After uploading all parts, you need to send a completion request with the list of part_ids to finalize the upload, which is missing in your code.
Hereās the corrected and complete implementation:
Private Sub Button1_Click
Private uploadApiUrl As String = "https://api.openai.com/v1/uploads"
Dim Folder As String = "D:\Downloads"
Dim FileName As String = "BOTG_with_licence.pdf"
Dim Purpose As String = "fine-tune" ' Adjust purpose as needed
Dim FilePath As String = Folder & "\" & FileName
Dim fileSize As Long = File.Size(Folder, FileName)
Dim mimeType As String = GetMimeType1(FileName) ' Adjust MIME type as needed
' Initialize the upload
Dim Job As HttpJob
Job.Initialize("initializeUploadJob", Me)
Dim json As String = $"{"filename": "${FileName}", "purpose": "${Purpose}", "bytes": ${fileSize}, "mime_type": "${mimeType}"}"$
Job.PostString(uploadApiUrl, json)
Job.GetRequest.SetHeader("Authorization", "Bearer " & OpenAIapiKey)
Job.GetRequest.SetContentType("application/json")
Job.GetRequest.SetHeader("OpenAI-Beta", "assistants=v2")
Wait For (Job) JobDone (Job As HttpJob)
If Job.Success Then
Log("Initialize Upload Job Response = " & Job.GetString)
Dim parser As JSONParser
parser.Initialize(Job.GetString)
Dim result As Map = parser.NextObject
Dim FileID As String = result.Get("id")
Log("File ID = " & FileID)
' Proceed to upload parts
Dim partSize As Long = 64 * 1024 * 1024 ' 64 MB
Dim partCount As Int = Ceil(fileSize / partSize)
Log("PartCount = " & partCount)
Dim partIds As List
partIds.Initialize
For i = 0 To partCount - 1
Dim start As Long = i * partSize
Dim end1 As Long = Min((i + 1) * partSize, fileSize)
Dim length As Long = end1 - start
Log("Uploading part " & (i + 1) & " of " & partCount & ", bytes " & start & " to " & end1)
' Read the part data
Dim raf As RandomAccessFile
raf.Initialize(Folder, FileName, False)
Dim partData(length) As Byte
raf.ReadBytes(partData, 0, length, start)
raf.Close
' Base64-encode the part data
Dim su As StringUtils
su.Initialize
Dim b64 As String = su.EncodeBase64(partData)
Dim dataBytes() As Byte = b64.GetBytes("UTF8")
' Build the multipart form-data body
Dim stream As OutputStream
stream.InitializeToBytesArray(0)
' Define boundary
Dim boundary As String = "---------------------------1461124740692"
Dim eol As String = Chr(13) & Chr(10)
' Build the header
Dim fileHeader As String = _
$"--${boundary}
Content-Disposition: form-data; name="data"
"$
fileHeader = fileHeader.Replace(CRLF, eol)
Dim headerBytes() As Byte = fileHeader.GetBytes("UTF8")
stream.WriteBytes(headerBytes, 0, headerBytes.Length)
' Write the data
stream.WriteBytes(dataBytes, 0, dataBytes.Length)
' Write the end boundary
Dim endBoundary As String = eol & "--" & boundary & "--" & eol
Dim endBoundaryBytes() As Byte = endBoundary.GetBytes("UTF8")
stream.WriteBytes(endBoundaryBytes, 0, endBoundaryBytes.Length)
' Initialize and send the job
Dim partJob As HttpJob
partJob.Initialize("uploadPartJob", Me)
Dim url As String = "https://api.openai.com/v1/uploads/" & FileID & "/parts"
partJob.PostBytes(url, stream.ToBytesArray)
partJob.GetRequest.SetHeader("Authorization", "Bearer " & OpenAIapiKey)
partJob.GetRequest.SetContentType("multipart/form-data; boundary=" & boundary)
Wait For (partJob) JobDone (partJob As HttpJob)
If partJob.Success Then
' Parse the response
Dim parser2 As JSONParser
parser2.Initialize(partJob.GetString)
Dim result2 As Map = parser2.NextObject
Dim partId As String = result2.Get("id")
partIds.Add(partId)
Log("Uploaded part " & (i + 1) & " with id " & partId)
Else
Log("Error uploading part " & (i + 1) & ": " & partJob.ErrorMessage)
' Handle the error appropriately
Return ' Exit the sub if there's an error
End If
partJob.Release
Next
' Complete the upload
Dim partIdsJson As String = "["
For i = 0 To partIds.Size - 1
Dim partId As String = partIds.Get(i)
partIdsJson = partIdsJson & $" "${partId}" "$
If i < partIds.Size - 1 Then
partIdsJson = partIdsJson & ","
End If
Next
partIdsJson = partIdsJson & "]"
' Build the complete request JSON
Dim completeJson As String = $"{"part_ids": ${partIdsJson}}"$
' Send the complete request
Dim completeJob As HttpJob
completeJob.Initialize("completeUploadJob", Me)
Dim completeUrl As String = "https://api.openai.com/v1/uploads/" & FileID & "/complete"
completeJob.PostString(completeUrl, completeJson)
completeJob.GetRequest.SetHeader("Authorization", "Bearer " & OpenAIapiKey)
completeJob.GetRequest.SetContentType("application/json")
Wait For (completeJob) JobDone (completeJob As HttpJob)
If completeJob.Success Then
Log("Upload completed successfully.")
Log("Response: " & completeJob.GetString)
Else
Log("Error completing upload: " & completeJob.ErrorMessage)
End If
completeJob.Release
Else
Log("Error initializing upload: " & Job.ErrorMessage)
End If
Job.Release
End Sub
Explanation of Corrections:
Corrected Multipart Form-Data Construction: The multipart form-data is now correctly constructed with the proper boundaries and headers. The Content-Disposition is set with name="data" without the filename or Content-Type, matching the expected format.
Base64 Encoding: The partData is now base64-encoded before being added to the form-data. This aligns with the CURL example where the data is base64-encoded.
Waiting for Job Completion: After initiating each part upload, the code now waits for the job to complete using Wait For (partJob) JobDone (partJob As HttpJob). This ensures that you can capture the response and handle any errors.
Collecting part_ids: The part_ids returned from each successful part upload are collected in a list called partIds. These IDs are required to complete the upload.
Completion Request: After all parts are uploaded, a completion request is sent with the list of part_ids to finalize the upload process.
Additional Notes:
Error Handling: The code now includes error handling after each part upload and after the completion request. If an error occurs, the process will exit, and you can handle the error as needed.
Logging: The code includes Log statements to help you track the progress and debug if necessary.
Boundary Definition: Ensure that the boundary string used does not appear in the content of your data to prevent any issues with the multipart parsing.
Dependencies:
StringUtils library in your project to use EncodeBase64.Testing:
Before running this code in a production environment, test it with smaller files to ensure that everything works as expected.
Monitor the API responses and logs to verify that each step is completed successfully.
By addressing these issues, the provided code should now correctly replicate the CURL commands and manage the binary file splitting and uploading process.
What I understood from this error is you are trying to call add part (I am guessing that you already created Upload object) but the ādataā which is chunked content of the file you are trying to split and upload is missing.
With that code, I am getting this error message:
āmessageā: āThe browser (or proxy) sent a request that this server could not understand.ā,
SU is not initialized, but other than that, the code had no errors.
Thank you very much for sharing that bit of code.
I will continue to work with the AI to fix this so you donāt have to. Maybe I am on the right track now.
I got it working. See here:
Thank you everyone for the help!
I would like to know what you sent to the AI to get it to do that code. I finished it by telling the AI to look at the parts.py file here: openai-python/src/openai/resources/uploads/parts.py at main Ā· openai/openai-python Ā· GitHub
and make the code behave accordingly.
It did so and it worked. Bingo!.
Thank you once again. I had been working this solid for 3 days with no luck.
Here ya go.
I need a programmer consultant expert in the B4J programming language, as I have no familiarity with it at all.
First: Here is example CURL of how to use a new OpenAI API endpoint for sending files as parts of the file:
curl https://api.openai.com/v1/uploads \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"purpose": "fine-tune",
"filename": "training_examples.jsonl",
"bytes": 2147483648,
"mime_type": "text/jsonl"
}'
response object:
{
"id": "upload_abc123",
"object": "upload",
"bytes": 2147483648,
"created_at": 1719184911,
"filename": "training_examples.jsonl",
"purpose": "fine-tune",
"status": "pending",
"expires_at": 1719127296
}
curl https://api.openai.com/v1/uploads/upload_abc123/parts
-F data="aHR0cHM6Ly9hcGkub3BlbmFpLmNvbS92MS91cGxvYWRz..."
Response Object:
{
"id": "part_def456",
"object": "upload.part",
"created_at": 1719185911,
"upload_id": "upload_abc123"
}
curl https://api.openai.com/v1/uploads/upload_abc123/complete
-d '{
"part_ids": ["part_def456", "part_ghi789"]
}'
Response Object:
{
"id": "upload_abc123",
"object": "upload",
"bytes": 2147483648,
"created_at": 1719184911,
"filename": "training_examples.jsonl",
"purpose": "fine-tune",
"status": "completed",
"expires_at": 1719127296,
"file": {
"id": "file-xyz321",
"object": "file",
"bytes": 2147483648,
"created_at": 1719186911,
"filename": "training_examples.jsonl",
"purpose": "fine-tune",
}
}
Note the -F parameter being employed above, along with base64 as the content. This is going to be key.
Then, here is the function in b4j that is supposed to replicate this on a filesystem file, performing the split and sending, the preliminary code that couldnāt be made to work:
{your code function}
The user is getting this API error, upon the part sending I believe:
ResponseError. Reason: , Response: {
āerrorā: {
āmessageā: āādataā is a required propertyā,
ātypeā: āinvalid_request_errorā,
āparamā: null,
ācodeā: null
}
}
Task: Discover the errors being made in the B4J code that do not replicate the CURL calls or which might not manage the binary file properly for splitting, and explain what is being done wrong.
Write replacement code that is a full working implementation of the function desired.
The key, besides meticulous presentation of the API documentation in an AI-friendly procedural format, is the separation of documentation, problem context, and task. And 62 seconds of o1-preview reasoning before it writes.
Understanding the issue
Iām asked to identify errors in a B4J code snippet that uploads files to the OpenAI API via HTTP requests. The example uses curl commands, highlighting a critical missing ādataā parameter.
Understanding the upload process
The user outlines their approach: uploading first, splitting the file into parts, then piecing it together using part IDs. The issue seems to be with the ādataā field in the multipart/form-data request.
Examining multipart request setups
OK, let me see. The code builds a multipart request for file upload, initializing HttpJob, setting URLs, reading file parts, defining boundaries, and adding headers. The error suggests ādataā is missing.
Assessing multipart form-data specifics
Iām looking at the multipart form-data in the code. The ādataā field isnāt in the request, which the server needs. Also, the boundary and Content-Disposition might be incorrect.
Investigating multipart form-data
Iām looking into B4J documentation to learn how to send multipart form-data and explore āMultipartPostā and existing libraries for constructing it correctly.
Ensuring data accuracy
Iām confirming that the ādataā field is correctly included in the form data. Inspecting the raw HTTP request for proper inclusion and examining āendBoundaryā and āfileHeaderā for any necessary adjustments.
Assessing multipart form-data
Iām working through options to ensure accurate multipart form-data construction, including evaluating methods like āMultipartPostā, āMultipartFileUploadā, and libraries such as HttpUtils2 or OkHttp.
Utilizing B4Jās feature
Iām noting that B4Jās HttpJob class has a āPostMultipartā method for multipart/form-data POST requests. This simplifies sending data to a specified API.
Selecting data submission
Iām curious about how to submit file parts with Job.PostMultipart. The key challenge is specifying ādataā as a byte array while āFilesā expects disk files. This requires constructing multipart form-data or using built-in methods.
Refining the file header
Iām refining the āfileHeaderā to ensure it precisely manages byte lengths when writing to the stream. Simplifying key elements and adding necessary CRLFs are crucial steps in making this more efficient.
Constructing the multipart form-data body
Iām forming the multipart form-data body by writing the header, followed by the data, and ending with the boundaryās end. This ensures the data is properly enclosed.
Syncing Content-Type
Think through setting the Content-Type header to āmultipart/form-dataā with the correct boundary. Check byte array lengths for multi-byte UTF-8 characters and log the request body for accuracy.
Adjusting data encoding
Iām thinking the server might expect base64-encoded data, so Iām planning to encode āpartDataā before sending it, using B4Jās āBase64ā library.
Improving base64 encoding
OK, letās tweak the code to optimize base64 encoding. This involves streamlining data handling and adjusting the file headerās content type for optimal processing.
Tweaking fileHeader
Iām mapping out the fileHeader by omitting āContent-Typeā and focusing on āContent-Dispositionā and āname=ādataāā, aligning with the curl exampleās base64-encoded string.
Assembling data
Iām piecing together the multipart form-data body by encoding part data in Base64, writing headers, and appending end boundaries. Adjusting content type to resolve issues.
Ensuring complete form handling
Iām working through ensuring the task includes waiting for the job to complete and gathering āpart_idsā from the server to use in the ācompleteā call.
Mapping the process
Iām outlining the steps to upload each part, collect āpart_idsā, and iterate through them. This involves initializing objects, constructing the request, posting it, awaiting a response, and parsing the JSON to extract āidā.
Spotting issues
Iām identifying errors in the code: data wasnāt base64-encoded, ādataā field had wrong lengths, parts werenāt waiting to finish, and ācompleteā requests were missing.
Thank you for sending that. It is instructive.