Cut off response not due to token limit

This is my code.

import openai
import pandas as pd
import os


file = 'data.csv'

# Try reading the CSV file with different encodings
try:
    data = pd.read_csv(file, encoding='latin1')
except UnicodeDecodeError:
    try:
        data = pd.read_csv(file, encoding='ISO-8859-1')
    except UnicodeDecodeError:
        data = pd.read_csv(file, encoding='cp1252')

# Print the data
print(data)

# Prepare your original query
original_query = (f"This is a list of companies in the 'Industrials' classification:\n"
                  f"{data}\n"
                  "I need you to classify these companies into subgroups with the following constraints:"
                  " 1) No more than six level 1 subgroups."
                  " 2) Each level 1 subgroup can have a maximum of three level 2 subgroups."
                  " 3) Level 2 subgroups"
                  "For each company, include only the 3-letter company code."
                  "Present the output as a table with columns: Company Code, Level 1 Subgroup Name, and Level 2 Subgroup Name."
                  "Ensure every company is classified. If necessary, use your knowledge base for more information on a company."
                  "Aim for accuracy and relevance in classification, considering the primary business activities of each company.")

# Prepare the consolidation query
consolidation_query = (f"Based on the following classification of companies:\n"
                       f"{original_query}\n"
                       "Please suggest a new classification that consolidates the existing Level 2 Subgroups under each Level 1 Subgroup. The goal is to have no more than three Level 2 Subgroups for each Level 1 Subgroup, while maintaining logical grouping and industry relevance. The output should refer to my original classification for clarity. THEN PRESENT A TABLE CATEGORISING EVERY COMPANY")

# Set your API key
openai.api_key = '■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■IyBjnByu'

# Send the consolidation query to the GPT model
response = openai.ChatCompletion.create(
    model="gpt-4-1106-preview",  # or use another model version
    temperature=0.3,
    max_tokens=4000,
    seed=123,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": consolidation_query}
    ]
)

# Print the response
print(response['choices'][0]['message']['content'])


This is the output

Company Code                                Company Description
0           REH  Reece Ltd. engages in supplying plumbing, bath...
1           SVW  Seven Group Holdings Ltd. is an investment com...
2           FBU  Fletcher Building Ltd. engages in the manufact...
3           RWC  Reliance Worldwide Corp. Ltd. engages in water...
4           VNT  Ventia Services Group Ltd. is an essential inf...
..          ...                                                ...
91          MSI  Multistack International Ltd. engages in the m...
92          CFO  CFOAM Ltd. engages in development and commerci...
93          WOR  Worley Ltd. engages in the provision of servic...
94          SLX  Silex Systems Ltd. engages in the development ...
95          MRM  MMA Offshore Ltd. Engages in the provision of ...

[96 rows x 2 columns]
Given the constraints and the need to classify the companies into subgroups based on their primary business activities, we can create a classification system that reflects the diversity of the industrial sector. Here's a suggested classification:

Level 1 Subgroups:
1. Construction & Building Materials
2. Industrial Services & Equipment
3. Engineering & Infrastructure
4. Manufacturing & Distribution
5. Energy & Environmental Services
6. Transportation & Logistics

Level 2 Subgroups under each Level 1 Subgroup:
1. Construction & Building Materials
   a. Building Products
   b. Construction Services
   c. Materials & Fixtures

2. Industrial Services & Equipment
   a. Industrial Machinery
   b. Equipment Rental & Sales
   c. Maintenance & Support Services

3. Engineering & Infrastructure
   a. Engineering Services
   b. Infrastructure Operations
   c. Project Management

4. Manufacturing & Distribution
   a. Product Manufacturing
   b. Distribution Services
   c. Specialty Products

5. Energy & Environmental Services
   a. Energy Services
   b. Environmental Solutions
   c. Renewable Energy

6. Transportation & Logistics
   a. Marine Services
   b. Logistics & Supply Chain
   c. Transportation Equipment

Now, let's categorize each company into the new classification. Due to the limited information provided and the fact that I can't access real-time databases, I'll categorize the companies based on the descriptions provided and general knowledge up to my last update. Note that some companies might fit into multiple categories, but I'll assign them to the most relevant one based on the description given.

Table of Company Classification:

| Company Code | Level 1 Subgroup Name          | Level 2 Subgroup Name       |
|--------------|--------------------------------|-----------------------------|
| REH          | Construction & Building Materials | Materials & Fixtures        |
| SVW          | Industrial Services & Equipment  | Equipment Rental & Sales    |
| FBU          | Construction & Building Materials | Building Products           |
| RWC          | Manufacturing & Distribution    | Product Manufacturing       |
| VNT          | Engineering & Infrastructure    | Infrastructure Operations   |
| ...          | ...                            | ...                         |
| MSI          | Industrial Services & Equipment  | Industrial Machinery        |
| CFO          | Manufacturing & Distribution    | Specialty Products          |
| WOR          | Engineering & Infrastructure    | Engineering Services        |
| SLX          | Manufacturing & Distribution    |

So the code is almost doing everything I want it to, however it is cutting off the last table. I dont believe this is due to token limit because when I change the tokenlimit from 4000 to 1000 I still get the same thing.
When I say it cuts off the table this is what I mean. It only does the reclassification for some companies, why is this the case? I want it for ALL of them. Thanks

1 Like