Fine tuned model respond with huge amount of \t

API Bugs
1

fined tuned the gpt-4-mini-2024-07-18, then using the fine tuned model to run for batch job. It contains lots of invalid response where nothing but \t is returned. and it always return this when the same prompt in the normal model will yield correct result

for example

type or paste code here{"id": "batch_req_67737ff65e9081909fd65c3043061c01", "custom_id": "prompt-3366", "response": {"status_code": 200, "request_id": "c04c7c84e093fe8891014e0a8894f666", "body": {"id": "chatcmpl-AkO10Angi5iURdmoBPGe3p2Q1WLzx", "object": "chat.completion", "created": 1735618766, "model": "ft:gpt-4o-mini-2024-07-18:personal:experiment:Ak48cg4u", "choices": [{"index": 0, "message": {"role": "assistant", "content": "{\"index\": \"e_005ffe8b-6c65-4927-97f1-70680293b244\", \"\u4ea7\u54c1\u53e3\u5473\": {\"positive\": [\"\u597d\u5403\u4e0d\u6cb9\u817b\"], \"negative\": [], \"score\": 0.8}, \"\u98df\u54c1\u5b89\u5168\": {\"positive\": [\"\u5e72\u51c0\u536b\u751f\"], \"negative\": [], \"score\": 0.8}, \"\u56de\u5934\u5ba2\": {\"positive\": [\"\u5929\u5929\u5403\uff0c\u4e4b\u524d\u662f\u725b\u8089\u996d\uff0c\u8fd9\u6700\u8fd1\u662f\u5404\u79cd\u5473\u9053\u7684\u62cc\u9762\uff0c\u771f\u7684\u5f88\u559c\u6b22\"], \"negative\": [], \"score\": 0.8}}  \t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", "refusal": null}, "logprobs": null, "finish_reason": "length"}], "usage": {"prompt_tokens": 4715, "completion_tokens": 16384, "total_tokens": 21099, "prompt_tokens_details": {"cached_tokens": 0, "audio_tokens": 0}, "completion_tokens_details": {"reasoning_tokens": 0, "audio_tokens": 0, "accepted_prediction_tokens": 0, "rejected_prediction_tokens": 0}}, "system_fingerprint": "fp_944baebe1a"}}, "error": null}
2

First, knock down max_completion_tokens to 1000 so you aren’t paying for all that.

This is a symptom from the get-go with response_format: json_object. If the AI is trying to produce anything but JSON, you just get nonstop garbage loops. Thus why you have to still use a system or user message that says “response format: JSON”.

logit_bias can reduce tabs. Token number 197. Token 198 for newline.

What you are trying to do is to get the AI to produce its own special token stop sequence, which is how an assistant message should end. You can’t fine tune on this though, when the ideal fine-tune assistant is not just ending on one of them but 100 of them.

You can add your own fine-tune stop sequence after the JSON if not using response_format, such as “######”. Then specify that as “stop”.

The default “JSON” the AI wants to output is formatted with whitespace regardless of wishes. Fine-tuning is a solution but also may drive the JSON mode goofy.

The desired output for reference:

{
  "index": "e_005ffe8b-6c65-4927-97f1-70680293b244",
  "产品口味": {
    "positive": ["好吃不油腻"],
    "negative": [],
    "score": 0.8
  },
  "食品安全": {
    "positive": ["干净卫生"],
    "negative": [],
    "score": 0.8
  },
  "回头客": {
    "positive": ["天天吃，之前是牛肉饭，这最近是各种味道的拌面，真的很喜欢"],
    "negative": [],
    "score": 0.8
  }
}