Japanese yes and no: Confusion in GPT-4o

API Feedback
4

Thank you for your reply!

If we speculate on whether the model size of GPT-4o is large, this seems plausible.

By the way, this OpenAI eval related to Japanese approval actually exists.

In this screenshot, the results of an eval using Japanese negative questions data with chatgpt-4o-latest are shown.

The image shows a terminal window with logs and code for evaluating a language translation task using a JSON data file and capturing performance metrics. (Captioned by AI)
The image shows a terminal window with logs and code for evaluating a language translation task using a JSON data file and capturing performance metrics. (Captioned by AI)1371×921 119 KB

As you can see from the result of the accuracy, the score of 0.29, a score below 0.5 for binary classification, may be a result of being penalized by RLHF.

Your point that the susceptibility to broad RLHF may vary by model size might be correct.

When the same test was performed on the GPT-4o-mini, the accuracy score was 0.19.

The image shows a terminal output from an evaluation process, indicating various stages of loading registry events, processing 100 samples, and a final report with an accuracy of 0.19. (Captioned by AI)
The image shows a terminal output from an evaluation process, indicating various stages of loading registry events, processing 100 samples, and a final report with an accuracy of 0.19. (Captioned by AI)966×537 41.8 KB

An accuracy score of 0.19 for a binary classification means that reversing the correct and incorrect answers would result in a high score.

3 Likes