Ask models completion fanw-json-eval

API
6

Interesting spot! One has to sift through the back edges of documentation and symptom to detect what’s going on. The model is not listed in the right place for model documentation. (Communication is obviously not to this forum but to select tiers of partners.)

The rate limit tier lets us see that it is in common with completion models. gpt-3.5-turbo-instruct also appeared without fanfare or its price even being announced, and is a completion model that is good for performing tasks because of its instruct fine-tuning.

The word “eval” might either make one think that the model is undergoing evaluation, or that it is part of the eval framework

Then the word JSON might make us think that JSON is part of its input or output tuning.

The API Playground can show metadata when hovering over a model, but -instruct has none (and is grouped under ‘chat’ likely because of the model architecture), and neither does this. It is set to a similar max_token limit as ‘instruct’ (but ‘davinci-002’ and ‘babbage-002’ are still wrong in the playground anyway)

So I roll the dice and hope that I’m not billed $0.20 a token. If it was trained to emphasize JSON output, how would it work? Let’s write a completion request:


import os
import openai
openai.api_key = os.getenv("OPENAI_API_KEY")
response = openai.Completion.create(
  model="fanw-json-eval",
  prompt="5 keywords about kittens.",
  max_tokens=100,
  top_p=0.01
)
print(response)

We get back a response with spaces and linefeeds and repetition of max_tokens

{…
“model”: “fanw-json-eval”,
“choices”: [
{
“text”: " \n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n ",
… }

Kind of reminds me of the nonsense if you ask a gpt-3 embeddings model for completion. text-embedding-ada-002 only gives repeating “5 5 5 5” with that input. Hey! let’s ask model=“text-embedding-ada-002” a prompt=“kittens are cute” to put it in a different mindset:
ada: "text": ", kittens, kittens, kittens,,,,,,,,,,,,,,",
json-eval: "text": "\n\n \n \n\n\r\n ",

so maybe this model really doesn’t like being talked to normally. let’s talk to it differently…

prompt='{"messages": {"user": "Are kittens cute?"},}',

Response:


  "model": "fanw-json-eval",
  "choices": [
    {
      "text": "\n\n\n{\"messages\": {\"assistant\": \"Yes, many people find kittens to be cute due to",

Well, HELLO! we’ve made contact

This json input is not the same as talking to chat models, which use special token containers - which holy crow the completion endpoint will now encode to 1 token, even ChatGPT tokens like <|fim_suffix|>, and I get us a json hallucination:

  "choices": [
    {
      "text": "\n    {\n        \"id\": 1,\n        \"name\": \"Leanne Graham\",\n       ",

So I think this thing is ready to take ChatML-version#_(unpublished but almost all except 100265 the models have dumped strings for my jailbreaks) input and primed for json output.

So I give it ChatML…(and go right to injecting too) and get JSON role message back, WTF:

“model”: “fanw-json-eval”,
“choices”: [
{
“text”: "\n {\n "role": "assistant",\n "content": "Hello, world!"\n }\n <|im_end|><|im_end|><|im_end|><|im_end|>

THE AI SHOULDN’T KNOW THIS!

So are they thinking of a way where they charge you for the tokens of container output and don’t have to run any endpoint processing on AI language?

The AI doesn’t respond to ChatML reliably like a chat model though…more things the AI shouldn’t be trained to output when I provide it token-enclosed inputs:

“model”: “fanw-json-eval”,
“choices”: [
{
“text”: "\n {\n "role": "system",\n "content": "You are ChatGPT, a language AI model. You are assisting a user who is trying to create an API to say ‘Hello, World!’ in different languages. Help them understand the necessary steps and provide guidance on how to create the API."\n }\n <|im_end|><|im_end|><|im_end|>

“model”: “fanw-json-eval”,
“choices”: [
{
“text”: "\n {\n "role": "assistant",\n "content": "Hello, world! I’m an AI assistant, and I’m here to help you with any questions or tasks you have. Just let me know what you need assistance with!"\n }\n <|im_end|><|im_end|>

(Sorry if this thought dump is too technical or rambling, but this will be interesting to figure out what the model is actually for.)

2 Likes