Editing Endpoint Weirdness

So, there’s really two things, but they’re somewhat related.

I’m using the editing endpoint with text-davinci-edit-001 and for the most part it works really well. I’m working with an application for writers, among all sorts of magical things it does, it feeds a manuscript through the API and creates new edited versions of each scene (so the original isn’t lost because…)

I’m breaking the scene into paragraphs of HTML which is the content raw data, and the promt is just:

“Edit the following HTML without altering any HTML tags or proper names:”

It was editing proper names that weren’t um… entirely anglo… as if they were mispelled. So, that’s not super great. Anyway, small modification and now it’s fine.

Except sometimes it just spits back entirely random text. Or a single line repeated two hundred times. I’ve got the temperature set to 0.05.

For example, it was given this

paragraph

:

I opened the door to our house and a wall of smell hit me, cutting off my frantic chain of thought as I breathed in something sweet, something savory, something I couldn’t identify other than that it smelled good, some kind of smoky pepper and a melange of other things that all made my mouth immediately begin to water. “Is it just him?” I asked Harrison as he stood from the couch to meet me at the door. “Does he just leave a trail of good smells wherever he goes?”

And returned this output:

I opened the door to our house and a wall of smell hit me, cutting off my frantic chain of thought as I breathed in something sweet, something savory, something I couldn’t identify other than that it smelled good, some kind of smoky pepper and a melange of other things that all made my mouth immediately begin to water. “Is it just him?” I asked Harrison as he stood from the couch to meet me at the door. “Does he just leave a trail of good smells wherever he goes?”

“I don’t know,” Harrison said, “but I’m going to find out.” He walked past me and into the kitchen, where I heard him say, “Hey, man, what’s cooking?”

“I’m making a stew,” I heard a voice say, and I recognized it as belonging to the man who had been sitting on our couch when I left. “I hope you don’t mind.”

“Not at all,” Harrison said. “I’m just surprised.”

“I’m a man of many talents,” the man said. “I’m also a man of many appetites.”

“I’m sure you are,” Harrison said. “I’m Harrison, by the way.”

“I know,” the man said. “I’m John.”

“I know,” Harrison said. “I’m just being polite.”

“I’m not,” John said. “I’m just being honest.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that either,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

“I’m not sure I like that,” Harrison said. “I’m not sure I like that at all.”

“I’m not sure I care,” John said. “I’m not sure I care at all.”

Is there some setting I should be using that I’m not? Or is this just the weird variable nature of the AI system itself that is never going to be that much more reliable (in this iteration) than it currently is?

Here’s the call I’m making, in javascript:

async function gptHTMLCheck(text) {
  const url = "https://api.openai.com/v1/edits";

  const body = {
    model: "text-davinci-edit-001",
    input: text,
    instruction:
      "Correct any spelling or grammar mistakes in the following HTML without altering any HTML tags or proper names. NEVER add any additional text, only corrections:",
    top_p: 0.1,
  };

  const requestOptions = {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${gptAPIKey}`,
    },
    body: JSON.stringify(body),
  };

  try {
    const response = await fetch(url, requestOptions);
    const data = await response.json();
    const correctedText = data.choices[0].text;
    return correctedText;
  } catch (error) {
    console.error("Error:", error);
  }
}

Any advice? Or, you know, crushing revelations?

ETA: I updated the prompt, and switched to top_p instead of temperature. So faaaaar it seems to be more reliable? But I will have to run a few more scenes through it be sure.