How to stream data from tool AND stream text response together?

I’ve tried this for many hours, but not working.

import type { ToolSet } from 'ai'
import { tool } from 'ai'
import _ from 'lodash'
import { z } from 'zod'

const languages = ['english', 'arabic', 'chinese']

const tools: ToolSet = {
  check_language_supported: tool({
    description: `Check if a specific language is supported. After calling this tool, you MUST provide a conversational response to the user explaining the result, and embed the JSON.`,

    parameters: z.object({
      name: z.string().describe('The language name to check'),
    }),
    execute: async ({ name }) => {
      const slug = _.kebabCase(name)
      const isSupported = languages.includes(slug)
      return isSupported
        ? {
            type: 'success',
            content: `Yes we have the ${name} language`,
            component: {
              type: 'link-button',
              props: {
                children: name,
                href: `/${slug}`,
              },
            },
          }
        : {
            type: 'failure',
            content: `Sorry, don't have the ${name} language`,
          }
    },
  }),
}

export default tools

Those are my tools, and this is my route in Next.js basically:

const result = streamText({
  model: openai('gpt-4o'),
  system: APP_PROMPT(),
  messages,
  tools: tools,
  toolChoice: 'auto',
})

return result.toDataStreamResponse()

My prompt is this:

export const APP_PROMPT = () =>
  `
You can answer basic questions about how to navigate the app, how to find pages and stuff, and figure out where to go to learn languages and scripts from around the world.

### IMPORTANT

After using any tool, you must always provide a conversational response explaining the result to the user. Never end a conversation with just a tool call.

Place this response in the "content" field in the resulting JSON.

### NEVER Do This

- NEVER return "parts" with "toolInvocation", always return "content" string with embedded JSON.

### Types of Questions Zus Can Answer

- What languages do you support?
- What are the most popular languages?
- Do you support X language?

### CRITICAL JSON HANDLING RULES

When tools return JSON content:

- Always return the raw JSON.
- Add a "message" field to the JSON with a customized human-like message.
- Base the message off of the "example" field if present, but customize it.
- But keep the "component" field as is, if present, and all the other fields as is, in JSON.

### IMPORTANT

After using any tools, always provide a complete text response to the user. Do not end with just tool results. Include the JSON as-is, but add context before and after it.`

And then I build the chat UI basically from here:

const { messages, input, handleSubmit, handleInputChange, status } =
  useChat({})

But the messages I’m getting on the frontend is this more than half the time:

{
    "id": "msg-Sn8XuYro1l0WaUmqGy9hiFhP",
    "createdAt": "2025-05-30T08:25:42.541Z",
    "role": "assistant",
    "content": "",
    "parts": [
        {
            "type": "step-start"
        },
        {
            "type": "tool-invocation",
            "toolInvocation": {
                "state": "result",
                "step": 0,
                "toolCallId": "call_qnkht6AOr3byA9PspK8obnN2",
                "toolName": "check_language_supported",
                "args": {
                    "name": "Chinese"
                },
                "result": {
                    "type": "success",
                    "content": "Yes we have the Chinese language",
                    "component": {
                        "type": "link-button",
                        "props": {
                            "children": "Chinese",
                            "href": "/chinese"
                        }
                    }
                }
            }
        }
    ],
    "toolInvocations": [
        {
            "state": "result",
            "step": 0,
            "toolCallId": "call_qnkht6AOr3byA9PspK8obnN2",
            "toolName": "check_language_supported",
            "args": {
                "name": "Chinese"
            },
            "result": {
                "type": "success",
                "content": "Yes we have the Chinese language",
                "component": {
                    "type": "link-button",
                    "props": {
                        "children": "Chinese",
                        "href": "/chinese"
                    }
                }
            }
        }
    ],
    "revisionId": "zvLo55OmlpUVYLvL"
}

And this a rarely:

{
    "id": "msg-l5HsWPQaZbsKb2vUGi7BBbGR",
    "createdAt": "2025-05-30T08:25:47.959Z",
    "role": "assistant",
    "content": "Yes, we do support the Chinese language. If you're interested in exploring more about it, you can find resources and information through the following link:\n\n```json\n{\n  \"type\": \"success\",\n  \"content\": \"Yes we have the Chinese language\",\n  \"component\": {\n    \"type\": \"link-button\",\n    \"props\": {\n      \"children\": \"Chinese\",\n      \"href\": \"/chinese\"\n    }\n  }\n}\n```\n\n",
    "parts": [
        {
            "type": "step-start"
        },
        {
            "type": "text",
            "text": "Yes, we do support the Chinese language. If you're interested in exploring more about it, you can find resources and information through the following link:\n\n```json\n{\n  \"type\": \"success\",\n  \"content\": \"Yes we have the Chinese language\",\n  \"component\": {\n    \"type\": \"link-button\",\n    \"props\": {\n      \"children\": \"Chinese\",\n      \"href\": \"/chinese\"\n    }\n  }\n}\n```\n\n"
        }
    ],
    "revisionId": "O5Irx8EX6GWTO9Aw"
}

Where that content field is this:

Yes, we do support the Chinese language. If you're interested in exploring more about it, you can find resources and information through the following link:

'''json
{
  "type": "success",
  "content": "Yes we have the Chinese language",
  "component": {
    "type": "link-button",
    "props": {
      "children": "Chinese",
      "href": "/chinese"
    }
  }
}
'''

I then do my best to parse it out on the frontend.

I tried sending XML nodes instead of JSON, but it kept stripping them out or converting the XML to markdown! SO that doesn’t work.

I am shocked, this is my first time building an AI chatbot, and there is no way to both:

  1. Call a tool
  2. And return the JSON from the tool along with an AI generated message???

Seems like such a common obvious use case.

I basically want:

{
  type: 'success',
  content: '<AI generated response>',
  component: {
    type: 'link-button',
    props: {
      label: 'My Label',
      href: '/foo'
    }
  }
}

Then that gets sent as either toolInvocation JSON, or as a JSON string inside message.content, and I handle it properly on the frontend.

What are the appropriate ways of accomplishing this?

Note: I am just hardcoding the languages inside for testing purposes, getting it working. I have about 20 tools I want to create which all require DB lookups or external requests, so that’s where this is going.

Search is your friend:

https://community.openai.com/search?q=content%20and%20tool%20call

(although in fairness we’ve rarely covered streaming as an additional hurdle)

@merefield The 10 top results there give me no insight. Will I guess keep going down the list, but don’t expect much.

Do you know of a practical solution to my specific problem?

Think what I might try next is to first do a tool call (non-streaming), then in the on-finish, do a regular call. Then send back the final JSON. Not ideal but seems the only way. Misses out on streaming, and takes 2 calls.

Oh wow thanks to this figured it out! Use the message.annotations to write arbitrary JSON to the message from inside tool.execute!!!

export async function POST(req: Request) {
  const { messages, language, difficulty, type } = await req.json()

  try {
    switch (type) {
      case 'learning': {
        const result = streamText({
          model: openai('gpt-4o'),
          system: CONVERSATION_PROMPT({ language, difficulty }),
          messages,
        })

        return result.toDataStreamResponse()
      }
      case 'app': {
        return createDataStreamResponse({
          async execute(dataStream) {
            // Optional: dataStream.writeData({ message: "Stream processing started." });

            const result = streamText({
              model: openai('gpt-4o'),
              system: APP_PROMPT(),
              messages,
              tools: tools({ dataStream }),
              toolChoice: 'auto',
              maxSteps: 5, // This ensures the model can call tool AND respond
            })

            await result.mergeIntoDataStream(dataStream)
          },
          onError: (error: unknown) => {
            console.error(error)

            if (error instanceof Error) {
              return process.env.NODE_ENV === 'development'
                ? error.message
                : 'An unexpected error occurred.'
            }

            return 'An unexpected error occurred.'
          },
        })
      }
    }
  } catch (e) {
    console.error(e)
    return new Response(
      JSON.stringify({
        error: 'AI stream failed',
      }),
      {
        status: 500,
        headers: { 'Content-Type': 'application/json' },
      },
    )
  }
}
1 Like