Streaming Markdown or Other Formatted Text

[TimJohns]

Folks,

I am wondering if anyone has some recommended libraries or practices for rendering streamed content in real-time, as it is returned by the API, such that lists, bold, italics, etc. are rendered as such.

ChatGPT’s UX itself does this (just ask it to return a list of anything), and it’s really trivial for non-streaming content, but streaming formatted text content doesn’t seem to be that common. I have a few workarounds and prototypes, but nothing I feel is very robust.

We’re using Node on the backend and Vue3 on the front-end, so bonus for anyone with libraries in those ecosystems they can point me at, but frankly any recommended practices, libraries, or SDKs for streaming formatted text are welcome. Also happy to collaborate on an existing or new open-source project for this purpose - seems like a need exists.

Thanks!

-Tim

[l.dijkman]

not sure
i am playing with a md to html javascript lib
marked js org parser

to parse the chat stream string
only have a problem when in stream is html code

[ybt2027]

Hey @TimJohns, did you finally find the solution for this? I am facing a similar issue where the markdown formatted text won’t come along with streaming no matter what we do to the system or user prompt. And as far as I could see, there isn’t anything else we could do as of now rather than fiddling with the prompt text.

[TimJohns]

I didn’t find a truly streaming solution, but the workaround I implemented was to re-render and sanitize the entire contents of the completion on each incoming chunk. It’s a good workaround, because the code is simple and it looks great MOST of the time, but there are a couple of issues:

• If the markdown spans multiple chunks, it will render as undecorated text briefly before the next chunk comes in. This bothers ME, visually, but no users have complained about it, so my assumption is for most users it’s probably a minor annoyance that doesn’t rise to the level of complaint. We can tolerate that temporarily.

• This approach is computationally inefficient. That said, we haven’t had any complains about using too much CPU in the users’ browsers and in our own profiling, it’s immaterial. That said, the obviously inefficient implementation is a distraction in the CODE - pretty much every developer who’s looked at it has tried to ‘fix’ it with no luck so far.

• Similarly, obviously rendering partial markdown as HTML is distracting our secure coding folks. While none (so far) have found a vulnerability, we’ve spent a lot of time analyzing it. Everyone we’ve had look at it has ultimately determined they’re comfortable with it – but just like the inefficiency issue, it’s a distraction for pretty much anyone who looks at it. Parsing partial stuff is the kind of general pattern that can expose injection and over/under run vulnerabilities.

Here’s a stripped-down version of our Vue3 component. Apropos to the above commentary, v-sanitize on line 4 is a reference to vue-sanitize-directive, and (Vue-specific) summaryMarkdown is re-computed every time events.onmessage updates summary.value.

EDIT: markdown-it is also doing a lot of the heavy lifting here.

<template>
  <div class="card w-100">
    <div class="card-body w-100">
      <span v-sanitize="summaryMarkdown"></span>
    </div>
  </div>
</template>

<script setup lang="ts">
import { computed, ref, watch } from 'vue';
import MarkdownIt from "markdown-it";
import { base64Decode } from "./utils";

const props = defineProps<{
  streamUrl: string | undefined,
}>();

const summary = ref<string>('');
const isStreaming = ref();

const markdown = new MarkdownIt();
const summaryMarkdown = computed(() => {
  if (!summary.value) return;
  return markdown.render(summary.value);
})

function startStream(streamUrl: string) {   
  isStreaming.value = true; 
  summary.value = '';

  const events = new EventSource(streamUrl);

  events.onmessage = (event) => {
    summary.value = summary.value.concat(base64Decode(event.data));
  };

  events.onerror = (event) => {
    console.error(`EventSource.onerror: ${JSON.stringify(event, null, 2)}`);
  }
  events.addEventListener("control", (event) => {
    const control = JSON.parse(atob(event.data));
    console.log(`Incoming control command: ${JSON.stringify(control, null, 2)}`);
    switch (control.command) {
      case 'done':
        events.close();
        break;
      default:
        console.log('Unknown SSE command.');
    }
  })
};


watch(() => props.streamUrl, () => {
  if (props.streamUrl) {
    startStream(props.streamUrl);
  }
}, {immediate: true});

</script>

[ybt2027]

Thanks @TimJohns for the detailed response! :>

[itsvnk]

Thanks a ton for this @TimJohns

Would this be possible in just vanilla javascript? or, how can we do this by referencing vue or other utils libraries etc.?

May i request you to please provide a full example? It will help me a lot

Thanks

[isto]

As I also needed one and didn’t find anything suitable, I tried to do one by myself. If you or anyone is brave enough to test it
get in github - search for StreamMdProcessor

[redouaneboukamir]

Thanks for you contributing, it worked for me ! with a bit of arrangement.

Keep going is a great code !