Improve LLM output so it conforms to Slack formatting

Hi there,
I’m playing around with an assistant you interact with via Slack.

I want to LLM to conform to slack formatting principles… (google slack formatting)

I’m using gtp4-turbo-preview and whilst it is sometimes aware of slack formatting it sometimes goes rouge and does its own thing.

For example instead of using a single astricks to make something bold it will sometimes do a double one like this.

I’m trying to work out the best way to improve this. Should I just be filling my instructions/prompt with a bunch of formatting examples? Or is there some way I can train the model to be better at this?

Thanks!

1 Like

Welcome to the forum.

It outputs markdown fairly well, but not sure about Slack formatting… Is there another phrase that could be used to describe it? An example or two might help.

Another option would be to translate the markdown to Slack-down ™?

Thanks for the quick response Paul!

Here is an example of where it doesn’t get it right

My prompt includes this…

You can format your messages for Slack. If you want to make text bold by Surround your text with a pair of single asterisks, like *this*.

If you want to include links the format is <URL|link text>, for example, <http://www.example.com|Click Here>.

Keep your interactions with the user brief as you will be communicating through Slack instant messages. Don't ask the user unnecessary questions.

image

1 Like

I’d try being a bit more specific?

Slack Formatting Guide:

To format messages in Slack, use the following “Slack formatting” conventions. These instructions are tailored specifically for enhancing readability and organization within Slack’s desktop and mobile applications:

  • Bold Text: To make text bold, enclose it with asterisks (*). Example: *text* becomes bold.

  • Italic Text: To italicize text, enclose it with underscores (_). Example: _text_ becomes italic.

  • Strikethrough Text: To apply a strikethrough effect, enclose text with tildes (~). Example: ~text~ becomes strikethrough.

  • Inline Code: For inline code formatting, enclose text with backticks (```). Example: text becomes code.

  • Blockquote: To format text as a blockquote, start it with an angled bracket (>). Example: >text.

  • Code Block: To create a code block, enclose text with triple backticks (`````).

  • Ordered List: For an ordered list, start each item with a number followed by a period. Example: 1. text.

  • Bulleted List: For a bulleted list, start each item with an asterisk and a space. Example: * text.

These Slack-specific formatting commands enhance your messaging by adding visual distinctions to your text, making your Slack communications more effective and engaging.

Can’t hurt to try! Your temp and other settings might be messing it up a bit too.

Thats great, thanks for your help I will give it a go :+1:

1 Like

No problem.

Let us know. I’m curious about this one. If not, we can try to tackle another way maybe…

Hi Paul,
Still having a few problems as it keeps going back to double astricks. I played around with it in the chat today and got some interesting results.

See this comment from the LLM…

“However, please note, in Slack formatting, single asterisks are used for italics and double asterisks are actually used for bold text.”

It seems like it has incorrectly picked up this information somewhere so it keeps reverting back to it. Any idea how I can be more forceful to tell it to ignore its previous knowledge of the subject and only do single * ?

Currently my promo includes this…

You are a fun-loving chatbot called "Manda" who helps people have better meetings. 

You sometimes use emojis when communicating with people to make your messages more friendly.

Slack Formatting:
- To make text bold, enclose it with asterisks (*). Example: *text* becomes bold.
- If you want to include links the format is <URL|link text>, for example, <http://www.example.com|Click Here>.

Keep your interactions with the user brief as you will be communicating through Slack instant messages. 

Where appropriate you should format your messages for Slack.

When you provide a response to the user, do not offer to help with anything else. 

Also one more example of it hallucinating with the URL links as well.

In the prompt I say…

If you want to include links the format is <URL|link text>, for example, <http://www.example.com|Click Here>.

And as you can see it goes a bit rouge, and if I ask it about it it eventually corrects itself but doesn’t provide the best user experience… :sweat_smile:

This might actually be a legit use-case for fine tuning.

It feels like the newer models are less and less able to follow instructions like that, because they think they’re chatgpt - so they’ll prefer to spit out gfm over anything else. This behavior is even starting to spill over into github copilot

Personally, I’d bite the bullet and just add a post processor that uses regex to identify and rewrite the links. Try to get a prompt to spit out consistent output and then rewrite it rather than bend the mistrained model out of shape.

However, you can also just try an older model, you ‘might’ get better results - at the cost of slightly more hallucinatory behavior if it’s not prompted correctly.

3 Likes

Thanks for the reply @Diet

I’ve taken the post processor path and so far results have been very good. I think you are right with sticking with whatever is more consistent

2 Likes

The most reliable method is mentioned “postprocessing” - converting the output after receiving it to the format used by Slack. There are ready to use libs to do that. In the JavaScript ecosystem: GitHub - jsarafajr/slackify-markdown: Convert markdown into Slack-specific markdown (last up[date 4 years ago).

Another option is to convert markdown message to JSON format used by slack API. This library may help in this approach: GitHub - tryfabric/mack: Markdown To Slack: Convert Markdown and GitHub Flavoured Markdown to Slack BlockKit block objects 🔀💬

1 Like