Hi there,
I’m playing around with an assistant you interact with via Slack.
I want to LLM to conform to slack formatting principles… (google slack formatting)
I’m using gtp4-turbo-preview and whilst it is sometimes aware of slack formatting it sometimes goes rouge and does its own thing.
For example instead of using a single astricks to make something bold it will sometimes do a double one like this.
I’m trying to work out the best way to improve this. Should I just be filling my instructions/prompt with a bunch of formatting examples? Or is there some way I can train the model to be better at this?
It outputs markdown fairly well, but not sure about Slack formatting… Is there another phrase that could be used to describe it? An example or two might help.
Another option would be to translate the markdown to Slack-down ™?
Here is an example of where it doesn’t get it right
My prompt includes this…
You can format your messages for Slack. If you want to make text bold by Surround your text with a pair of single asterisks, like *this*.
If you want to include links the format is <URL|link text>, for example, <http://www.example.com|Click Here>.
Keep your interactions with the user brief as you will be communicating through Slack instant messages. Don't ask the user unnecessary questions.
To format messages in Slack, use the following “Slack formatting” conventions. These instructions are tailored specifically for enhancing readability and organization within Slack’s desktop and mobile applications:
Bold Text: To make text bold, enclose it with asterisks (*). Example: *text* becomes bold.
Italic Text: To italicize text, enclose it with underscores (_). Example: _text_ becomes italic.
Strikethrough Text: To apply a strikethrough effect, enclose text with tildes (~). Example: ~text~ becomes strikethrough.
Inline Code: For inline code formatting, enclose text with backticks (```). Example: text becomes code.
Blockquote: To format text as a blockquote, start it with an angled bracket (>). Example: >text.
Code Block: To create a code block, enclose text with triple backticks (`````).
Ordered List: For an ordered list, start each item with a number followed by a period. Example: 1. text.
Bulleted List: For a bulleted list, start each item with an asterisk and a space. Example: * text.
These Slack-specific formatting commands enhance your messaging by adding visual distinctions to your text, making your Slack communications more effective and engaging.
Can’t hurt to try! Your temp and other settings might be messing it up a bit too.
Hi Paul,
Still having a few problems as it keeps going back to double astricks. I played around with it in the chat today and got some interesting results.
See this comment from the LLM…
“However, please note, in Slack formatting, single asterisks are used for italics and double asterisks are actually used for bold text.”
It seems like it has incorrectly picked up this information somewhere so it keeps reverting back to it. Any idea how I can be more forceful to tell it to ignore its previous knowledge of the subject and only do single * ?
Currently my promo includes this…
You are a fun-loving chatbot called "Manda" who helps people have better meetings.
You sometimes use emojis when communicating with people to make your messages more friendly.
Slack Formatting:
- To make text bold, enclose it with asterisks (*). Example: *text* becomes bold.
- If you want to include links the format is <URL|link text>, for example, <http://www.example.com|Click Here>.
Keep your interactions with the user brief as you will be communicating through Slack instant messages.
Where appropriate you should format your messages for Slack.
When you provide a response to the user, do not offer to help with anything else.
This might actually be a legit use-case for fine tuning.
It feels like the newer models are less and less able to follow instructions like that, because they think they’re chatgpt - so they’ll prefer to spit out gfm over anything else. This behavior is even starting to spill over into github copilot
Personally, I’d bite the bullet and just add a post processor that uses regex to identify and rewrite the links. Try to get a prompt to spit out consistent output and then rewrite it rather than bend the mistrained model out of shape.
However, you can also just try an older model, you ‘might’ get better results - at the cost of slightly more hallucinatory behavior if it’s not prompted correctly.
