I have a product that allows a caller to use a simple scripting language to model a financial transaction, perform calculations on the model, and query the results.
My goal is to hook this up to GPT-4 using a Plugin or Function Call and have GPT-4 write ad hoc scripts that return whatever information GPT-4 requires in service to some goal. Ideally, I’d like the model to become an expert in writing scripts, just like it can write programs in other computer languages.
I’ve experimented with Plugins and adding instructions and scripting examples at the start of my prompt and been amazed at how well GPT-4 is able to generalize and write some simple scripts. However, trying to describe the entire scripting language with examples would quickly use up the context so I’m wondering if fine-tuning one of the other OpenAI models is possible and would that give me good or better results.
I have no experience with fine-tuning other than watching YouTube videos, so any advice on best practices is appreciated. Also, please let me know if there’s a better way to go that I’m not considering. thx!
2 Likes
First, give chat.openai.com a try with selecting GPT-4 or use playground. This will give you an idea of how the system learns. Afterwards, you can choose to use chat context or fine-tune the model according to your needs.
Fine tunings models to perform new tasks is exactly what it’s design for, but the act of fine tuning and making it a success is a hugely complex area.
It should also be noted that current “chat” models such as GPT-3.5-Turbo and GPT-4 do not have a fine tuning ability, so the task becomes even more complex, what you ask can be achieved but it is involved and will require considerable effort to be put in.
I’m not sure if it’s worth the effort of fine-tuning for this. You could try formulating your language in a formal language like EBNF. GPT-4 can handle this quite well. I’ve tried it once with the syntax for Swift messages. Swift here refers to the data format for financial transactions, not the programming language. This syntax isn’t a programming language, but the principle should be the same.
User:
The SWSyntax is described with the following EBNF
<SWSyntax> ::= <Syntaxitem> (<Syntaxdelimitter> <Syntaxitem>)* (<Syntaxdelimitter><RowSyntax>)?
<Syntaxitem> ::= <OptionalSyntax> | <NonOptionalSyntax>
<NonOptionalSyntax> ::= <FieldSyntax>
<OptionalSyntax> ::= "[" <NonOptionalSyntax> "]"
<RowSyntax> ::= <MaximumNumberOfRows> "*" <FieldSyntax>
<FieldSyntax> ::= <FieldLength> <FixedLengthIndicator>? <CharacterSet>
<FieldLength> ::= <Number>
<MaximumNumberOfRows> ::= <Number>
<FixedLengthIndicator> ::= "!"
<CharacterSet> ::= <AlphaCharacterSet> | <NumCharacterSet>
<AlphaCharacterSet> ::= "a" | "n" | "c"
<NumCharacterSet> ::= "x" | "d"
<Syntaxdelimitter>::= "/"|":"
<Number> ::= [1-9] ([0-9]+)?
Examples for syntax items:
1!a = One character from the character set a
20n = 1 to 20 characters from the character set n
15!x = 15 characters from the character set x
[3!a] - Optionally exactly three characters from the character set ‘a’.
3*10!x - Means that up to three lines of exactly ten characters from the character set ‘x’ are allowed.
Examples for SWSyntax:
assistant:
From the provided EBNF, I will now generate some examples of SWSyntax. In each case, I will also provide a brief explanation of what the syntax represents based on the definitions and examples provided in your question.
-
10!a/15n/3*5!d
- This syntax means: Exactly 10 characters from the character set ‘a’, followed by anywhere from 1 to 15 characters from the character set ‘n’, and then up to three lines of exactly 5 characters from the character set ‘d’ are allowed.
-
20!c:[4!x]
- This syntax means: Exactly 20 characters from the character set ‘c’, followed optionally by exactly 4 characters from the character set ‘x’.
-
5n:2*10!a
- This syntax means: Anywhere from 1 to 5 characters from the character set ‘n’, followed by up to two lines of exactly 10 characters from the character set ‘a’.
-
[8!n]/3*7!d
- This syntax means: Optionally exactly 8 characters from the character set ‘n’, followed by up to three lines of exactly 7 characters from the character set ‘d’.
-
1!a:2!x/[10n]
- This syntax means: Exactly one character from the character set ‘a’, followed by exactly 2 characters from the character set ‘x’, and then optionally anywhere from 1 to 10 characters from the character set ‘n’.
Please note that the character sets defined are ‘a’, ‘n’, ‘c’, ‘x’, and ‘d’. If other character sets are used, they would need to be defined in your EBNF.
user:
Please generate a SWSyntax to match exactly 132 characters of character set x.
assistant:
Sure, the SWSyntax that matches exactly 132 characters of the character set ‘x’ would be:
132!x
Here, 132 represents the exact number of characters, the ! symbol indicates that the field must be of fixed length, and x is the specified character set. So, this syntax means “exactly 132 characters from the character set ‘x’”.
2 Likes
Thanks to everyone who responded to my question. @frank_behr That’s an interesting example. My scripting language uses XML syntax where the XML tags reference object and fields in the data model. So, the syntax is easy (well-formed XML which GPT already understands), The hard part is teaching GPT all the fields in the data model and what they are used for. I can provide examples in-context, but I think the full example-space is larger than what will fit in 32K tokens. But, I’m going to see how far I can push it. My experience with GPT-4 so far is that it always surprises me when I think it can’t do something! I’m still considering the fine-tuning approach. Although it avoids the context limitations, I’m not convinced that fine-tuning a lessor model is going to give me better results and I’m not sure how to use a fine-tuned model with ChatGPT… guess I’ll just have to experiment.
2 Likes