How do you make a bpe file for Tokenizer

[ruby_coder]

@raymonddavey

I need to make a tokenizer in C# for 3.5-turbo.

Wouldn’t it be easier to call the Python tokenizer from C#?

This is how I did it using Ruby and it works fine for me, which I use for many tasks including (1) counting tokens in text and (2) creating logit_bias params.

Using TikToken in Ruby to Generate `logit_bias` for a String

For those of you who use Ruby, here is how I use TikToken in my Ruby code since there is no Ruby tiktoken gem for the turbo model (yet). Just a quick hack…You can make this more elegant if you wish, of course. I just tossed this salad together today for some testing: Create a Python script, like this (after installing tiktoken, of course) import tiktoken import sys def tik(words): encoding = tiktoken.encoding_for_model("gpt-3.5-turbo") tokens = encoding.encode(words) return to…

Here is a random tutorial demonstrating how to call a Python script from C#. There are many others tutorials on the net on the topic:

Calling Python from C#: an introduction to PythonNET

HTH

Note, you can also call Python directly in a C# program using IronPython, FYI (but I have not tested it as it’s been a few years since I wrote C# code):

https://ironpython.net/