Validate JSONL Training File

[_j]

The file should have all strings in double-quotes, thus no need to escape an apostrophe.

The data is JSON, not python autoquotes.

This code should crash nicely on bad json:

Fine-Tuning of my personal blog

tiktoken, which I mentioned above, is a python token counting library that you can install simply with pip. pip install tiktoken Because I’m a fun guy, I just sat down and coded a per-line token counter for jsonl training files. import json import tiktoken class Tokenizer: def __init__(self, model_name): self.tokenizer = tiktoken.get_encoding(model_name) def count(self, text): encoded_text = self.tokenizer.encode(text) return len(encoded_text) def messag…