Seeking technical feedback on a numerical encoding and text reconstruction method

ChatGPT Use cases and examples
1

Hello,

I am looking for a technically knowledgeable person who would be willing to discuss an experimental method I have been developing for representing tokenizer-generated token IDs through a numerical encoding scheme.

The method is intended to preserve sequence, context and the information required for faithful reconstruction of the original text. The receiving component reconstructs the text before it is sent to the language model.

My preliminary tests are only an initial indication, not proof of an overall efficiency improvement. I would like to understand whether the complete approach is technically sound and whether it could offer practical benefits in terms of data representation, transmission or storage, including all metadata and reconstruction requirements.

I am not an experienced programmer and do not have the resources to build a full prototype independently. I am therefore looking for an informed technical discussion and an assessment of feasibility, rather than asking for someone to endorse the idea without testing it.

Would anyone with relevant experience in tokenization, data encoding, compression or information theory be willing to discuss the approach and suggest an appropriate way to evaluate it?

Thank you.

2

Hello everyone,

I am looking for a technically knowledgeable person who can help assess an experimental method I have been developing over the past several days. The project concerns the numerical representation of token IDs generated by an existing tokenizer and the subsequent reconstruction of the original text.

The objective

The idea is to investigate whether a structured numerical representation of tokenizer-generated IDs could provide practical advantages when storing or transmitting text-related data, while preserving the information required to reconstruct the original text faithfully.

This is not a proposal for a new tokenizer. It is an approach that operates on token IDs produced by an existing tokenizer.

The general approach

The method involves several elements:

  • Numerical transformations based on a reference value.
  • Grouping and ordering of numerical values.
  • Preservation of sequence and contextual information.
  • Identification of the encoding scheme used.
  • An associative memory or mapping mechanism where required.
  • Reconstruction of the original text by a receiving component.

The complete method must be considered as a system. Evaluating an isolated arithmetic transformation would not be sufficient to assess the full approach, because grouping, ordering, contextual information and reconstruction are also relevant.

I am deliberately keeping some implementation details private at this initial stage.

Preliminary observations

I have performed initial experiments using an existing tokenizer and compared the token counts of an example sentence with those of a numerical representation.

These observations are preliminary. They do not establish that the complete system achieves compression or reduces total transmission costs. A proper evaluation would need to include the full encoded representation, metadata, any mapping or associative memory requirements, and the resources needed for decoding.

An important clarification

The receiving component is intended to reconstruct the original text before that text is sent to the language model.

Therefore, I am not claiming that the numerical representation necessarily reduces the number of tokens billed by an LLM API. The potential benefit I want to investigate concerns the representation and transport of data before reconstruction, or possibly its storage.

What I am looking for

I would appreciate a technical discussion with someone experienced in one or more of the following areas:

  • Tokenization and token ID representations.
  • Data compression and lossless encoding.
  • Information theory.
  • Numerical coding and reversible transformations.
  • Data structures, mapping systems and reconstruction.
  • Experimental benchmarking and system-level efficiency.

The main questions I would like to investigate are:

  1. Is the overall architecture technically feasible?
  2. What conditions would be necessary to guarantee exact reconstruction?
  3. What are the risks of information loss, ambiguity or collisions?
  4. How should the complete representation be measured against conventional compression and encoding methods?
  5. Could the method offer a practical advantage once metadata, memory, processing and transmission costs are included?

I am not an experienced software developer and currently do not have the resources to build a complete prototype independently. I am looking for an informed technical discussion and, if possible, someone willing to help evaluate the concept rigorously.

I am not asking anyone to assume that the method works. I am asking whether its complete architecture can be examined fairly and what evidence would be needed to establish its strengths and limitations.

Thank you for your time. I welcome constructive technical feedback.

3

Hello everyone,

I am looking for a technically knowledgeable person who can help assess an experimental method I have been developing over the past several days. The project concerns the numerical representation of token IDs generated by an existing tokenizer and the subsequent reconstruction of the original text.

The objective

The idea is to investigate whether a structured numerical representation of tokenizer-generated IDs could provide practical advantages when storing or transmitting text-related data, while preserving the information required to reconstruct the original text faithfully.

This is not a proposal for a new tokenizer. It is an approach that operates on token IDs produced by an existing tokenizer.

The general approach

The method involves several elements:

  • Numerical transformations based on a reference value.
  • Grouping and ordering of numerical values.
  • Preservation of sequence and contextual information.
  • Identification of the encoding scheme used.
  • An associative memory or mapping mechanism where required.
  • Reconstruction of the original text by a receiving component.

The complete method must be considered as a system. Evaluating an isolated arithmetic transformation would not be sufficient to assess the full approach, because grouping, ordering, contextual information and reconstruction are also relevant.

I am deliberately keeping some implementation details private at this initial stage.

Preliminary observations

I have performed initial experiments using an existing tokenizer and compared the token counts of an example sentence with those of a numerical representation.

These observations are preliminary. They do not establish that the complete system achieves compression or reduces total transmission costs. A proper evaluation would need to include the full encoded representation, metadata, any mapping or associative memory requirements, and the resources needed for decoding.

An important clarification

The receiving component is intended to reconstruct the original text before that text is sent to the language model.

Therefore, I am not claiming that the numerical representation necessarily reduces the number of tokens billed by an LLM API. The potential benefit I want to investigate concerns the representation and transport of data before reconstruction, or possibly its storage.

What I am looking for

I would appreciate a technical discussion with someone experienced in one or more of the following areas:

  • Tokenization and token ID representations.
  • Data compression and lossless encoding.
  • Information theory.
  • Numerical coding and reversible transformations.
  • Data structures, mapping systems and reconstruction.
  • Experimental benchmarking and system-level efficiency.

The main questions I would like to investigate are:

  1. Is the overall architecture technically feasible?
  2. What conditions would be necessary to guarantee exact reconstruction?
  3. What are the risks of information loss, ambiguity or collisions?
  4. How should the complete representation be measured against conventional compression and encoding methods?
  5. Could the method offer a practical advantage once metadata, memory, processing and transmission costs are included?

I am not an experienced software developer and currently do not have the resources to build a complete prototype independently. I am looking for an informed technical discussion and, if possible, someone willing to help evaluate the concept rigorously.

I am not asking anyone to assume that the method works. I am asking whether its complete architecture can be examined fairly and what evidence would be needed to establish its strengths and limitations.

Thank you for your time. I welcome constructive technical feedback.

4

Sounds to me that an encryption implementation is your best solution.

5

Tokens are simply a dictionary of mapping strings to integers and integers to strings. Byte pair encoding will map inputs to integers, recursively collapsing matched text. It is also lossless in recovering the original signal. It serves as a compressed semantic-based signal when training AI.

Even with the same dictionary, there is potential for a different token encoding as a transmission format if the software were to be altered, or if the output was prediction by an AI that doesn’t follow strict or deterministic rules. Decoding converts back to string sequence regardless.

Tokenization for AI input is optimized and trained, such that frequent and profitable sequences are resolved down finally to a single token number. A very large corpus and lots of computation was used on creating the underlying token dictionary.

You seem to ask questions about cost savings for AI input once, but the main line of inquiry remains in other directions about data theory that have little to do with AI. You send natural language to AI models, and tokenization is done for you by the service.

Have a read about an application using tokenization as a possible compression:

Compression

If short and especially English, cl100k tokenization can win if you can employ a codec made larger by a 2MB dictionary.
If longer input file or binary, typical compression like lzma with a self-built symbol dictionary can be better.

Storage

Since 100k symbols is more than a computer word of 16 bits, there’s a 2.3% loss of compression if you send a bitstream that maintains discrete packed 17 bit integers. You could use 2-byte token data and disregard the rest of the token dictionary entries past 65535, since tokens are also indexed by frequency (except for hard-coded entries). Or you could use numeric coding that doesn’t use bit-length words, it uses larger numeric packing.