Tokenization fun: long character strings of only 1 token, for prompt and code formatting and more

Prompting
1

ChatGPT’s tokenizer optimizes many long series of characters into a single token. We can craft some radical character reductions out of them.

For example, a section header only four tokens long, picking a long sequence including a final carriage return:

################################################################################
 instructions----------------------------------------------------------------------
################################################################################

This makes for some interesting code compression techniques.
You can make prompt sections or code comments fed to the AI abundantly clear at little expense.

I expect that lower token numbers, being more common, will have more semantic meaning such as β€œsection divider”.

List of longest tokens, some of no interest removed.
Notes:

  • infer the length of spaces not seen; below 80, there’s a token for every series of only spaces
  • a few begin with an unpictured free carriage return
number len token characters
058041 128 ’ ’
090281 114 β€˜//----------------------------------------------------------------------------------------------------------------’
067308 113 ’ ----------------------------------------------------------------------------------------------------------------’
057443 098 β€˜//------------------------------------------------------------------------------------------------’
060910 097 ’ ------------------------------------------------------------------------------------------------’
087645 097 β€˜/************************************************************************************************’
061730 096 β€˜////////////////////////////////////////////////////////////////////////////////////////////////’
099575 096 β€˜------------------------------------------------------------------------------------------------’
087867 095 ’ ’
080972 091 ’ ’
066372 087 ’ ’
052576 083 ’ ’
051662 082 β€˜//--------------------------------------------------------------------------------’
067001 082 ’ =================================================================================’
088668 082 β€˜//------------------------------------------------------------------------------\n\n’
094783 082 ’ ******************************************************************************/\n\n’
095404 082 β€˜//================================================================================’
099421 082 β€˜////////////////////////////////////////////////////////////////////////////////\n\n’
037814 081 β€˜/*******************************************************************************\n’
040474 081 ’ --------------------------------------------------------------------------------’
046915 081 β€˜//------------------------------------------------------------------------------\n’
059970 081 β€˜////////////////////////////////////////////////////////////////////////////////\n’
076733 081 β€˜/********************************************************************************’
077651 081 ’ ********************************************************************************’
080472 081 ’ ******************************************************************************/\n’
080504 081 β€˜*******************************************************************************/\n’
086100 081 β€˜/******************************************************************************/\n’
091831 081 β€˜################################################################################\n’
029327 080 β€˜////////////////////////////////////////////////////////////////////////////////’
041587 080 β€˜################################################################################’
044550 080 β€˜--------------------------------------------------------------------------------’
045243 080 β€˜//-----------------------------------------------------------------------------\n’
054297 080 β€˜/******************************************************************************\n’
062794 080 β€˜********************************************************************************’
064495 080 β€˜================================================================================’
077838 080 β€˜///////////////////////////////////////////////////////////////////////////////\n’
080549 080 β€˜###############################################################################\n’
086886 080 ’ ******************************************************************************\n’
052915 079 ’ -----------------------------------------------------------------------------\n’
069233 079 β€˜/*****************************************************************************\n’
080039 079 β€˜//----------------------------------------------------------------------------\n’
083150 079 ’ =============================================================================\n’
095565 079 β€˜//---------------------------------------------------------------------------\n\n’
018499 078 β€˜//----------------------------------------------------------------------------’
058408 078 β€˜//---------------------------------------------------------------------------\n’
059007 078 β€˜/*----------------------------------------------------------------------------’
059108 078 β€˜/****************************************************************************\n’
064639 078 ’ ----------------------------------------------------------------------------\n’
065327 078 β€˜//****************************************************************************’
084995 078 β€˜/////////////////////////////////////////////////////////////////////////////\n’
090419 078 ’ ============================================================================\n’
023382 077 β€˜/****************************************************************************’
024794 077 ’ ----------------------------------------------------------------------------’
029745 077 ’ ****************************************************************************’
062016 077 β€˜#----------------------------------------------------------------------------’
062351 077 ’
072089 077 β€˜/***************************************************************************\n’
079858 077 ’ ---------------------------------------------------------------------------\n’
087301 077 ’ ############################################################################’
023152 076 β€˜----------------------------------------------------------------------------’
028283 076 β€˜////////////////////////////////////////////////////////////////////////////’
033142 076 β€˜############################################################################’
034619 076 β€˜(76 asterisks)’
043181 076 ’
099043 076 ’ --------------------------------------------------------------------------\n’
017995 075 ’ **************************************************************************’
026973 075 β€˜--------------------------------------------------------------------------\n’
072609 075 ’ //////////////////////////////////////////////////////////////////////////’
081651 075 ’ -------------------------------------------------------------------------\n’
039469 074 ’ =========================================================================’
066181 074 β€˜//************************************************************************’
092016 074 ’ -------------------------------------------------------------------------’
097682 074 ’ ------------------------------------------------------------------------\n’
098106 074 ’ /************************************************************************’
010758 073 ’ ************************************************************************’
011625 073 β€˜/************************************************************************’
094959 073 ’ ########################################################################’
005714 072 β€˜************************************************************************’
036210 072 β€˜////////////////////////////////////////////////////////////////////////’
070162 072 β€˜########################################################################’
090795 072 ’ ----------------------------------------------------------------------\n’
063218 071 β€˜----------------------------------------------------------------------\n’
096281 071 ’ //////////////////////////////////////////////////////////////////////’
054365 070 β€˜----------------------------------------------------------------------’
044301 068 β€˜////////////////////////////////////////////////////////////////////’
036720 067 ’ //----------------------------------------------------------------’
068650 067 ’ /*----------------------------------------------------------------’
087094 067 ’ //================================================================’
010090 066 β€˜//----------------------------------------------------------------’
016564 066 ’ =================================================================’
024037 066 β€˜//================================================================’
030966 066 β€˜/*----------------------------------------------------------------’
045539 066 β€˜/*================================================================’
056364 066 β€˜//****************************************************************’
078250 066 ’ *----------------------------------------------------------------’
100090 066 ’ /****************************************************************’
008634 065 ’ ----------------------------------------------------------------’
020767 065 β€˜/****************************************************************’
023090 065 ’ ****************************************************************’
049598 065 ’ ################################################################’
073105 065 β€˜#================================================================’
076611 065 β€˜#----------------------------------------------------------------’
082143 065 β€˜//--------------------------------------------------------------\n’
087173 065 ’ (65 periods)’
003598 064 β€˜----------------------------------------------------------------’
004170 064 β€˜(64 asterisks)’
008316 064 β€˜================================================================’
010024 064 β€˜////////////////////////////////////////////////////////////////’
013368 064 β€˜################################################################’
043370 064 β€˜(64 periods)’
048033 064 β€˜________________________________________________________________’
080619 064 β€˜%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%’
071763 063 ’ ==============================================================’
087733 061 ’ ------------------------------------------------------------’
065676 060 β€˜////////////////////////////////////////////////////////////’
098945 060 β€˜############################################################’
066296 057 ’ ********************************************************’
067260 057 β€˜/********************************************************’
054248 056 β€˜********************************************************’
062009 056 β€˜////////////////////////////////////////////////////////’
091139 056 β€˜########################################################’
083946 052 β€˜////////////////////////////////////////////////////’
067105 051 ’ //------------------------------------------------’
029686 050 ’ =================================================’
031990 050 β€˜//------------------------------------------------’
074163 050 β€˜/*------------------------------------------------’
077608 050 β€˜//================================================’
018528 049 ’ ------------------------------------------------’
068262 049 ’ ################################################’
072937 049 ’ ************************************************’
079472 049 β€˜/************************************************’
096048 049 ’ \n’
009412 048 β€˜------------------------------------------------’
019312 048 β€˜================================================’
028506 048 β€˜////////////////////////////////////////////////’
030955 048 β€˜################################################’
047677 048 β€˜************************************************’
079094 045 ’ \n’
089905 042 ’ \n \n’
057697 041 ’ \n’
062461 041 ’ ****************************************’
074062 041 β€˜/****************************************’
041173 040 β€˜****************************************’
083679 040 β€˜########################################’
046908 037 ’ \n’
061710 035 ’ //////////////////////////////////’
074639 035 ’ //--------------------------------’
082109 035 ’ __________________________________’
036875 034 ’ =================================’
049357 034 β€˜//--------------------------------’
060750 034 ’ \n \n’
079447 034 β€˜("--------------------------------’
020309 033 ’ --------------------------------’
023815 033 ’ ********************************’
026343 033 β€˜/********************************’
034742 033 ’ \n’
055402 033 ’ ################################’
082473 033 ’ (32 periods)’
001435 032 β€˜--------------------------------’
001725 032 β€˜********************************’
003135 032 β€˜================================’
003986 032 β€˜////////////////////////////////’
005135 032 β€˜################################’
016972 032 β€˜(32 periods)’
017925 032 β€˜________________________________’
034110 032 β€˜%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%’
066749 032 β€˜~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~’

validate your one token sequence by clearing this tokenizer until it reads 0 tokens, then paste.

Actual token numbers used internally are one lower, since I didn’t start at 0…

1 Like
2

It’s really interesting to see that the same character at different number of repetitions have a unique token number. I wonder what prompted this during the encoding…

3

The tokens are generated from coding efficiency on training corpus, along with others manually built (like all numbers up to 999).

Just programmers doing programmer stuff:

image
image1035Γ—813 13.8 KB

4

This is how I’m going to define any quirks in my systems from now on XD

2 Likes
5

List of longest tokens, some of no interest removed.

They’re all interesting to me, could you share the longest ones?

6

I found β€˜abcdefghijklmnopqrstuvwxyz’ to be one single token of 26 characters.

7

Good find! The same exists in all upper-case

Also interesting, and allowing us to infer the common source might have been in computer code (enclosed in quotes of a string), is that if there is a leading space, this input representation is broken by BPE encoding (but still may be what an AI produces as an output token).

image

I just ran a similar processing with gpt-4o’s o200k_base token encoder, and we have a new set of β€œlongest 1-token strings”. Those starting with slashes lose out in the new optimization.

72056,128,'                                                                                                                                '
179041,113,' ----------------------------------------------------------------------------------------------------------------'
182513,112,'----------------------------------------------------------------------------------------------------------------'
141084,97,' ------------------------------------------------------------------------------------------------'
85810,96,'------------------------------------------------------------------------------------------------'
168258,96,'************************************************************************************************'
193856,96,'================================================================================================'
195732,95,'                                                                                               '
174893,91,'                                                                                           '
146780,88,'****************************************************************************************'
134788,87,'                                                                                       '
116288,83,'                                                                                   '
166044,82,' ================================================================================='
79593,81,' --------------------------------------------------------------------------------'
111383,81,'/*******************************************************************************\n'
163690,81,' ********************************************************************************'
50529,80,'--------------------------------------------------------------------------------'
73681,80,'////////////////////////////////////////////////////////////////////////////////'
76470,80,'*******************************************************************************\n'
82394,80,'********************************************************************************'
103402,80,'################################################################################'
111845,80,'================================================================================'
73838,79,' ******************************************************************************'
82724,79,'                                                                               '
133633,79,'------------------------------------------------------------------------------\n'
162673,79,'******************************************************************************\n'
42538,78,'******************************************************************************'
113027,78,'------------------------------------------------------------------------------'
127961,78,'-----------------------------------------------------------------------------\n'
132548,78,' ============================================================================='
174517,78,'=============================================================================='
187205,78,'                                                                              '
196792,78,'*****************************************************************************\n'
70007,77,' ****************************************************************************'
77304,77,' ----------------------------------------------------------------------------'
109438,77,'/****************************************************************************'
130377,77,'-----------------------------------------------------------------------------'
133272,77,'                                                                             '
150130,77,' |--------------------------------------------------------------------------\n'
172453,77,'----------------------------------------------------------------------------\n'
28943,76,'----------------------------------------------------------------------------'
41547,76,'****************************************************************************'
63146,76,'============================================================================'
76050,76,'////////////////////////////////////////////////////////////////////////////'
81787,76,'############################################################################'
104232,76,'|--------------------------------------------------------------------------\n'
136356,76,'                                                                            '
30723,75,'                                                                           '
39967,75,' **************************************************************************'
61743,75,'--------------------------------------------------------------------------\n'
78407,75,'---------------------------------------------------------------------------'
176860,75,' //////////////////////////////////////////////////////////////////////////'
180864,75,'==========================================================================='
97361,74,'                                                                          '
126247,74,' ========================================================================='
198676,74,'-------------------------------------------------------------------------\n'
22567,73,' ************************************************************************'
102820,73,'                                                                         '
117837,73,'/************************************************************************'
12217,72,'************************************************************************'
32635,72,'------------------------------------------------------------------------'
88017,72,'                                                                        '
143581,72,'////////////////////////////////////////////////////////////////////////'
154118,72,'========================================================================'
185617,72,'########################################################################'
195967,72,' +----------------------------------------------------------------------'
46371,71,'                                                                       '
98699,70,'----------------------------------------------------------------------'
113903,70,'                                                                      '
94875,69,'                                                                     '
122722,68,'                                                                    '
174665,68,'////////////////////////////////////////////////////////////////////'
38959,67,'                                                                   '
93582,67,' //////////////////////////////////////////////////////////////////'
130774,67,' //----------------------------------------------------------------'
36014,66,' ================================================================='
110321,66,'                                                                  '
131495,66,'//----------------------------------------------------------------'
18882,65,' ----------------------------------------------------------------'
50885,65,' ****************************************************************'
82955,65,'                                                                 '
113341,65,'/****************************************************************'
125012,65,' ################################################################'
164601,65,' ................................................................'
168750,65,'*----------------------------------------------------------------'
7535,64,'----------------------------------------------------------------'
8599,64,'****************************************************************'
9344,64,'                                                                '
17686,64,'================================================================'
25493,64,'////////////////////////////////////////////////////////////////'
29834,64,'################################################################'
74358,64,'................................................................'
80553,64,'________________________________________________________________'
30319,63,'                                                               '
175190,63,' =============================================================='
84839,62,'                                                              '
66188,61,'                                                             '

Right after this is the first language, at 61 bytes:
ΰΈͺำนักเΰΈ₯ΰΈ‚ΰΈ²ΰΈ™ΰΈΈΰΈΰΈ²ΰΈ£ΰΈ­ΰΈ‡ΰΈ„ΰΉŒΰΈΰΈ£

The phrase translates to: β€œOffice of the Secretary of the Organization”

1 Like
8

It’s because long strings of the same token like a - * or _ just count as one token β€”β€”β€”β€”β€”β€”β€” =1 and β€”β€”- = 1 as does - = 1

:duck:

Older forums had similar character counts for posts, but ASCII art used similar trick.

9

I found this super helpful for making code more clear, like in long scripts or prompts. Using stuff like dashes (- ) or stars (* ), you can make dividers that look good and only take one token. It’s easier to read and saves space too. Like, //-------------------------------------------------------------------------------- works to split sections and doesn’t use a lot of tokens, so it’s better. Makes it simple and works good for keeping things neat without wasting tokens.