Did you see this?
Language models can explain neurons in language models
among other things there is a tool with published code on GitHub.
If you use the tool you will see that it has access to every neuron in GPT-2 (not GPT-4) and also the tokens.
If you look at the code you will find
So while it may not be the tokens you are looking for, it might be part of what you seek in the long run but for a simpler model.