AI model fingerprints are not unique, making them fairly useless for tracking model updates

The system fingerprint was introduced November 2023, as something that would let you know when one of the models released then or after was updated. It’s part of the response object that you get with the language generated.

However, as I show here, there’s up to half-dozen values returned when making the same call a hundred times in a row.

So to really detect some model change rolled out, you’d have to again get dozens and compare to the past.

If it affected determinism, you’d have to again duplicate the calls many times to happen upon the same fingerprint being returned.

Seed repeats the same random algorithm again for sampling tokens, but the present models are not reliable even with the same seed.

2 Likes