Yeah, I saw that thread but they don’t recommend a solution that generates and limits individual tokens for users.
I just discovered GitHub - bricks-cloud/BricksLLM: Simplifying LLM ops in production and might give it a try.
2 Likes