For anyone running gpt-oss-20b on a laptop, what does your setup look like and how does it perform? I’m trying to understand how much it depends on dedicated GPU VRAM versus plain system RAM, and whether a laptop GPU with 8GB or 12GB VRAM makes a noticeable difference.
Also curious whether people find it comfortable for daily use, or if it’s mostly a “works but slow” experience on typical laptops. Any tokens-per-second numbers with your specs would be really helpful.
I ran it a while ago, when it first came out, on my home rig, which has an RTX5070 (12GB VRAM) and 64GB system memory. On that setup I found it quite unusable on any intermediate-to-advanced setup (e.g. coding tasks or processing larger pieces of text). I found that it had to offload to RAM quite a lot. I used it via LMStudio, Ollama, llama.cpp and Unsloth, and didn’t see much different between those.
If you are on a MacBook and have “unified memory”, like >32GB, then you can find MLX (Apple Silicon) optimised checkpoints, and then it should be decent rate. If you go with quants, like 4-bit, then it’s even faster, but notice the performance drops off significantly (even though lot of benchmarks claim “no loss”).
Thanks, this is really useful, especially hearing that even 12GB VRAM with 64GB RAM ends up offloading a lot. Did you notice whether the slowdown got much worse with longer context, or was it slow from the start? And did you get a rough tokens-per-second figure on that setup?
GPT-OSS models were not state-of-the-art of OpenAI when released, are loaded up with refusal post-training, and are now quite long in the tooth.
Without theft-loaded Chinese models being considered, you can compare smaller (and a bit larger):
Here’s the guidance that OpenAI never quite offered, where they should have produced optimized granular quants with multiple inference software targets, besides providing a model size parameterized around specific hardware not in the datacenter.
I have ran some local models on my rtx 4090 + 64GB ddr5 6400MT/s system RAM, and basically any time the model spills over to system RAM it’s unusably slow (imo). And “good-er” models in the 27B parameter range only allowed me to have ~60k tokens in context before spilling over… which is way too low for programming.
