Faster Api throughput using Cerebras

Community
6

Well, it’s true that enterprise customers need a protected environment and gpt-oss-120b could be a compelling solution when hosted by Cerebras…

But enterprises can ceate their own H100 stack and host their own gpt-oss-120b, right?

Enterprises can create their own H100 stack and host gpt-oss-120b on-premise. Hosting the model in-house offers greater control over data, security, and costs, but it requires significant investment in specialized hardware and infrastructure.

Requirements for hosting gpt-oss-120b on-premise

Hardware

  • H100 GPUs: A single NVIDIA H100 80GB GPU is sufficient for running gpt-oss-120b for inference due to its highly optimized Mixture-of-Experts (MoE) architecture. For higher throughput and performance, especially under heavy load, deploying the model across multiple H100 GPUs (e.g., 4x or 8x) is recommended.

  • GPU memory (VRAM): A single H100 provides the 80GB of VRAM required to load the model and its active parameters.

  • System memory (RAM): The server or workstation requires a minimum of 128GB of system RAM for the 120B model, in addition to the VRAM on the GPU.

  • Networking: High-speed networking, such as the 350Gbps offered on cloud services, is essential for minimizing bottlenecks, especially in multi-GPU deployments.Reference designs: Hardware vendors like Dell offer validated solutions, such as the “Dell AI Factory,” which combines PowerEdge servers with NVIDIA GPUs for implementing AI workloads on-premises.

Software and deployment

  • Inference servers: To run the model efficiently, enterprises need to deploy an inference server. Common options include:

    • NVIDIA NIM: Optimized for NVIDIA’s Hopper and Blackwell GPUs.

    • vLLM: A popular and highly performant inference engine.

    • Ollama: A simpler tool for local or on-premise model hosting.

  • Model optimization: Frameworks like NVIDIA’s TensorRT-LLM can be used to optimize the model for specific hardware configurations and improve performance.

  • Licensing: Since gpt-oss-120b is released under a permissive Apache 2.0 license, enterprises can use it for experimentation, commercial deployment, and fine-tuning without significant restrictions.

Business and security considerations

  • Data sovereignty: For enterprises handling sensitive or regulated data, hosting the model on-premise ensures that data processing remains within a controlled and chosen region, meeting data sovereignty requirements.

  • Customization: Self-hosting gives enterprises complete control to fine-tune and adapt the model with their proprietary data, creating a more tailored and powerful solution.

  • Cost avoidance: Enterprises can avoid recurring API fees and manage costs more predictably than with a pay-per-use cloud service.