Install SGLang on your GPU host, launch an OpenAI-compatible server for structured generation, then integrate with Bifrost for virtual keys, budgets, and multi-provider routing. Complete setup in under 20 minutes.
Bifrost can front a self-hosted SGLang server so teams share one gateway with budgets, observability, and structured-output workloads.
| Property | Details |
|---|---|
| Description | SGLang is an open-source serving framework for fast LLM inference with structured output support and an OpenAI-compatible HTTP API. |
| Provider route on Bifrost | sgl/<model> |
| Provider doc | SGLang |
| API endpoint for provider | http://localhost:8000/v1 |
| Supported endpoints | /v1/models, /v1/completions, /v1/chat/completions, /v1/responses, /v1/embeddings |
SGLang documentation, GitHub repository, and model hub references.
Before you begin, you will need:
[ QUICK START ]
Use pip on a GPU machine when possible.
On a machine with CUDA, install SGLang with extras. See the SGLang docs for hardware requirements.
$ pip install "sglang[all]"
For CPU-only hosts:
$ pip install sglang
Pick a model ID from Hugging Face (for example meta-llama/Meta-Llama-3.1-8B-Instruct). Ensure you have access tokens if the model is gated.
SGLang listens on port 8000 by default.
Start the server with your model path. Wait until the logs show the server is ready.
$ python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --port 8000
http://localhost:8000/v1.Point clients at localhost:8000/v1.
Call the local OpenAI-compatible endpoint:
$ curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "messages": [{"role":"user","content":"Hello from SGLang!"}] }'
Constrain generation with SGLang programs or schema-aware APIs.
SGLang supports constrained decoding for JSON-like outputs. Example using the Python frontend:
import sglang as sgl import json @sgl.function def extract_info(s): s += sgl.gen( "output", max_tokens=500, regex=r'\{.*"name".*"age".*\}', ) state = extract_info() result = json.loads(state["output"]) print(result)
[ MODELS ]
| Model | API ID | Best for |
|---|---|---|
| Llama 3.1 8B Instruct | meta-llama/Meta-Llama-3.1-8B-Instruct | Common starter model for SGLang. |
| Mistral 7B Instruct v0.3 | mistralai/Mistral-7B-Instruct-v0.3 | Efficient 7B instruct serving. |
| Llama 3.3 70B Instruct | meta-llama/Llama-3.3-70B-Instruct | Larger production deployment. |
| Qwen 2.5 7B Instruct | Qwen/Qwen2.5-7B-Instruct | Strong small model for coding. |
| Llama 2 7B Chat | meta-llama/Llama-2-7b-chat-hf | Legacy 7B chat baseline (~14GB VRAM). |
| Code Llama 34B | codellama/CodeLlama-34b-Instruct-hf | Code-heavy workloads (high VRAM). |
Models and VRAM requirements vary by quantization and tensor parallelism. See Hugging Face and the SGLang docs for deployment guidance.
[ TROUBLESHOOTING ]
| Error | Likely Cause | What to Do |
|---|---|---|
CUDA OOM | Model exceeds available GPU memory. | Use a smaller model, enable quantization, or add GPUs per SGLang docs. |
CUDA not found | NVIDIA drivers or CUDA toolkit missing. | Install drivers and CUDA, or use the CPU-only pip install path. |
Port 8000 in use | Another process is bound to the default port. | Pass --port 8001 or stop the conflicting process. |
Slow inference | CPU fallback or undersized GPU for the model. | Confirm GPU is used, reduce model size, or tune batch settings in SGLang. |
[ PRODUCTION-READY ]
Bifrost fronts your SGLang server: keep OpenAI-compatible client code and point the base URL at the gateway. Bifrost handles cost tracking, virtual keys, budgets, and failover automatically.
Run the Bifrost gateway and add your SGLang base URL in the Web UI.
$ npx -y @maximhq/bifrost
✓ Bifrost started ├─ HTTP server listening on http://localhost:8080 ├─ Web UI available at http://localhost:8080 └─ Configure providers and virtual keys in the dashboard
Update your client to route through Bifrost's SGLang-compatible gateway instead of localhost:8000 directly.
from openai import OpenAI client = OpenAI( api_key="sk-bf-your-virtual-key", base_url="http://localhost:8080/sgl" ) response = client.chat.completions.create( model="sgl/meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[{"role": "user", "content": "Hello from Bifrost!"}] ) print(response.choices[0].message.content)
x-bf-vk or Authorization: Bearer sk-bf-* per the Bifrost documentation.[ WHAT'S NEXT ]
You have your API key. Add governance, guardrails, and MCP controls for production.
[ BIFROST FEATURES ]
Everything you need to run AI in production, from free open source to enterprise-grade features.
01 Governance
SAML support for SSO and Role-based access control and policy enforcement for team collaboration.
02 Adaptive Load Balancing
Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics.
03 Cluster Mode
High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal.
04 Alerts
Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook and more.
05 Log Exports
Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export capabilities for compliance, monitoring, and analytics.
06 Audit Logs
Comprehensive logging and audit trails for compliance and debugging.
07 Vault Support
Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration.
08 VPC Deployment
Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls.
09 Guardrails
Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents.
[ SHIP RELIABLE AI ]
Change just one line of code. Works with OpenAI, Anthropic, Vercel AI SDK, LangChain, and more.
[ FAQ ]
SGLang is an open-source framework for efficient LLM serving with structured output guarantees, including JSON schemas and regex-constrained generation.
Local SGLang often runs without auth. Put authentication and budgets at the Bifrost gateway instead.
An NVIDIA GPU with at least 8GB VRAM is recommended for smaller models. CPU-only installs are possible but slower.
Yes. SGLang uses constrained decoding so outputs can match JSON schemas, regex patterns, and other structured formats.
Register your SGLang base URL in Bifrost and route OpenAI-compatible requests via http://localhost:8080/sgl.
Any Hugging Face compatible checkpoint, including Llama, Mistral, Qwen, and other popular open-weight models.