Create a Mistral account at docs.vllm.ai, generate your API key, store it securely, then integrate with Bifrost for virtual keys, budgets, and cost governance. Complete setup in minutes.
Bifrost can front a self-hosted vLLM server so teams share one gateway with budgets and observability.
| Property | Details |
|---|---|
| Description | vLLM is an open-source inference engine for fast LLM serving with an OpenAI-compatible HTTP API. |
| Provider route on Bifrost | vllm/<model> |
| Provider doc | vLLM |
| API endpoint for provider | http://localhost:8000/v1 |
| Supported endpoints | /v1/models, /v1/completions, /v1/chat/completions, /v1/responses, /v1/embeddings, /v1/audio/transcriptions, /v1/rerank |
vLLM documentation and GitHub repository.
Before you begin, you will need:
[ QUICK START ]
Use pip on a GPU machine.
On a machine with CUDA, install vLLM: pip install vllm. See the vLLM docs for hardware requirements.
Pick a model ID from Hugging Face (for example meta-llama/Meta-Llama-3.1-8B-Instruct). Ensure you have access tokens if the model is gated.
vLLM listens on port 8000 by default.
Start the server with your model. For local testing, API key checks are often disabled or use a placeholder.
$ python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --port 8000
Point clients at localhost:8000/v1.
Call the local OpenAI-compatible endpoint:
$ curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "messages": [{"role":"user","content":"Hello from vLLM!"}] }'
[ MODELS ]
| Model | API ID | Best for |
|---|---|---|
| Llama 3.1 8B Instruct | meta-llama/Meta-Llama-3.1-8B-Instruct | Common starter model for vLLM. |
| Llama 3.3 70B Instruct | meta-llama/Llama-3.3-70B-Instruct | Larger production deployment. |
| Mistral 7B Instruct v0.3 | mistralai/Mistral-7B-Instruct-v0.3 | Efficient 7B instruct serving. |
| Qwen 2.5 7B Instruct | Qwen/Qwen2.5-7B-Instruct | Strong small model for coding. |
| Qwen 2.5 72B Instruct | Qwen/Qwen2.5-72B-Instruct | Large Qwen on multi-GPU vLLM. |
| DeepSeek V2.5 | deepseek-ai/DeepSeek-V2.5 | MoE model with vLLM support. |
Models and availability change over time. See the vLLM's supported models for the latest list and pricing.
[ TROUBLESHOOTING ]
| Error | Likely Cause | What to Do |
|---|---|---|
401 Unauthorized | Invalid or missing API key. | Verify your API key is correct. Generate a new key if needed. |
400 Bad Request | Invalid request format or unsupported model. | Check request format and confirm model ID is valid. |
429 Rate Limited | Rate limit exceeded for your plan. | Upgrade your plan or implement exponential backoff. Use Bifrost for intelligent load distribution. |
502/503 Service Error | Temporary Mistral service unavailability. | Retry after a delay. Check Mistral status page. Configure failover with Bifrost. |
[ PRODUCTION-READY ]
Bifrost is a drop-in replacement for vLLM SDKs: keep your client code and change the base URL to your gateway. Bifrost handles cost tracking, virtual keys, budgets, and failover automatically.
Run the Bifrost gateway and configure your Mistral credentials in the Web UI.
$ npx -y @maximhq/bifrost
✓ Bifrost started ├─ HTTP server listening on http://localhost:8080 ├─ Web UI available at http://localhost:8080 └─ Configure providers and virtual keys in the dashboard
Update your vLLM SDK client to route through the Bifrost gateway.
from openai import OpenAI client = OpenAI( api_key="sk-bf-your-virtual-key", base_url="http://localhost:8080/vllm" ) response = client.chat.completions.create( model="vllm/meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[{"role": "user", "content": "Hello from Bifrost!"}] ) print(response.choices[0].message.content)
x-bf-vk or Authorization: Bearer sk-bf-* per the Bifrost documentation.[ WHAT'S NEXT ]
You have your API key. Add governance, guardrails, and MCP controls for production.
[ BIFROST FEATURES ]
Everything you need to run AI in production, from free open source to enterprise-grade features.
01 Governance
SAML support for SSO and Role-based access control and policy enforcement for team collaboration.
02 Adaptive Load Balancing
Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics.
03 Cluster Mode
High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal.
04 Alerts
Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook and more.
05 Log Exports
Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export capabilities for compliance, monitoring, and analytics.
06 Audit Logs
Comprehensive logging and audit trails for compliance and debugging.
07 Vault Support
Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration.
08 VPC Deployment
Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls.
09 Guardrails
Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents.
[ SHIP RELIABLE AI ]
Change just one line of code. Works with OpenAI, Anthropic, Vercel AI SDK, LangChain, and more.
[ FAQ ]
vLLM is an open-source library for high-throughput LLM inference with PagedAttention and an OpenAI-compatible server.
Local vLLM often runs without auth. Put authentication and budgets at the Bifrost gateway instead.
An NVIDIA GPU with sufficient VRAM for your model is recommended. CPU-only setups are possible but slower.
Typically one model per server process. Run multiple vLLM instances or use Bifrost to route across hosts.
Register your vLLM base URL as a custom provider in Bifrost and route via http://localhost:8080.
Tune tensor parallelism, batch size, and GPU count per vLLM documentation.