Self-hosted control plane for LLM inference Your own AI models, run like a service.

The control plane for open AI models on infrastructure you control. Bring a model in, deploy it, give your teams an endpoint, and keep it healthy.

Get early access

For platform teams that run AI for an organization, and for places where the data cannot leave the building.

Deployments

Every model, where it runs, and how it is doing.

Deploy model

Active deployments 12 across 3 clusters

Requests served 1.4M last 7 days

GPU utilization 68% 22 of 32 GPUs

Deployment Environment Status GPUs

llama-3.3-70b-instruct prod-us-east rolling out 8×H100

qwen3-32b prod-eu-west serving 2×H100

mistral-small-3.1-24b prod-us-east serving 1×A100

gemma-3-27b shared-pool paused 1×A100

Safe rollout llama-3.3-70b-instruct · 3.3 to 3.3-r2

Canary 10%SwappingSoakingPromoted

Ready in 41 s
Errors 0.1%, under 1%
p95 812 ms, under 1.5 s
Soak, 15 minutes

Rolls back on its own if a check fails.

The convenience of a provider, with the ownership of your own hardware.

Your data stays home

Prompts, documents and answers travel between your applications and your GPUs, and nowhere else.

An endpoint in minutes

Ask for a model, get an address and a key, and call it from the SDK you already use.

One console, every cluster

Models, clusters, endpoints, upgrades and cost, in one place instead of a pile of scripts.

Teams run internal assistants, search over their own documents, coding help, and extraction from files on it.

Three ways to run a model.

Rent one, build it yourself, or run it with Servary. What each choice costs you, in the things that matter.

	Hosted API	Do it yourself	Servary
Where your data goes	To the provider	Stays with you	Stays with you
Time to an endpoint	Minutes	Days per model	Minutes
What it costs	Per token, forever	Your GPUs, unwatched	Your GPUs, with the cost on the dashboard
Which model answers	Whatever the provider ships this month	The one you pinned	The one you pinned, every version recorded
Will the model fit	Not your problem	Find out at 3 AM	Checked against free GPU memory before anything is applied
Upgrading a model	When the provider decides	A maintenance window	A canary, checks, automatic rollback
Keys, limits and audit	Included	Build it yourself	Built in, per workspace
Air-gapped network	Not an option	Mirror everything by hand	Models from your registry or an upload, add-ons from your mirror
When the cluster drifts	Not your problem	Someone gets paged	Put back on its own

Where your data goes

Hosted API

To the provider

Do it yourself

Stays with you

Servary

Stays with you
Time to an endpoint

Hosted API

Minutes

Do it yourself

Days per model

Servary

Minutes
What it costs

Hosted API

Per token, forever

Do it yourself

Your GPUs, unwatched

Servary

Your GPUs, with the cost on the dashboard
Which model answers

Hosted API

Whatever the provider ships this month

Do it yourself

The one you pinned

Servary

The one you pinned, every version recorded
Will the model fit

Hosted API

Not your problem

Do it yourself

Find out at 3 AM

Servary

Checked against free GPU memory before anything is applied
Upgrading a model

Hosted API

When the provider decides

Do it yourself

A maintenance window

Servary

A canary, checks, automatic rollback
Keys, limits and audit

Hosted API

Included

Do it yourself

Build it yourself

Servary

Built in, per workspace
Air-gapped network

Hosted API

Not an option

Do it yourself

Mirror everything by hand

Servary

Models from your registry or an upload, add-ons from your mirror
When the cluster drifts

Hosted API

Not your problem

Do it yourself

Someone gets paged

Servary

Put back on its own

What a team stops doing.

Each of these is manageable alone. Together they are a platform team's full-time work, held in the heads of one or two people.

01
Writing a Helm chart per model, and rewriting it for every upgrade.
02
Guessing whether the 70B fits, and finding out at 3 AM.
03
Announcing a maintenance window to swap a model version.
04
Keeping inference keys in a spreadsheet and rotating them by hand.
05
Wiring DCGM, Prometheus and an HPA by hand on every cluster.
06
Being the one person who knows what runs where, and why.

From a model to a production endpoint, and everything after.

Five things Servary does for every model, in the order they happen.

Step 01

Bring a model in

From Hugging Face, your own registry, or an upload. Every version is checked and recorded, so you always know what is running where.
- · Hugging Face, a private registry, or files you upload
- · Every version checked and recorded
- · The same model, deployed the same way, anywhere
Add a model

Hugging Face Registry Upload

llama-3.3-70b-instruct imported

version 3.3 · 142 GB · sha256 9f3a…c21e

qwen3-32b importing imported

version 3 · 65 GB · sha256 41d0…8be7
Step 02

Deploy where it fits

Before anything goes live, Servary checks that the GPUs have room. No guessing, and no failed launch at 3 AM.
- · Memory checked against what is free on the cluster
- · A clear verdict before anything is applied
- · Native vLLM or through KServe, chosen per deployment
Will it fit? Fits on gpu-node-a
llama-3.3-70b-instruct needs 140 GB of GPU memory across 2 GPUs.

gpu-node-a · 8×H100 320 GB free

gpu-node-b · 4×A100 40 GB free

In use This model Checked before apply
Step 03

Give your teams an endpoint

One OpenAI-compatible address with keys and limits. Applications and SDKs work unchanged.
- · Works with any OpenAI SDK, streaming included
- · Keys with scopes, rate limits and timeouts
- · Revoke one key without touching the others
Endpoint Healthy
https://models.yourcompany.com/v1

Key sk_inf_••••••••7f2a chat, completions

Limits 600 requests per minute · 600 s timeout

Python

client = OpenAI( base_url="https://models.yourcompany.com/v1", api_key=key, ) client.chat.completions.create( model="llama-3.3-70b-instruct", messages=messages, )
Step 04

Upgrade without a maintenance window

The new version proves itself on a slice of the traffic before it takes over. If it fails, the old one was never touched.
- · A canary on a share of the traffic you choose
- · Checks on readiness, errors and latency
- · Automatic rollback, same address throughout
Upgrade llama-3.3-70b-instruct Swapping
Canary 10% of traffic
Checks errors 0.1%, p95 812 ms
Swap traffic moving over
Soak 15 minutes

Traffic on the new version 3.3 to 3.3-r2

The endpoint address never changes. A failed check rolls back on its own.
Step 05

Keep it that way

Scaling with demand, repairs when something drifts, live logs and metrics, and an alert when something needs a person.
- · More replicas when the GPUs are busy, fewer when idle
- · Changes made by hand on the cluster, put back
- · Email and webhooks when a person is needed
Today
GPU utilization · prod-us-east · last hour 68%

GPU load 84% for 5 min

Scaled prod-us-east to 4 replicas

09:14

Drift on qwen3-32b

Replicas set to 1 by hand, restored to 3

11:02

Upgrade promoted

llama-3.3-70b-instruct on 3.3-r2, soak passed

14:37

Alert sent

gemma-3-27b unhealthy for 2 checks, email and webhook

16:20

Add a model

Hugging Face Registry Upload

llama-3.3-70b-instruct imported

version 3.3 · 142 GB · sha256 9f3a…c21e

qwen3-32b importing imported

version 3 · 65 GB · sha256 41d0…8be7

Will it fit? Fits on gpu-node-a

llama-3.3-70b-instruct needs 140 GB of GPU memory across 2 GPUs.

gpu-node-a · 8×H100 320 GB free
gpu-node-b · 4×A100 40 GB free

In use This model Checked before apply

Endpoint Healthy

https://models.yourcompany.com/v1

Key sk_inf_••••••••7f2a chat, completions

Limits 600 requests per minute · 600 s timeout

Python

client = OpenAI(
  base_url="https://models.yourcompany.com/v1",
  api_key=key,
)
client.chat.completions.create(
  model="llama-3.3-70b-instruct",
  messages=messages,
)

Upgrade llama-3.3-70b-instruct Swapping

Canary 10% of traffic
Checks errors 0.1%, p95 812 ms
Swap traffic moving over
Soak 15 minutes

Traffic on the new version 3.3 to 3.3-r2

The endpoint address never changes. A failed check rolls back on its own.

Today

GPU utilization · prod-us-east · last hour 68%

GPU load 84% for 5 min

Scaled prod-us-east to 4 replicas

09:14
Drift on qwen3-32b

Replicas set to 1 by hand, restored to 3

11:02
Upgrade promoted

llama-3.3-70b-instruct on 3.3-r2, soak passed

14:37
Alert sent

gemma-3-27b unhealthy for 2 checks, email and webhook

16:20

Where it sits

Inside your network, between your applications and your GPUs.

Nothing leaves. Servary runs no agent in your clusters, and installs add-ons only when you choose to.

And everything else a production endpoint needs.

The work that usually lives in scripts, tickets and one person's head, done by the platform.

Keys and limits

Scoped keys, rate limits and timeouts, revoked one at a time.
Scale with demand

More replicas when the GPUs are busy, fewer when they are not.
Self-repair

Changes made by hand on the cluster are put back.
See everything

Live logs, latency, throughput and GPU load per deployment, a playground, and redacted request capture.
Alerts

Email and webhooks when something needs a person.
Workspaces

Teams isolated from each other, each with its own sign-in and roles.
Cost in view

A GPU price per workspace; utilization, the day's cost and cost per million tokens on the dashboard.
Air-gapped

Nothing calls out. Models from your registry, add-ons from your mirror.

Built for places where the data cannot leave.

Servary runs inside your network, reaches your clusters with the access you give it, runs no agent there, and keeps a record of every change.

01

Your identity provider

Single sign-on per workspace, enforceable, with two-factor for local accounts.
02

Access by role

Admins, operators and viewers, per workspace. Sessions you can see and revoke.
03

Every change on record

An audit log of who did what and when, for every action that changes anything.
04

Secrets encrypted

Encrypted at rest under a key held outside the database, rotated without downtime.
05

No agent in your clusters

Servary works with the access you grant. Optional add-ons install only when you choose to.
06

Nothing calls out

Models from your registry or an upload, add-ons from your mirror. Ready for air-gapped networks.

The open models teams ask for.

Servary runs any open model vLLM can serve, which is nearly every open model in use, and the list grows with each vLLM release. These are the families teams ask for most.

Llama 3.3 and 4

Meta

70B · Scout · Maverick
Qwen 3

Alibaba

0.6B to 235B
DeepSeek V3 and R1

DeepSeek

671B · distills 1.5B to 70B
Mistral Small and Large

Mistral AI

24B · 123B
Phi-4

Microsoft

14B
Gemma 3

Google

1B to 27B
Command-R+

Cohere

104B
Any model vLLM can serve. Bring your own from a registry or an upload.

Something else? Most additions are a configuration change, not a code change. Ask us.

Things people ask first.

What is Servary, exactly?

Servary is a self-hosted control plane for running open AI models on your own infrastructure. Point it at a model and at the clusters you run, and it handles the whole lifecycle: bringing the model in, deploying it, publishing an endpoint your applications call, and keeping it healthy from then on. No hosted service, no shared tenancy. Your models and your data stay inside your network.

Does my data leave my network?

No. Servary runs inside your network and makes no calls out of its own. Prompts, documents and responses travel between your applications and your GPUs, and nowhere else. Hugging Face imports need the internet; registry sources and uploads do not.

Which models can I serve?

Any open model vLLM can serve, which is nearly every open model in use: Llama, Qwen, DeepSeek, Mistral, Phi, Gemma, Command-R and many more. The list grows with each vLLM release, and a model that is not on it yet is usually a configuration change away.

Do I need Kubernetes?

Yes. Servary deploys to Kubernetes clusters you run, on-premises or in any cloud, with NVIDIA GPUs. It runs no agent in them, and installs optional add-ons only when you choose to. Other targets are on the roadmap.

What do I need to start?

A Kubernetes cluster with NVIDIA GPUs, on-premises or in a cloud, and a place to run the Servary console and its database, which is any machine that runs containers. We set the preview up with you.

Is Servary open source?

Yes. Servary will be open source. It is in private preview today, and the code will be published as we open it up, and we will share the timing as we get closer. Self-hosted from the start, so you keep the models, the traffic and the audit log either way.

Private preview

Bring a model. We will help you get it to an endpoint.

Servary is in private preview with a small group of teams. Here is how it goes.

Get early access

01

Tell us what you run

Which models, which clusters, and what your teams need from an endpoint.
02

We set you up

A preview install on your infrastructure, with us alongside for the first model.
03

Bring a model to an endpoint

From a registry to a production address your applications can call, in one session.

Self-hosted control plane for LLM inference Your own AI models, run like a service.

The convenience of a provider, with the ownership of your own hardware.

Your data stays home

An endpoint in minutes

One console, every cluster

Three ways to run a model.

What a team stops doing.

From a model to a production endpoint, and everything after.

Bring a model in

Deploy where it fits

Give your teams an endpoint

Upgrade without a maintenance window

Keep it that way

Inside your network, between your applications and your GPUs.

And everything else a production endpoint needs.

Keys and limits

Scale with demand

Self-repair

See everything

Alerts

Workspaces

Cost in view

Air-gapped

Built for places where the data cannot leave.

Your identity provider

Access by role

Every change on record

Secrets encrypted

No agent in your clusters

Nothing calls out