**Running Self-Hosted LLM Inference in Kubernetes with vLLM and LINSTOR**
Organizations are increasingly exploring how to run large language model (LLM) workloads locally rather than relying solely on managed APIs. Running models in-house offers benefits such as cost predictability at high request volumes, greater control over latency, and the ability to meet data residency requirements for regulatory or contractual compliance. A practical approach for many teams is a hybrid architecture: using self-hosted open source models for high-volume or sensitive workloads while leveraging managed APIs for tasks that benefit from external capabilities.
This article walks through setting up a self-hosted LLM inference stack in a Kubernetes lab environment using **vLLM** for inference and **LINBIT Storage (LINSTOR)** for persistent storage.
—
### Background on vLLM
**vLLM** is an open source, high-performance inference engine designed for serving large language models efficiently in cluster environments. One of its key advantages is that it exposes an **OpenAI-compatible REST API**, meaning that applications already built to use the OpenAI API can be redirected to a self-hosted instance with only a URL change. This compatibility makes it an ideal choice for hybrid AI architectures.
—
### Overview of the Setup
The instructions in this article assume a Kubernetes cluster with **LINSTOR** providing persistent storage through the **LINSTOR Container Storage Interface (CSI) driver**. Kubernetes orchestrates the entire stack, while LINSTOR—built on DRBD®—provides replicated block storage. This replication is particularly important for model weight files, which can be tens of gigabytes in size and must survive pod rescheduling or node failures.
The example model used is **`meta-llama/Llama-3.2-1B-Instruct`**, a lightweight, instruction-fine-tuned model from Meta. Because it has only 1 billion parameters, it can run on CPU, making it suitable for lab environments without GPU nodes.
—
### Prerequisites
Before deployment, you’ll need:
1. **Access to the model** – Meta Llama models are gated on Hugging Face. You must:
– Create a Hugging Face account
– Request access to the model
– Generate an access token with read permissions
2. **LINSTOR deployed in Kubernetes** – Use the **Piraeus Operator** to deploy LINSTOR and its CSI driver. You’ll need a StorageClass named `linstor-csi-lvm-thin-r2`.
—
### Deployment
The deployment requires three Kubernetes resources:
– A **PersistentVolumeClaim (PVC)** for model storage
– A **Secret** containing your Hugging Face token
– A **Deployment** and **Service** for the vLLM inference server
#### Creating the PVC and Secret
The PVC uses the `linstor-csi-lvm-thin-r2` StorageClass, which provisions a thin-provisioned LVM volume with two replicas for redundancy. The model weights are cached in `/root/.cache/huggingface`, so persisting this path ensures fast restarts and avoids re-downloading.
“`bash
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: vllm-models
spec:
storageClassName: linstor-csi-lvm-thin-r2
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 50Gi
---
apiVersion: v1
kind: Secret
metadata:
name: hf-token-secret
type: Opaque
stringData:
token: "REPLACE_WITH_YOUR_TOKEN"
EOF
```> ⚠️ Replace `REPLACE_WITH_YOUR_TOKEN` with your actual Hugging Face token and verify the StorageClass name.
#### Deploying vLLM
“`bash
VLLM_IMAGE=public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:latest
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-server
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: vllm
template:
metadata:
labels:
app.kubernetes.io/name: vllm
spec:
containers:
- name: vllm
image: $VLLM_IMAGE
command: ["/bin/sh", "-c"]
args:
- "vllm serve meta-llama/Llama-3.2-1B-Instruct --gpu-memory-utilization 0.80"
env:
- name: HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
key: token
ports:
- containerPort: 8000
volumeMounts:
- name: llama-storage
mountPath: /root/.cache/huggingface
volumes:
- name: llama-storage
persistentVolumeClaim:
claimName: vllm-models
---
apiVersion: v1
kind: Service
metadata:
name: vllm-server
spec:
selector:
app.kubernetes.io/name: vllm
ports:
- protocol: TCP
port: 8000
targetPort: 8000
type: ClusterIP
EOF
```> 💡 **Note:** Even in CPU-only deployments, vLLM requires `–gpu-memory-utilization` to be set. Without it, memory defaults can cause startup failures. A value of `0.80` provided sufficient headroom in testing.
#### Watching the Logs
The first startup can take several minutes as the model (~2.5GB) downloads from Hugging Face. Monitor progress with:
“`bash
kubectl logs -f deployment/vllm-server
“`
Once you see `INFO: Application startup complete`, the server is ready.
—
### Testing the Deployment
To test from inside the cluster, deploy a temporary curl pod:
“`bash
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: curl-client
spec:
containers:
- name: curl
image: curlimages/curl:latest
command: ["sleep", "infinity"]
restartPolicy: Never
EOF
```Then interact with it:```bash
kubectl exec -it curl-client -- sh
```Send a test request:```bash
curl
-H "Content-Type: application/json"
-d '{
"model": "meta-llama/Llama-3.2-1B-Instruct",
"messages": [
{"role": "user", "content": "What is the capital of France?"}
]
}'
```Expected response (OpenAI format):```json
{
"choices": [
{
"message": {
"role": "assistant",
"content": "The capital of France is Paris."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 15,
"completion_tokens": 9,
"total_tokens": 24
}
}
```When finished, clean up:```bash
kubectl delete pod curl-client
```---### Pausing the DeploymentTo pause the deployment temporarily (e.g., for debugging), scale it to zero:```bash
kubectl scale deployment vllm-server --replicas=0
```✅ **Tip:** The PVC and cached model weights remain intact. Restarting is as simple as scaling back up:```bash
kubectl scale deployment vllm-server --replicas=1
```---### ConclusionThis lab demonstrates a foundational pattern for self-hosted LLM inference in Kubernetes:
- A vLLM inference server
- Persistent model storage via LINSTOR
- An OpenAI-compatible API endpointBy storing model weights on a replicated LINSTOR volume, pod restarts are fast, and the storage layer avoids becoming a single point of failure.From here, you can:
- Add GPU nodes for improved performance
- Fine-tune models on private datasets
- Introduce an inference router (such as **llm-d**) to intelligently route requests between local and managed APIs based on cost, latency, or capabilityFor questions or to discuss LINBIT solutions for containerized AI workloads, join the **LINBIT Community Forum**.
