InferenceService

MaxReplicas is the upper limit for the number of replicas.

MinReplicas is the lower limit for the number of replicas.

Path is the HTTP path for the inference endpoint

Port is the service port

Type is the Kubernetes service type (ClusterIP, NodePort, LoadBalancer)

Name of the environment variable. May consist of any printable ASCII characters except '='.

Variable references $(VAR_NAME) are expanded using the previously defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "".

Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

CPUOffload enables model weight offloading to system RAM when GPU VRAM is insufficient. Requires the accelerate package in the container image.

Quantize4Bit enables NF4 4-bit quantization for reduced VRAM usage (~9.6 GB vs ~14 GB). Requires the bitsandbytes package in the container image.

A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod: 1. The owning GID will be the FSGroup 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) 3. The permission bits are OR'd with rw-rw---- If unset, the Kubelet will not modify the ownership and permissions of any volume. Note that this field cannot be set when spec.os.name is windows.

fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. Note that this field cannot be set when spec.os.name is windows.

The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows.

Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.

The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows.

seLinuxChangePolicy defines how the container's SELinux label is applied to all volumes used by the Pod. It has no effect on nodes that do not support SELinux or to volumes does not support SELinux. Valid values are "MountOption" and "Recursive". "Recursive" means relabeling of all files on all Pod volumes by the container runtime. This may be slow for large volumes, but allows mixing privileged and unprivileged Pods sharing the same volume on the same node. "MountOption" mounts all eligible Pod volumes with `-o context` mount option. This requires all Pods that share the same volume to use the same SELinux label. It is not possible to share the same volume among privileged and unprivileged Pods. Eligible volumes are in-tree FibreChannel and iSCSI volumes, and all CSI volumes whose CSI driver announces SELinux support by setting spec.seLinuxMount: true in their CSIDriver instance. Other volumes are always re-labelled recursively. "MountOption" value is allowed only when SELinuxMount feature gate is enabled. If not specified and SELinuxMount feature gate is enabled, "MountOption" is used. If not specified and SELinuxMount feature gate is disabled, "MountOption" is used for ReadWriteOncePod volumes and "Recursive" for all other volumes. This field affects only Pods that have SELinux label set, either in PodSecurityContext or in SecurityContext of all containers. All Pods that use the same volume should use the same seLinuxChangePolicy, otherwise some pods can get stuck in ContainerCreating state. Note that this field cannot be set when spec.os.name is windows.

A list of groups applied to the first process run in each container, in addition to the container's primary GID and fsGroup (if specified). If the SupplementalGroupsPolicy feature is enabled, the supplementalGroupsPolicy field determines whether these are in addition to or instead of any group memberships defined in the container image. If unspecified, no additional groups are added, though group memberships defined in the container image may still be used, depending on the supplementalGroupsPolicy field. Note that this field cannot be set when spec.os.name is windows.

Defines how supplemental groups of the first container processes are calculated. Valid values are "Merge" and "Strict". If not specified, "Merge" is used. (Alpha) Using the field requires the SupplementalGroupsPolicy feature gate to be enabled and the container runtime must implement support for this feature. Note that this field cannot be set when spec.os.name is windows.

CPU requests (e.g., "2" or "2000m")

GPU count required per pod For multi-GPU inference, each pod gets this many GPUs Note: Multi-GPU sharding config comes from Model CRD

GPUMemory specifies GPU memory limit per pod (e.g., "16Gi") Used for scheduling and validation

HostMemory specifies the system RAM required for hybrid GPU/CPU offloading (e.g., "64Gi"). Used when MoE expert weights or KV cache are offloaded to CPU via moeCPUOffload or noKvOffload. Translated to pod resources.requests.memory, taking precedence over Memory when set. Without this, the K8s scheduler has no visibility into the pod's actual RAM consumption, which can lead to OOM kills after model load.

Memory requests (e.g., "4Gi")

Factor is the scale multiplier (--rope-scale), e.g. "2.0" to double the native context. A string to avoid CRD float pitfalls; the runtime parses it as a float. Optional.

OriginalContext is the model's native training context length (--yarn-orig-ctx), e.g. 131072 for a 128K model. Recommended with yarn.

Type is the scaling method (--rope-scaling). "yarn" is the usual choice for extending context (e.g. 128K to 256K).

AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no_new_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is: 1) run as Privileged 2) has CAP_SYS_ADMIN Note that this field cannot be set when spec.os.name is windows.

Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false. Note that this field cannot be set when spec.os.name is windows.

procMount denotes the type of proc mount to use for the containers. The default value is Default which uses the container runtime defaults for readonly paths and masked paths. Note that this field cannot be set when spec.os.name is windows.

Whether this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows.

The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows.

Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.

The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows.

Dtype sets the model data type (float16, bfloat16).

MaxInputLength sets the maximum input token length.

MaxTotalTokens sets the maximum total tokens (input + output).

Quantize sets the quantization method (bitsandbytes, gptq, awq, eetq).

Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.

Key is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be Exists; this combination means to match all values and all keys.

Operator represents a key's relationship to the value. Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal. Exists is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category. Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).

TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system.

Value is the taint value the toleration matches to. If the operator is Exists, the value should be empty, otherwise just a regular string.

AttentionBackend selects the attention implementation used by vLLM. FLASHINFER is typically fastest on recent NVIDIA GPUs (especially Blackwell); FLASH_ATTN is a solid default; XFORMERS and torch_sdpa are portability fallbacks. Requires a vLLM version that supports the chosen backend. Both uppercase (vLLM's native form) and lowercase spellings are accepted for backwards compatibility with earlier LLMKube releases. Maps to vLLM --attention-backend flag.

CPUOffloadGB increases the GPU memory size. When set, passes --cpu-offload-gb to vLLM. Per-rank, so 4 on TP=2 means 4 GB of CPU RAM per GPU. Use when FP8 model weights don't fit VRAM. Throughput hit is 2-5x on the offloaded path.

Dtype sets the model data type (auto, float16, bfloat16).

EnableChunkedPrefill interleaves long prefills with decode steps so a large paste (e.g. a 32K-token file) does not starve concurrent decode streams. Only emitted when explicitly set to true. Maps to vLLM --enable-chunked-prefill flag.

EnableExpertParallel distributes MoE experts across tensor-parallel ranks instead of replicating them. Only meaningful for MoE models. Maps to vLLM --enable-expert-parallel flag.

EnablePrefixCaching turns on vLLM's automatic prefix caching for repeated prompts. Significantly reduces time-to-first-token for conversational and agentic workloads where requests share a common system prompt. Only emitted when explicitly set to true — when nil or false, vLLM's own default is used (do not emit the flag). Maps to vLLM --enable-prefix-caching flag.

GPUMemoryUtilization controls how much GPU memory each stage can use. When set, passes --gpu-memory-utilization to vLLM. Range from 0.1 - 0.99 and default unset (vLLM uses 0.90).

KVCacheCustomDtype sets a custom vLLM KV cache element type that is not in the standard enum. Used for vLLM versions with additional cache formats such as TurboQuant 2-bit (turbo2, shipped in v0.20.0). Maps to vLLM --kv-cache-dtype. The runtime image must understand the value or vLLM will fail to start; LLMKube does not validate the string. Mirrors the llama.cpp-side CacheTypeCustomK/V escape hatch. Takes precedence over KVCacheDtype when both are set.

KVCacheDtype selects the KV cache element type. fp8_e5m2 and fp8_e4m3 cut KV cache memory roughly in half versus auto (which follows dtype), which is what unlocks 128K+ context on consumer VRAM for agentic workloads. Maps to vLLM --kv-cache-dtype flag. For custom build types not in the enum (e.g. TurboQuant turbo2 from vLLM v0.20+), use KVCacheCustomDtype instead.

MaxModelLen sets the maximum model context length.

MaxNumBatchedTokens sets the maximum number of tokens batched together per step. This is the main throughput knob: too low means prefill-bound, too high risks OOM on long context. No default — only emitted when set. Maps to vLLM --max-num-batched-tokens flag.

Quantization method. awq, gptq, squeezellm are classic 4-bit formats. fp8 targets 8-bit FP checkpoints (Qwen FP8, Llama FP8, etc.). nvfp4 is NVIDIA's Blackwell-native 4-bit format. compressed-tensors is the neuralmagic/vLLM cross-format loader used by Unsloth and other recent releases.

TensorParallelSize sets the number of GPUs for tensor parallelism.

lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.

message is a human readable message indicating details about the transition. This may be an empty string.

observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.

reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.

status of the condition, one of True, False, Unknown.

type of condition in CamelCase or in foo.example.com/CamelCase.