Run a large LLM across two GPU nodes on EKS to avoid single‑GPU limits

Deploying an LLM across two GPU nodes on Amazon EKS reduces the risk of hitting single‑GPU memory limits but adds cluster and dependency work you must manage.

What actually changed

AWS demonstrated deploying one large language model across two GPU nodes on Amazon Elastic Kubernetes Service (EKS) using the Ray LLM Deep Learning Containers. The example places separate prefill and decode processes on different GPU nodes and exposes them behind a single OpenAI‑compatible endpoint. The Ray Serve Deep Learning Container (DLC) can span multiple nodes, and the container images include the GPU stack (including CUDA libraries) so you do not need to reassemble those libraries yourself.

Who it affects

• Small ops or engineering teams running foundation or open‑weight models that do not fit on a single GPU.
• Teams that need more inference throughput than a single replica can deliver and are prepared to manage a multi‑node cluster.
• Teams that want to separate inference phases (for example, prefill vs decode) to improve performance or isolate resource profiles.

What it costs or what it replaces

• Replaces single‑node Ray Serve DLC deployments with a multi‑node Ray Serve setup on EKS when model size or throughput requires it. AWS previously showed a single‑GPU example; this extends that pattern to multiple GPUs and nodes.
• Reduces the time you spend assembling CUDA and related GPU libraries, because the Ray LLM Deep Learning Containers include the GPU stack. The announcement does not state pricing for EKS, instances, or DLC images; you must budget for GPU nodes and cluster management separately.
• Note that customers still need to build and verify the full dependency chain themselves: you remain responsible for choosing compatible libraries for your GPU stack and application code.

What we don't know

• The exact Ray LLM Deep Learning Container image names, tags or registry locations for the multi‑node setup.
• The Kubernetes manifests, Helm charts or CLI commands used to provision the example on EKS.
• Which GPU instance types or CUDA versions were used in the demonstration.
• How the example handles autoscaling, load balancing, or failure recovery across the two nodes.
• Performance numbers: latency, throughput, memory usage, or network overhead for prefill/decode disaggregation.
• Any changes to monitoring, logging or cost management required when moving from single‑node to multi‑node.

What to do next

1. Provision a short‑lived test EKS cluster with two GPU nodes and reserve a block of time this week to run experiments. Treat this as a costed engineering spike: expect to validate compatibility and deployment scripts rather than reach production readiness.
2. Start with a single‑node Ray Serve DLC deployment (the earlier AWS example) to confirm GPU stack compatibility in your environment, then extend the Ray Serve deployment to span the second node and separate the "prefill and decode" stages across the two nodes.
3. While testing, log exact container images, Kubernetes manifests, GPU instance types and CUDA versions you use; measure latency and throughput so you can decide whether to operate multi‑node inference in production or revert to single‑node replicas.