Introduction: Running Open-Source LLMs on EKS

Large Language Models (LLMs) like Mistral 7B are revolutionizing the field of natural language processing (NLP) with their powerful text generation capabilities. Running these models on Kubernetes, specifically Amazon Elastic Kubernetes Service (EKS), allows for scalable and efficient deployment. This guide will explore setting up GPU-accelerated inference for open-source LLMs on AWS EKS.

Prerequisites: Tech Stack and Assumptions

Before we dive into the setup, let’s outline the prerequisites:

  • AWS Account: Ensure you have an AWS account with the necessary permissions.
  • EKS Cluster: A running EKS cluster.
  • Karpenter: For efficient node provisioning.
  • kubectl: Command-line tool for interacting with Kubernetes clusters.
  • NVIDIA GPUs: Required for accelerating model inference.

Making GPUs Accessible to Pods: Hardware and Software Configurations

To run GPU-accelerated workloads on EKS, you must ensure your cluster can utilize GPU nodes. This involves both hardware and software configurations:

  1. Hardware: Choose the appropriate NVIDIA GPUs based on your model’s requirements.
  2. Software: Install the necessary drivers and device plugins to expose GPU resources to your Kubernetes pods.

GPU Requirements: Identifying the Right GPU for Mistral 7B

The Mistral 7B model is resource-intensive. Based on its computational demands, consider the following GPUs:

  • NVIDIA Tesla V100: Ideal for high performance.
  • NVIDIA A100: Offers the best performance for large-scale models. Evaluate your budget and performance needs to select the most suitable GPU.

Provisioning GPU Nodes with Karpenter: Cost-Effective Strategies

Karpenter is an open-source node provisioning project that helps scale your Kubernetes cluster efficiently. To provision GPU nodes:

  1. Install Karpenter: Follow the official Karpenter installation guide.
  2. Configure GPU Instances: Define node templates that include GPU instance types, such as p3.2xlarge or p4d.24xlarge.

Exposing GPUs to Pods: Installing the NVIDIA K8S Device Plugin

To make GPUs available to your pods, you need the NVIDIA Kubernetes device plugin:

  1. Deploy NVIDIA Device Plugin:

    kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.9.0/nvidia-device-plugin.yml
  1. Verify Deployment: Ensure the plugin is running and exposing GPU resources.

Validating GPU Access: A Quick Test

Validate that your pods can access the GPU:

  1. Create a Test Pod:

    apiVersion: v1

kind: Pod

metadata:

  name: gpu-test

spec:

  containers:

  – name: cuda-container

    image: nvidia/cuda:10.0-base

    resources:

      limits:

        nvidia.com/gpu: 1

    command: [“nvidia-smi”]

  1. Deploy the Pod:

    kubectl apply -f gpu-test.yaml
  1. Check Logs:

    kubectl logs gpu-test

You should see the output of nvidia-smi, confirming GPU access.

Deploying the Text Generation Inference Server: Simple Kubernetes Manifests

Deploy the inference server using a Kubernetes manifest:

  1. Create Deployment Manifest:

    apiVersion: apps/v1

kind: Deployment

metadata:

  name: text-gen-server

spec:

  replicas: 1

  selector:

    matchLabels:

      app: text-gen

  template:

    metadata:

      labels:

        app: text-gen

    spec:

      containers:

      – name: text-gen

        image: your-docker-repo/text-gen-server:latest

        resources:

          limits:

            nvidia.com/gpu: 1

        ports:

        – containerPort: 5000

  1. Apply the Manifest:

    kubectl apply -f text-gen-server.yaml

Testing the Inference Server: A Basic Curl Command

Verify the inference server is working by sending a request:

curl -X POST “http://<your-server-ip>:5000/generate” -H “Content-Type: application/json” -d ‘{“text”: “Hello, world!”}’

Ensure the response contains the generated text, confirming the server’s functionality.

Bonus: Building a Local Chat Client Interface with Gradio

For an enhanced user experience, build a local chat client with Gradio:

  1. Install Gradio:

    pip install gradio
  1. Create a Simple Interface:

    import gradio as gr

def chat(text):

    # Replace with the endpoint of your inference server

    response = requests.post(“http://<your-server-ip>:5000/generate”, json={“text”: text})

    return response.json()[“generated_text”]

iface = gr.Interface(fn=chat, inputs=”text”, outputs=”text”)

iface.launch()

Conclusion

Deploying GPU-accelerated LLM inference on AWS EKS involves several steps, from selecting the proper GPU to validating the setup and deploying the inference server. Following this guide, you can efficiently run powerful LLMs like Mistral 7B on your Kubernetes cluster, ensuring scalability and performance.

References

Deploying generative AI applications with NVIDIA NIMs on Amazon EKS

Scale LLMs with PyTorch 2.0 FSDP on Amazon EKS