Introduction

Amazon Elastic Kubernetes Service (EKS) generates logs that provide valuable insights into cluster health, application performance, and security. Sending these logs to AWS OpenSearch Service (formerly Elasticsearch) enables real-time monitoring and analysis. This guide outlines the step-by-step process to configure Fluent-Bit for efficient log forwarding from Amazon EKS to AWS OpenSearch.

Prerequisites

Before setting up Fluent-Bit, ensure the following:

  • An active AWS account with Amazon EKS and AWS OpenSearch Service configured.
  • A running EKS cluster.
  • The Kubernetes command-line tool (kubectl) installed and configured.
  • AWS CLI installed and authenticated.

Step 1: Deploy Fluent-Bit on Amazon EKS

Fluent-Bit is a lightweight and highly efficient log processor that helps forward logs from Amazon EKS to AWS OpenSearch.

  1. Create a Kubernetes namespace for logging:

kubectl create namespace logging

  1. Create a Fluent-Bit ConfigMap:


apiVersion: v1

kind: ConfigMap

metadata:

  name: fluent-bit-config

  namespace: logging

data:

  fluent-bit.conf: |

    [SERVICE]

        Flush         5

        Log_Level     info

    

    [INPUT]

        Name         tail

        Path         /var/log/containers/*.log

        Parser       docker

        Tag         kube.*

    [OUTPUT]

        Name        opensearch

        Match       *

        Host        <opensearch-domain>

        Port        443

        TLS         On

        AWS_Auth    On

       Region      <aws-region>

  • Apply this configuration:

kubectl apply -f fluent-bit-config.yaml

  1. Deploy Fluent-Bit as a DaemonSet:


apiVersion: apps/v1

kind: DaemonSet

metadata:

  name: fluent-bit

  namespace: logging

spec:

  selector:

    matchLabels:

      name: fluent-bit

  template:

    metadata:

      labels:

        name: fluent-bit

    spec:

      serviceAccountName: fluent-bit

      containers:

      – name: fluent-bit

        image: fluent/fluent-bit:latest

        volumeMounts:

        – name: varlog

          mountPath: /var/log

        – name: config-volume

          mountPath: /fluent-bit/etc

      volumes:

      – name: varlog

        hostPath:

          path: /var/log

      – name: config-volume

        configMap:

         name: fluent-bit-config

  • Apply the DaemonSet configuration:

Kubectl apply -f fluent-bit-daemonset.yaml

Step 2: Grant Fluent-Bit IAM Permissions

To allow Fluent-Bit to send logs securely to AWS OpenSearch, attach the necessary IAM permissions.

  1. Create an IAM policy:

aws iam create-policy –policy-name FluentBitOpenSearchPolicy –policy-document file://fluent-bit-policy.json

  1. Attach the policy to a new IAM role:

aws iam create-role –role-name FluentBitRole –assume-role-policy-document file://trust-policy.json

aws iam attach-role-policy –role-name FluentBitRole –policy-arn arn:aws:iam::<account-id>:policy/FluentBitOpenSearchPolicy

  1. Annotate the Kubernetes service account to use this IAM role:

kubectl annotate serviceaccount fluent-bit -n logging eks.amazonaws.com/role-arn=arn:aws:iam::<account-id>:role/FluentBitRole

Step 3: Verify Logs in AWS OpenSearch

Once Fluent-Bit is running, logs should begin streaming into AWS OpenSearch. To verify:

  1. Log into the AWS Management Console.
  2. Navigate to AWS OpenSearch Service.
  3. Open Kibana and search for logs using:

{ “query”: { “match_all”: {} } }

Conclusion

By following these steps, Amazon EKS logs can be efficiently collected and stored in AWS OpenSearch using Fluent-Bit. This setup enhances observability, enabling proactive troubleshooting and real-time analysis of Kubernetes workloads.