Introduction: Shipping Logs from ECS to OpenSearch
Shipping logs from Amazon ECS (Elastic Container Service) to OpenSearch (formerly Elasticsearch) can significantly enhance your ability to analyze, monitor, and troubleshoot your applications. By leveraging FireLens, you can seamlessly forward your ECS task logs to OpenSearch, making it easier to manage and visualize your log data.
Prerequisites: IAM Roles and Policies for ECS Task Execution
Before diving into the configuration, ensure you have the following prerequisites in place:
- An AWS account with administrative access.
- IAM roles and policies for ECS task execution.
- An ECS cluster with running tasks.
- An OpenSearch domain set up.
Creating IAM Roles and Policies
- Create an IAM Role for ECS Task Execution: This role allows ECS tasks to use AWS services.
aws iam create-role –role-name ecsTaskExecutionRole –assume-role-policy-document file://ecs-trust-policy.json
- Attach the required policies:
aws iam attach-role-policy –role-name ecsTaskExecutionRole –policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
Task Definition: Configuring FireLens and CloudWatch Logging
- Define a new ECS Task: Ensure your task definition includes the necessary FireLens configuration.
{
“family”: “your-task-family”,
“containerDefinitions”: [
{
“name”: “your-container”,
“image”: “your-image”,
“logConfiguration”: {
“logDriver”: “awsfirelens”,
“options”: {
“Name”: “es”,
“Host”: “https://your-opensearch-domain”,
“Port”: “443”,
“Index”: “your-index”,
“Type”: “_doc”
}
}
}
]
}
Application Configuration: OpenSearch Log Configuration with FireLens
Ensure your application container is configured to use the FireLens log driver:
“logConfiguration”: {
“logDriver”: “awsfirelens”,
“options”: {
“Name”: “es”,
“Host”: “https://your-opensearch-domain”,
“Port”: “443”,
“Index”: “your-index”,
“Type”: “_doc”
}
}
OpenSearch Domain Setup: Creating an Index and Configuring Access
- Create an OpenSearch Domain: Set up a domain in the AWS Management Console.
- Configure the Index: Create a new index for your ECS logs.
PUT /your-index
{
“settings”: {
“number_of_shards”: 1,
“number_of_replicas”: 1
}
}
Web UI Configuration: Setting up FireLens and Application Containers
Configure your ECS service to use the task definition that includes FireLens and ensure the containers are correctly set up to forward logs.
Fine-Grained Access Control: Mapping IAM Roles to OpenSearch
- Set Up Fine-Grained Access Control: Enable fine-grained access control on your OpenSearch domain.
- Map IAM Roles: Map your ECS task IAM role to the appropriate OpenSearch roles for access.
PUT _opendistro/_security/api/rolesmapping/opensearch_role
{
“backend_roles”: [“arn:aws:iam::your-account-id:role/ecsTaskExecutionRole”]
}
Testing and Verification: Confirming Successful Log Shipping in OpenSearch
- Deploy the ECS Task: Ensure your service runs with the updated task definition.
- Verify Logs in OpenSearch: Check the OpenSearch index to confirm logs are being shipped.
GET /your-index/_search
{
“query”: {
“match_all”: {}
}
}
Conclusion and Additional Resources: Further Learning and Exploration
Shipping logs from ECS to OpenSearch with FireLens provides a powerful log management and analysis solution. This guide will help you build a robust logging pipeline to enhance your monitoring capabilities.
References
Create a custom log parser for Amazon ECS using a Firelens log router