In this tutorial, we’ll walk through the steps to ingest incoming Gmail messages to AWS S3 using AWS Lambda and CloudWatch alarms. This setup will help you archive your emails securely and access them as needed. Let’s get started.

Prerequisites

  1. AWS Account: Ensure you have an AWS account.
  2. Gmail Account: A Gmail account to access incoming emails.
  3. AWS CLI: Install and configure the AWS CLI.
  4. IAM Role: Create an IAM role with the permissions for Lambda and S3.

Step 1: Setting Up Gmail API

  1. Enable Gmail API: Go to the Google Cloud Console, create a new project, and enable the Gmail API.
  2. Create Credentials: Navigate to “APIs & Services” > “Credentials” and create OAuth 2.0 credentials. Download the JSON file containing your credentials.

Step 2: Setting Up AWS S3

  1. Create S3 Bucket: Go to the AWS S3 console and create a new bucket to store your emails.
  2. Bucket Policy: Configure the bucket policy to allow the Lambda function to write to it.

Step 3: Configuring AWS Lambda

  1. Create Lambda Function:
    • Go to the AWS Lambda console and create a new function.
    • Choose the Python runtime.
    • Assign the IAM role created earlier.
  2. Upload Gmail API Credentials:
    • Upload the JSON file with your Gmail API credentials to your Lambda function as an environment variable or securely via AWS Secrets Manager.
  3. Lambda Code:
    • Use the following Python code as a starting point for your Lambda function:

import base64

import boto3

from googleapiclient.discovery import build

from google.oauth2.credentials import Credentials

import os

s3 = boto3.client(‘s3’)

BUCKET_NAME = ‘your-s3-bucket-name’

def get_gmail_service():

    credentials = Credentials.from_authorized_user_file(‘/path/to/credentials.json’)

    service = build(‘gmail’, ‘v1’, credentials=credentials)

    return service

def save_email_to_s3(email_data):

    email_id = email_data[‘id’]

    raw_email = email_data[‘raw’]

    s3.put_object(Bucket=BUCKET_NAME, Key=f’emails/{email_id}.txt’, Body=base64.urlsafe_b64decode(raw_email))

def lambda_handler(event, context):

    service = get_gmail_service()

    results = service.users().messages().list(userId=’me’, q=’is:unread’).execute()

    messages = results.get(‘messages’, [])

    for message in messages:

        msg = service.users().messages().get(userId=’me’, id=message[‘id’], format=’raw’).execute()

        save_email_to_s3(msg)

        # Mark the email as read after processing

        service.users().messages().modify(userId=’me’, id=message[‘id’], body={‘removeLabelIds’: [‘UNREAD’]}).execute()

Step 4: Setting Up CloudWatch Alarm

  1. Create a CloudWatch Event Rule:
    • Go to the CloudWatch console and create a new event rule.
    • Configure the rule to trigger the Lambda function at a regular interval (e.g., every 5 minutes).
  2. Add Target:
    • Add the Lambda function as the target for the event rule.

Step 5: Testing and Deployment

  1. Deploy Lambda Function: Deploy your Lambda function with the necessary dependencies.
  2. Test the Setup: Send a test email to your Gmail account and verify that it appears in your S3 bucket.

Conclusion

Following these steps, you’ve successfully set up a system to ingest incoming Gmail messages to AWS S3 using AWS Lambda and CloudWatch alarms. This solution ensures that your emails are securely archived and accessible when needed.