Overview of the Automated SFTP Monitoring Setup

Businesses widely use SFTP (Secure File Transfer Protocol) for secure data exchange, but manually monitoring folders for new files can be inefficient. Automating this process can save time and ensure that data is processed promptly. In this guide, we’ll explore how to automate the monitoring of an SFTP folder using AWS Lambda, Cronicle, and Slack for notifications. This setup ensures that your system is always alert to new files, records them, and notifies users in real-time.

Setting Up AWS Lambda for SFTP Monitoring

AWS Lambda is ideal for creating a serverless solution to monitor SFTP folders. The process begins by setting up an AWS Lambda function that connects to your SFTP server using an appropriate library, such as paramiko, a Python SSH2 protocol library. Lambda can be configured to run periodically, checking the SFTP folder for new files and taking necessary actions.

  1. Create a Lambda function: Start logging into AWS Lambda and creating a new function using Python.
  2. Install paramiko: To include the paramiko library, use Lambda layers. You can package it into a .zip file or use pre-built layers from trusted sources.
  3. Set up the SFTP connection: In your Lambda function code, establish a secure connection to your SFTP server using SSH credentials or critical pairs.

import paramiko

def lambda_handler(event, context):

    sftp_host = ‘sftp.example.com’

    username = ‘user’

    password = ‘password’

    

    ssh = paramiko.SSHClient()

    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

    ssh.connect(sftp_host, username=username, password=password)

    

    sftp = ssh.open_sftp()

    files = sftp.listdir(‘/path/to/sftp/folder’)

    

    # Your logic to process files

    sftp.close()

    ssh.close()

Establishing Secure Connections and Checking for New Files

Security is paramount when connecting to an SFTP server. Ensure you use secure credentials and encryption keys to avoid any vulnerability. Once the connection is established, the Lambda function will list all files in the SFTP folder. You can extend the logic to track newly added files by comparing the current list of files with previously stored records.

Integrating with DynamoDB for File Record Keeping

AWS DynamoDB can store records of already processed files, preventing the system from repeatedly handling duplicate files. Each time the Lambda function runs, it checks the file list against DynamoDB to determine whether there are any new files to process.

  1. Create a DynamoDB Table: Use AWS DynamoDB to create a table for storing file records, with file names as primary keys.
  2. Insert file records: Store the file name and timestamp in the DynamoDB table after processing each file.

import boto3

dynamodb = boto3.resource(‘dynamodb’)

table = dynamodb.Table(‘SFTPFiles’)

def record_file(file_name):

    table.put_item(

        Item={

            ‘FileName’: file_name,

            ‘ProcessedTime’: int(time.time())

        }

    )

Notifying Users via Slack and Confirming Files

Integrating Slack for notifications ensures that your team is instantly aware when new files are available. AWS Lambda can send messages to a Slack channel using an incoming webhook or Slack API, providing real-time alerts to users.

  1. Set up a Slack webhook: Create an incoming webhook in your Slack workspace.
  2. Send notifications: Use the Lambda function to send a message whenever new files are detected.

import requests

def notify_slack(file_name):

    slack_url = ‘https://hooks.slack.com/services/your/webhook/url’

    message = f’New file detected: {file_name}’

    

    requests.post(slack_url, json={‘text’: message})

Automating Checks with Cronicle and Deploying Lambda Functions

Cronicle is an open-source job scheduler for periodic tasks, such as triggering the Lambda function to monitor the SFTP folder. With Cronicle, you can set specific intervals for running the checks, ensuring that Lambda only runs when necessary.

  1. Install and configure Cronicle: Deploy Cronicle on an EC2 instance or other server and configure it to trigger AWS Lambda at specified intervals.
  2. Set up Cronicle tasks: Create a task in Cronicle that triggers the Lambda function through the AWS SDK or via API Gateway integration.
  3. Deploy the Lambda function: Once the setup is complete, deploy the Lambda function to AWS. Ensure the function has sufficient permissions to access DynamoDB, Slack, and your SFTP server.

Conclusion

By integrating AWS Lambda, DynamoDB, Cronicle, and Slack, you can build a robust and automated system for monitoring SFTP folders, processing files, and notifying users in real time. This approach minimizes manual intervention, improves efficiency, and promptly handles data.

References

Using AWS Lambda with Amazon DynamoDB streams

Monitoring your AWS SFTP environment