Using Logstash to monitor VPC flow logs

One of the common ways to monitor activity in a AWS environment is using VPC flow logs. AWS natively supports sending VPC flow logs to CloudTrail and a S3 bucket. When sending to CloudTrail it is very easy to use GuardDuty to monitor what is going on in the VPC.

What we will examine here is sending the logs to an S3 bucket. Once the logs are in a bucket, you have a few choices:

  • Analyzing them using GuardDuty
  • Using Amazon Athena to query the logs
  • Keeping them in S3 simply for retention purposes
  • Sending them to your SIEM

In this case we will look at sending them to a SIEM for analysis. One way to do that is using Logstash to ship the logs to a ELK stack.

To do this, you can use the Logstash S3 plugin to download the logs to your local instance.

In order to use the Logstash plugin you must enable it using the following command:
/usr/share/logstash/bin/logstash-plugin install logstash-input-s3

This will allow you to pull the logs from your S3 bucket, now we must create the Logstash config to utilize this plugin. Here is an example:

Create the necessary config in /etc/logstash/conf.d/

input { 
  s3 {
    bucket => “VPC_BUCKET_NAME”
    role_arn => “ARN_HERE”
    region => “REGION”

filter {
  csv {
    separator => " "
    columns => [version, account_id, interface_id, srcaddr, dstaddr, srcport, dstport, protocol, packets, bytes, start, end, action, log_status]

output {
 elasticsearch {
   hosts => [“Elasticsearch nodes here“]
   index => 'aws-vpclogs-%{+yyyy.MM.dd}'

You will notice that in addition to the input plugin we need an output plugin with the destination for the logs. We also need to convert the logs into json format, as by default the flow logs do not come in json.

@mroberts, you need use markup

You can put text between two ` or two pairs of ```

this is one type of markup

This is another type