AWS EC2 - How To Install Logstash and Output Logs To AWS S3

This blog shows you how to install logstash in a AWS EC2 instance and configure it to send sample logs to S3.

Environments:

• logstash: 5.6.2
• AMI: ami-009d6802948d06e52

Spin Up EC2 Instance:

1. Spin up a EC2 instance from your AWS console, I use t2.large for example. The default disk size, security group settings should be fine. I only has SSH opened for ingress rules. egress has no restriction.

2. After the instance is up, ssh into the instance.

Install Logstash 5.6.2:

1. Download the Logstash 5.6.2

$ cd /root
$ wget https://artifacts.elastic.co/downloads/logstash/logstash-5.6.2.rpm

2. Install Java-1.8.0

$ yum install java-1.8.0

3. Install Logstash

$ rpm -iUh logstash-5.6.2.rpm
warning: logstash-5.6.2.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
################################# [100%]
Updating / installing...
################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash

Configure Logstash:

1. I've created an working sample "test.conf" below, you will need to create the dummy "/tmp/yum.log" file.

$ cat test.conf
input {
  file {
    path => "/tmp/yum.log"
  }
}

output {
   s3 {
     access_key_id => "money_key"
     secret_access_key => "monkey_secret"
     region => "us-east-1"
     bucket => "tony-logstash-test"
     prefix => "test/"
   }
}

Note: It is okay to have "/" in your secret

2. Make sure there is no existing data:

$ aws s3 ls s3://tony-logstash-test

3. Output the logfile to the S3 bucket:

$ sudo /usr/share/logstash/bin/logstash -f test.conf --path.settings=/etc/logstash
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties

Note: It is important to use sudo here, otherwise you might run into permission erros:

main ERROR FileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException:
/var/log/logstash/logstash-plain.log (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)

4. Depends on the size of the file, it might take some time. You can also check the Logstash log file for progress:

$ less /var/log/logstash/logstash-plain.log
[2018-12-15T17:27:06,468][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
 ...
[2018-12-15T17:27:06,631][INFO ][logstash.pipeline        ] Pipeline main started
[2018-12-15T17:27:06,675][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

You should be able to see the output data in your S3 bucket.

Once you are fine with the test.conf, you can move it to "/etc/logstash/conf.d" and have Logstash running as a background process permanently.