Sunday, August 04, 2019

AWS CDK Python - Sample Project

This blog shows you how to create a sample project using AWS CDK Python. If you don't have the environment setup yet, check out my previous blog: https://www.lixu.ca/2019/08/aws-cdk-python-how-to-get-started.html

Create the Python Application

1. Fire up you Python3 virtual environment, create a project directory contains your Python project source code
$ source cdk-env/bin/activate
(cdk-venv) $ mkdir sample-dev
(cdk-venv) $ cd sample-dev
(cdk-venv) $ cdk init
Available templates:
* app: Template for a CDK Application
   └─ cdk init app --language=[csharp|fsharp|java|javascript|python|typescript]
* lib: Template for a CDK Construct Library
   └─ cdk init lib --language=typescript
* sample-app: Example CDK Application with some constructs
   └─ cdk init sample-app --language=[javascript|python|typescript]
 
2. The return information from cdk is very self-explanatary, we can choose "sample-app" so by the creation of this app, we have some exisitng constructs
$ cdk init --language python sample-app
Applying project template sample-app for python
Initializing a new git repository...
Executing Creating virtualenv...
...

3. The entire sample project looks like this:
.
├── README.md - Readme file
├── app.py    - Defines app stacks, also the "main" file
├── cdk.json  - Configuration file for CDK that defines what executable CDK should run to generate CDK construct tree
├── hello
│   ├── __init__.py         - To make a Python module
│   ├── hello.egg-info
│   │   ├── PKG-INFO
│   │   ├── SOURCES.txt
│   │   ├── dependency_links.txt
│   │   ├── requires.txt
│   │   └── top_level.txt
│   ├── hello_construct.py  - A custom CDK construct defined for use in your CDK application.
│   └── hello_stack.py      - A custom CDK stack construct for use in your CDK application
├── requirements.txt     - Required Python modules
├── setup.py             - Defines how this Python package would be constructed and what the dependencies are
└── tests                - Test folder
    ├── __init__.py
    └── unit             - ALl the unit tests
        ├── __init__.py
        └── test_hello_construct.py

You can take time to read and understand the source code, but esentially what this code does is to create two CloudFormation templates: "hello-cdk-1" and "hello-cdk-2", and deploy them to "us-east-1" and "us-west-1" regions. Each template includes:

  • IAM user with predefined policies
  • AWS sqs queue
  • AWS sns topic and subscribe to the sqs queue
  • Few AWS S3 buckets


4. Give a read of the README.md file, you can ignore the Python virtual env part, as we are already using a virtual env. As the README.md suggest, we going to install all required Python packages and run some unit tests.
(cdk-venv) $ pip install -r requirements.txt
Collecting pytest (from -r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/69/1d/2430053122a3c6106f7fd1ff0bc68eb73e27db8f951db70fcd942da52c7b/pytest-5.0.1-py3-none-any.whl (221kB)
    100% |████████████████████████████████| 225kB 4.5MB/s 

(cdk-venv) $ pytest
================================================================================================================= test session starts
=================================================================================================================
platform darwin -- Python 3.7.1, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /Users/txu/code/flashhop-dev
collected 1 item                                                                                                                                                                                                                                      

tests/unit/test_hello_construct.py . 

5. Generate AWS CloudFormation template
(cdk-venv) $ cdk synth
Successfully synthesized to /Users/txu/code/flashhop-dev/cdk.out
Supply a stack name (hello-cdk-1, hello-cdk-2) to display its template.
All the output files will be located in "cdk.out"

6. Deploy the generated templates to AWS account and validate the right resouces got created.
(cdk-venv) $ cdk deploy --profile YOU_AWS_PROFILE hello-cdk-1
...





7. To destroy the stack
(cdk-venv) $ cdk --profile flashhop-dev destroy hello-cdk-1
Are you sure you want to delete: hello-cdk-1 (y/n)? y
hello-cdk-1: destroying...


At this point, you learned how to use AWS CDK CLI to initialize a new Python project/app, how to synthesize hte Python code into AWS Cloudformation templates, deploy them, provision your infrastructure and destroy your infrastructure. You can learn more from AWS CDK developer documentations (https://docs.aws.amazon.com/cdk/latest/guide/home.html)

AWS CDK Python - How To Get Started

Assuming you are already familiar with the concept of "IaC" (Infrastructure As Code), well in AWS context, "IaC" means CloudFormation.

CloudFormation allows you to define your AWS infrastructure in JSON or YAML files that can be managed witin your source code repository (Git for example). You can do pull requests and code reviews. When everything looks good, you can use these files as input into an automated process (CI/CD) that deploys your infrastructure changes.

So AWS Python CDK builds on AWS CloudFormation and uses it as the engine for provisioning AWS resources. It allows you to compose new abstractions that hide details and simplify common use cases, then it packages the code up as a library in Python. This blog shows you how to get sarted with AWS Python CDK.

Prerequisites

  • Node.js (>= 8.11.x): Why? AWS CDK is developed in TypeScript and transpiled to JavaScript. Bindings for Python make use of the AWS CDK back-end running on Node.js, as does the cdk command-line tool.
  • Your AWS profile and credentials
  • Python >= 3.6

Install AWS Python CDK

1. Check npm version
$ npm -v 6.10.0
2. Install AWS CDK
$ npm install -g aws-cdk
/usr/local/bin/cdk -> /usr/local/lib/node_modules/aws-cdk/bin/cdk > core-js@2.6.9 postinstall /usr/local/lib/node_modules/aws-cdk/node_modules/core-js > node scripts/postinstall || echo "ignore" Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library! The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: > https://opencollective.com/core-js > https://www.patreon.com/zloirock Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -) + aws-cdk@1.3.0
3. Check CDK version
$ cdk --version
1.3.0 (build bba9914)
4. Check Python version
$ python3.7 -V
Python 3.7.1

5. Create a Python virtual env
$ python3.7 -m venv  cdk-venv

$ source cdk-venv/bin/activate

(cdk-venv) $
6. Updating Python Dependencies
(cdk-venv) $ which pip

/Users/txu/code/cdk-venv/bin/pip

(cdk-venv) $ pip install --upgrade aws-cdk.cdk
Collecting aws-cdk.cdk
  Downloading https://files.pythonhosted.org/packages/45/77/07f23d943aece234c91230844595f5495ae6ef5be668b2c9e818ee479ef3/aws_cdk.cdk-0.36.1-py3-none-any.whl
Collecting publication>=0.0.3 (from aws-cdk.cdk)
...
Installing collected packages: publication, typing-extensions, mypy-extensions, attrs, cattrs, six, python-dateutil, jsii, aws-cdk.cdk
Successfully installed attrs-19.1.0 aws-cdk.cdk-0.36.1 cattrs-0.9.0 jsii-0.13.4 mypy-extensions-0.4.1 publication-0.0.3 python-dateutil-2.8.0 six-1.12.0 typing-extensions-3.7.4
7. Create AWS profiles (If you don't have one already)
(cdk-venv) $ aws configure --profile test-dev
AWS Access Key ID [None]: xxxxx
AWS Secret Access Key [None]: xxxx
Default region name [None]: ca-central-1
Default output format [None]: json
8. Check CDK version
(cdk-venv) $ cdk --version
1.3.0 (build bba9914)
At this point, you have successfully installed AWS CDK for Python and connected your Python with AWS CDK.

In our next blog, we will create a example app stack.

Sunday, July 21, 2019

AWS EC2 - Install and Configure virtualenvwrapper with Python3

virtualenvwrapper is a very nice wrapper of Python virtualenv tool. It includes wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies. I really like that how easily it allows you to switch between different projects and virtual environments, saves you a lot of time. This blog shows you how to install and configure it on a AWS EC2 instance, using Python3

Install Python3

$ sudo yum install python3
$ pip3 -V
$ pip3 -V
pip 9.0.3 from /usr/lib/python3.7/site-packages (python 3.7)

Install virtualenvwrapper globally

$ sudo pip3 install virtualenvwrapper
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting virtualenvwrapper
  ...
Installing collected packages: virtualenv, virtualenv-clone, pbr, six, stevedore, virtualenvwrapper
  Running setup.py install for virtualenvwrapper ... done
Successfully installed pbr-5.4.1 six-1.12.0 stevedore-1.30.1 virtualenv-16.6.2 virtualenv-clone-0.5.3 virtualenvwrapper-4.8.4

Create project folder and virtual environment folder

$ ll /home/ec2-user/
total 8
drwxrwxr-x 2 ec2-user ec2-user   6 Jul 21 20:27 code
drwxrwxr-x 2 ec2-user ec2-user   6 Jul 21 20:27 venvs

Locates "virtualenvwrapper.sh"

$ which virtualenvwrapper.sh
/usr/local/bin/virtualenvwrapper.sh

Update bash profile

$ vim ~/.bash_profile
Append the following:
# Configure virtualenvwrapper
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh
export WORKON_HOME="/home/ec2-user/venvs"
export PROJECT_HOME="/home/ec2-user/code"

Resource bash profile

$ source ~/.bash_profile
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/premkproject
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postmkproject
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/initialize
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/premkvirtualenv
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postmkvirtualenv
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/prermvirtualenv
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postrmvirtualenv
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/predeactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postdeactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/preactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/get_env_details

Create a new project

$ mkproject new_proj
Using base prefix '/usr'
  No LICENSE.txt / LICENSE found in source
New python executable in /home/ec2-user/venvs/new_proj/bin/python3
Also creating executable in /home/ec2-user/venvs/new_proj/bin/python
Installing setuptools, pip, wheel...
done.

virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/preactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/postactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/get_env_details
Creating /home/ec2-user/code/new_proj
Setting project for new_proj to /home/ec2-user/code/new_proj
(new_proj) [ec2-user@ip-172-31-36-216 new_proj]$ ls ~/venvs/
new_proj
(new_proj) [ec2-user@ip-172-31-36-216 new_proj]$ ls ~/code/
new_proj

You can see that a "new_proj" folder inside "~/code" and a new virtual environment "new_proj" inside "~/venvs" are created automatically and the virtual environment of "new_proj" automatically activated. Let's create a second project and show how easily we can switch between projects and their virtual environments.
$ mkproject new_proj2
...
Creating /home/ec2-user/code/new_proj2
Setting project for new_proj2 to /home/ec2-user/code/new_proj2

Switch to "new_proj" and its virtual env:

(new_proj2) [ec2-user@hostname new_proj2]$ workon new_proj
(new_proj) [ec2-user@hostname new_proj]$
You can see how easily you are able to switch between projects without deactivate, reactivate, switch folder, ..etc.!

You can list all the exisitng virtual environments:

# List environments
$ workon
new_proj2
new_proj

More information, checkout https://virtualenvwrapper.readthedocs.io/en/latest/

Sunday, May 05, 2019

Datadog- Artifactory JMX Monitoring

Datadog JMX integration collects metrics from applications that expose JMX metrics. A lightweight Java plugin named "JMXFetch" is called by the Datadog Agent to connect to the MBean server and to collect metrics, it can also send service checks that report on the status of your monitored instances.

This plugin sends metrics to the Datadog Agent using DogStatsD server running within the Agent. In the blog we will show you how to collect and send metrics from Artifactory application.

Prerequisites:

  1. You have a running Artifactory application with JMX enabled (https://jfrog.com/knowledge-base/how-do-i-use-jmx-remote-monitoring/)
  2. You have a running Datadog agent which has access to your Artifactory JMX port

Configuration:

You need to enable JMX monitoring in Datadog by creating a file "/etc/datadog-agent/conf.d/jmx.d/conf.yaml" with the following configuration:

init_config:
  new_gc_metrics: true

instances:
  - host: localhost # If DD agent is running on the same host
    port: 9010
    user: <USER_NAME>
    password: <PASSWORD>
    name: artifactory-app
    
    tags:
      env: pord
    
    conf:
      - include:
        domain: org.jfrog.artifactory
        bean:
          - org.jfrog.artifactory:instance=Artifactory, type=Storage,prop=Binary Storage
          - org.jfrog.artifactory:instance=Artifactory, type=Repositories,prop=test-repo
        attribute:
          # Ginary storage size
          - Size
          # Repo
          - ArtifactsTotalSize
          - ArtifactsCount

      - include:
        domain: Catalina
        bean:
          - Catalina:type=ThreadPool,name="http-nio-8081"
        attribute:
          - maxThreads
          - currentThreadCount
          - currentThreadBusy
   

This following graph shows you a visual mapping between Mbeans and the actual Datadog configuration.


You can also use the new way of MBean configuration, as described following (https://docs.datadoghq.com/integrations/java/):

  conf:
    - include:
        domain: org.apache.cassandra.db
        attribute:
          - BloomFilterDiskSpaceUsed
          - BloomFilterFalsePositives
          - BloomFilterFalseRatio
          - Capacity
          - CompressionRatio
          - CompletedTasks
          - ExceptionCount
          - Hits
          - RecentHitRate
   

But I couldn't get the new confiuration to work as the old one yet, the new one looks much easier to config, but it didn't work for me, for some reason it is not sending metrics to Datadog platform.

You could use "datadog-agent jmx list collected" command to check what JMX metrics that you are collecting.

Note1:

To run more than one JMX check, create configuration files with the format jmx_<INDEX>.d/conf.yaml (e.g. jmx_1.d/conf.yaml, jmx_2.d/conf.yaml, etc). Each folder should be stored in the conf.d directory. Include the is_jmx option set to true in those configuration files.

The 350 metric limit

Due to the nature of these integrations, it is possible to submit an extremely high number of metrics directly to Datadog. Many customers agree that some of these metrics are not needed. Therefore, Datadog has set the limit at 350 metrics.

Sunday, April 28, 2019

Logstash - AWS S3 Bucket As Data Input

You can use the "S3 input plugin" to stream events from files from a AWS S3 bucket. Each line from each file generates an event. Files ending in ".gz" are handled as gzip'ed files. Glacier files will be skipped.

Logstash version: 6.7.1

Here is a basic configuration for streaming data:

input {
  s3 {
    "access_key_id" => "1234"
    "secret_access_key" => "secret"
    "bucket" => "logstash-test-aws-s3-bucket"
    "additional_settings" => {
      "force_path_style" => true
      "follow_redirects" => false
    }
    "region" => "us-east-1"
    "prefix" => "logstash-"
    "type" => "s3"
  }
}

output {
  elasticsearch {
    cacert => "/path/to/cert"
    hosts => "https://elasticsearch1.com:9243"
    index => "test-index-%{+YYYY.MM}"
    user => "logstash"
    password => "logstash"
  }
}


Files in this "logstash-test-aws-s3-bucket" AWS S3 bucket start with "logstash-" will match (including folders).

Start your logstash with "logstash -f confg/s3-input-logstash.conf", you should start seeing data coming into your Elasticsearch cluster.

Logstash - Send Data To Multiple Elastic Clusters

It is possible to send same data from one Logstash server to multiple Elasticsearch clusters. It is easy to configure, you just need to have multiple outputs definitions for the same events. However, it is not easy to find a clear example from the internet, so I've decided to make this blog.

My example Logstash configuration file:

input {
  file {
    path => ["/path/to/json/file"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    exclude => "*.gz"
  }
}

filter {
  mutate {
    replace => [ "message", "%{message}" ]
    gsub => [ 'message','\n','']
  }
  if [message] =~ /^{.*}$/ {
    json { source => message }
  }
}

output {
  elasticsearch {
    cacert => "/path/to/cert"
    hosts => "https://elasticsearch1.com:9243"
    index => "test-index-%{+YYYY.MM}"
    user => "logstash"
    password => "logstash"
  }

  elasticsearch {
    cacert => "/path/to/cert"
    hosts => "https://elasticsearch2.com:9243"
    index => "test-index-%{+YYYY.MM}"
    user => "logstash"
    password => "logstash"
  }
}

Example of JSON file:

{"foo":"bar", "bar": "foo"}
{"hello":"world", "goodnight": "moon"}

Note the JSON file content need to be in one line.

This setup is an all or nothing, if one of the output is down, the second will not work. You should think do you really need this setup at the first place, now you have two copy of data that you need to keep in sync, maybe output to one place, and have the role based control is a better option.

Sunday, March 03, 2019

Octopus Deploy - How To Install Octopus Deploy In AWS EC2


"Octopu Deploy" is an industry leading automated deployment and release management tool used by leading continuous delivery teams worldwide. It is designed to simplify deployment of .NET applications, Windows Services and databases.

This blog shows you how to install the lastest LTS version of "Octopus Deploy" in AWS EC2.

Environment:


  • Octopus Deploy: 2018.10.5 LTS
  • EC2: t2.large
  • OS: "Windows Server 2019 Datacenter"

Installation Steps:


1. Launch a Windows EC2 instance

2. Select "General purpose", "t2.large"

3. Configure Storage

4. Configure tags


5. Configure security group

6. Once the instance is up and running, RDP into the server, you can get the connection details from "Connect" button.

7. Open up IE in the remote server, download "Octopus Deploy 2018.10.5 LTS"
Note: by default, the stupid "Internet Explorer Enhanced Security Configuration" is on, you need to turn it off so you can download things.



8. Double click to install "Octopus.2018.10.5"






9. Complete the Octopus Deploy Server Setup Wizard






Note: You will need SQL Database for Octopus Deploy, if you don't have one, you can download the "SQL Server Express (free)" from Octopus "Database" installation window.

10. Install SQL server express basic





11. Connect to local "SQLEXPRESS" service, and create a "Octopus" database




12. Finally, install.


13. Congratulations, now you have successfully install "Octopu Deploy" in AWS Ec2 instance!

Now login into your Octopus Deploy and start play around with it!




Sunday, February 24, 2019

ElasticSearch - Part 1 - How To Deploy Single Node Cluster On AWS EC2

This blog shows you how to deploy and manage your own ElasticSeacrh cluster on AWS EC2.

Environment:

  • Instance: t2.2xlarge (32GB Mem, 8vCPUs)
  • ElasticSearch version: 6.6.1
Assuming you have all your AWS environment setup, you are ready to launch a EC2 instance and have ssh access to it.

Provision AWS EC2 instance:

Elasticsearch runs on various operating systems such as CentOS, Redhat, Ubuntu, and Amazon Linux. We suggest using the latest Amazon Linux AMI.

Choose "t2.2xlarge" instance type, which provides 8vCPUs, 32 GB of memory and EBS volume for data, which is a reasonable starting point. Go ahead and start up the instance. Two thing to notice:
  • 1. The security group open ports:
    • - port 22: SSH
    • - port 9200: ElasticSearch requests
  • 2. The storage:
    • - 30GB; You can expand later


Install ElasticSearch

Once the EC2 instance is up and running, we can start ElasticSearch installation.

1. Log into EC2 instance:
$ ssh -i ~/.ssh/tony-aws.pem ec2-user@54.1xx.1xx.xxx
$ sudo su -
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
tmpfs            16G  472K   16G   1% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/xvda1       30G  1.2G   29G   4% /
tmpfs           3.2G     0  3.2G   0% /run/user/0
tmpfs           3.2G     0  3.2G   0% /run/user/1000


2. Install Javav1.8.0
$ yum install java-1.8.0-openjdk
Installed:
  java-1.8.0-openjdk.x86_64 1:1.8.0.191.b12-0.amzn2


3. Download ElasticSearch 6.6.1
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.1.rpm
--2019-02-25 01:02:20--  https://artifacts.elastic.co/downloads/
elasticsearch/elasticsearch-6.6.1.rpm
Resolving artifacts.elastic.co (artifacts.elastic.co)... 151.101.250.222, 
2a04:4e42:3b::734
Connecting to artifacts.elastic.co (artifacts.elastic.co)|151.101.250.222
|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 114067654 (109M) [application/octet-stream]
Saving to: ‘elasticsearch-6.6.1.rpm’

2019-02-25 01:02:22 (115 MB/s) - ‘elasticsearch-6.6.1.rpm’ 
saved [114067654/114067654]

4. Install the Elasticsearch RPM package on each EC2 instance as instructed below.
$ rpm -iv elasticsearch-6.6.1.rpm
warning: elasticsearch-6.6.1.rpm: Header V4 RSA/SHA512 Signature, 
key ID d88e42b4: NOKEY
Preparing packages...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
elasticsearch-0:6.6.1-1.noarch

### NOT starting on installation, please execute the following 
statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service

### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch

5. By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the "--quiet" option must be removed from the ExecStart command line in the elasticsearch.service file.
$ vim /usr/lib/systemd/system/elasticsearch.service
# Remove --quiet by Tony
#ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet
ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet

6. Configure Elasticsearch
Elasticsearch defaults to using /etc/elasticsearch for runtime configuration. Elasticsearch loads its configuration from the /etc/elasticsearch/elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch (https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html).

Update the bind host:
$ vim /etc/elasticsearch/elasticsearch.yml
locate "network.host: 192.168.0.1", and update it to
network.host: 0.0.0.0

This opens up ElasticSearch to listen on traffic from all hosts.

Update the cluster name:
$ vim /etc/elasticsearch/elasticsearch.yml
locate "cluster.name: my-application", and update it to
cluster.name: tony-es-cluster

Setting the heap size:
By default, Elasticsearch tells the JVM to use a heap with a minimum and maximum size of 1 GB. However, it is important to configure heap size to ensure that Elasticsearch has enough heap available. Elasticsearch will assign the entire heap specified in jvm.options via the Xms (minimum heap size) and Xmx (maximum heap size) settings.
The value for these setting depends on the amount of RAM available on the instance, a rule of thumb is "Set Xmx to no more than 50% of your physical RAM, to ensure that there is enough physical RAM left for kernel file system caches." In our case, the value is 16GB.
$ vim /etc/elasticsearch/jvm.options
Locate "-Xms1g -Xmx1g", and update it to
-Xms16g
-Xmx16g


6. Start ElasticSearch
$ systemctl start elasticsearch.service
$ systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; 
disabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-02-25 01:32:29 UTC; 2s ago
     Docs: http://www.elastic.co
 Main PID: 13803 (java)
   CGroup: /system.slice/elasticsearch.service
           └─13803 /bin/java -Xms16g -Xmx16g -XX:+UseConcMarkSweepGC 
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
-Des.networkaddress.cache.ttl=60 -Des.networkaddr.


7. Verify from API request
$ curl -X GET http://54.1xx.1xx.1xx:9200/
{
  "name" : "ZAvN4SU",
  "cluster_name" : "tony-es-cluster",
  "cluster_uuid" : "bYSZ8nkqS-mnI8x2F3eHhQ",
  "version" : {
    "number" : "6.6.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "1fd8f69",
    "build_date" : "2019-02-13T17:10:04.160291Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}


Since we enabled the journal entries for ElasticSeach, you can list the entries now:
$ journalctl --unit elasticsearch
-- Logs begin at Mon 2019-02-25 00:58:47 UTC, end at Mon 2019-02-25 01:32:29 UTC. --

Feb 25 01:26:55 ip-172-31-88-104.ec2.internal systemd[1]: Started Elasticsearch.
Feb 25 01:26:55 ip-172-31-88-104.ec2.internal systemd[1]: Starting Elasticsearch...
Feb 25 01:28:13 ip-172-31-88-104.ec2.internal systemd[1]: Stopping Elasticsearch...
Feb 25 01:28:14 ip-172-31-88-104.ec2.internal systemd[1]: Stopped Elasticsearch.
Feb 25 01:29:38 ip-172-31-88-104.ec2.internal systemd[1]: Started Elasticsearch.
Feb 25 01:29:38 ip-172-31-88-104.ec2.internal systemd[1]: Starting Elasticsearch...
Feb 25 01:31:01 ip-172-31-88-104.ec2.internal systemd[1]: Stopping Elasticsearch...
Feb 25 01:31:01 ip-172-31-88-104.ec2.internal systemd[1]: Stopped Elasticsearch.
Feb 25 01:32:29 ip-172-31-88-104.ec2.internal systemd[1]: Started Elasticsearch.
Feb 25 01:32:29 ip-172-31-88-104.ec2.internal systemd[1]: Starting Elasticsearch...

At this point, you have a running ElasticSearch 6.6.1 single node cluster.

ECE - Small Trick: Query Zookeeper Statistics

ECE stands for "Elastic Cloud Enterprise". It shares most of its codebase with Elastic Cloud. The key tenets of the architecture are:
  • Service-oriented architecture
  • Containerization using Docker
  • Deployment state coordination using ZooKeeper
  • Easy access through the Cloud UI

Since ECE is a service-oriented architecture, it makes scale the platform very easy. Different services can have different reliability and performance requirements, as each service can be scaled separately. This also hides some of technical details from you, especially administrators that curious how it works inside. Sometimes it could be really hard to get information out of a vendor container.

In our case, we have ECE 1.x and 2.x both installed in our environment, one of challenges we had was how to monitor the Zookeeper status. The Zookeeper status from the admin console doesn't count :). Fortunately, ECE Zookeeper container exposes its port to "0.0.0.0" (i.e, 0.0.0.0:2192->2192/tcp), this means you can can query some of whitelisted information through the host IP and exposed port.

For example, say one of your Zookeeper role exposes port "2192". If you would like to output a list of variables that could be used for monitoring the health of the cluster. You could query the underline host that runs Zookeeper role like following:
$ echo mntr | nc ecedc1h1.lixu.ca 2192

zk_version  3.4.0
zk_avg_latency  0
zk_max_latency  0
zk_min_latency  0
zk_packets_received 70
zk_packets_sent 69
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count   4
zk_watch_count  0
zk_ephemerals_count 0
zk_approximate_data_size    27
zk_followers    4                   - only exposed by the Leader
zk_synced_followers 4               - only exposed by the Leader
zk_pending_syncs    0               - only exposed by the Leader
zk_open_file_descriptor_count 23    - only available on Unix platforms
zk_max_file_descriptor_count 1024   - only available on Unix platforms

Then, if you really want to go fancy, you could send the output to a monitoring and alerting system like "Datadog" peoridically (I might make another post for this). But for information query, this is a nice and easy way.

Our Zookeeper version is v3.5.3, and a few useful commands:


  • conf: rint details about serving configuration (not in the whitelist).
  • cons: List full connection/session details for all clients connected to this server. Includes information on numbers of packets received/sent, session id, operation latencies, last operation performed, etc... (not in the whitelist).
  • dump: Lists the outstanding sessions and ephemeral nodes. This only works on the leader (not in the whitelist).
  • envi: Print details about serving environment (not in the whitelist).
  • ruok: Tests if server is running in a non-error state. The server will respond with imok if it is running. Otherwise it will not respond at all.
  • A response of "imok" does not necessarily indicate that the server has joined the quorum, just that the server process is active and bound to the specified client port. Use "stat" for details on state wrt quorum and client connection information.
  • srvr: Lists full details for the server.
  • stat: Lists brief details for the server and connected clients.
  • wchs: Lists brief information on watches for the server (not in the whitelist).
  • wchc: Lists detailed information on watches for the server, by session. This outputs a list of sessions(connections) with associated watches (paths). Note, depending on the number of watches this operation may be expensive (ie impact server performance), use it carefully (not in the whitelist).
  • wchp: Lists detailed information on watches for the server, by path. This outputs a list of paths (znodes) with associated sessions. Note, depending on the number of watches this operation may be expensive (ie impact server performance), use it carefully.
  • mntr: Outputs a list of variables that could be used for monitoring the health of the cluster.

Sunday, February 10, 2019

Jenkins - How To Configure Behind Nginx HTTP/HTTPS Proxy

This blog shows you how to configure Jenkins behind a Nginx proxy server. One typical use case is that you want to add ssl certificate to secure your Jenkins traffic.

Environment:

Jenkins: v2.155
Nginx: v1.14.1

Assuming you already have Nginx and Jenkins installed and running. If you don't, I will put up some other posts for how to install Nginx and Jenkins.

By default, you Jenkins process is running on port 8080, you probably access it via a DNS name, such as:
http://jenkins.lixu.ca:8080
or
https://jenkins.lixu.ca:8080

This blog will help you configure Jenkins behind both http and https protocols. At the end, you should be able to access your Jenkins server via:
http://jenkins.lixu.ca
and
https://jenkins.lixu.ca

Prepare SSL Certificates for HTTPS

If you doing HTTPS as well, assume you already have the certificate (could be a self-signed cert). If you don't have it yet, you can check out my other blog Creating and Signing Your Certs.

Define upstream for both HTTP and HTTPS proxy_pass
Define a "upstream". An "upstream" could be one or more servers that can be referenced by "proxy_pass" later, this could save you define each servers later for HTTP and HTTPS separately.

upstream jenkins {
  server 127.0.0.1:8080 fail_timeout=0;
}

Note: You need to put define this upstream in the "http{}" block in your "nginx.conf".

Update Your Nginx HTTP Config:
You need to update the "server_name" and "location" definitions. For example:
server {
    upstream jenkins {
        server 127.0.0.1:8080 fail_timeout=0;
    }
 
    listen 80;
    server_name jenkins.lixu.ca;

    location / {
      proxy_set_header        Host $host:$server_port;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;
      # Fix the "It appears that your reverse proxy set up is broken" error.
      proxy_pass          http://jenkins;
      proxy_read_timeout  90;

      # Required for new HTTP-based CLI
      proxy_http_version 1.1;
      proxy_request_buffering off;
      # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
      add_header 'X-SSH-Endpoint' 'jenkins.lixu.ca:50022' always;
    }
}

Restart you Nginx process, once it comes back up, you should be able to access your Jenkins UI through: http://jenkins.lixu.ca

Update Your Nginx HTTPs
Config Similar to HTTP, update "server_name" and "location" definition:
server {
  listen 443 ssl;
  server_name jenkins.lixu.ca;

  ssl_certificate /etc/nginx/ssl/server.crt;
  ssl_certificate_key /etc/nginx/ssl/server.key;

  location / {
    proxy_set_header        Host $host:$server_port;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
    proxy_redirect http:// https://;
    proxy_pass              http://jenkins;

    # Required for new HTTP-based CLI
    proxy_http_version 1.1;
    proxy_request_buffering off;
    proxy_buffering off; # Required for HTTP-based CLI to work over SSL

    # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
    add_header 'X-SSH-Endpoint' 'jenkins.lixu.ca:50022' always;
  }
}

Restart you Nginx process, once it comes back up, you should be able to access your Jenkins UI through: https://jenkins.lixu.ca