Friday, January 24, 2020

Ansible Template Tip: Split a String

Ansible is the simplest way to automate apps and IT infrastructure. Application Deployment + Configuration Management + Continuous Delivery. We use Ansible heavily for our AWS deployment stack.

When working with Ansible template, sometimes you will need to split a string and use part of its value. For example, you have a application template which setup the application base url, like follow:

<?xml version='1.1' encoding='UTF-8'?>
    <appUrl>https://{{ app.split("-")[0] | lower }}-{{ env | lower }}</appUrl>

We have a variable "app" containing the string "" and want to extract the geolocation part of this string, we could just use "{{ app.split("-")[0] | lower }}"

The constructed url will be "".

Ansible use Jinja2 for formatting, so you can apply the split filter (

Sunday, August 04, 2019

AWS CDK Python - Sample Project

This blog shows you how to create a sample project using AWS CDK Python. If you don't have the environment setup yet, check out my previous blog:

Create the Python Application

1. Fire up you Python3 virtual environment, create a project directory contains your Python project source code
$ source cdk-env/bin/activate
(cdk-venv) $ mkdir sample-dev
(cdk-venv) $ cd sample-dev
(cdk-venv) $ cdk init
Available templates:
* app: Template for a CDK Application
   └─ cdk init app --language=[csharp|fsharp|java|javascript|python|typescript]
* lib: Template for a CDK Construct Library
   └─ cdk init lib --language=typescript
* sample-app: Example CDK Application with some constructs
   └─ cdk init sample-app --language=[javascript|python|typescript]
2. The return information from cdk is very self-explanatary, we can choose "sample-app" so by the creation of this app, we have some exisitng constructs
$ cdk init --language python sample-app
Applying project template sample-app for python
Initializing a new git repository...
Executing Creating virtualenv...

3. The entire sample project looks like this:
├── - Readme file
├──    - Defines app stacks, also the "main" file
├── cdk.json  - Configuration file for CDK that defines what executable CDK should run to generate CDK construct tree
├── hello
│   ├──         - To make a Python module
│   ├── hello.egg-info
│   │   ├── PKG-INFO
│   │   ├── SOURCES.txt
│   │   ├── dependency_links.txt
│   │   ├── requires.txt
│   │   └── top_level.txt
│   ├──  - A custom CDK construct defined for use in your CDK application.
│   └──      - A custom CDK stack construct for use in your CDK application
├── requirements.txt     - Required Python modules
├──             - Defines how this Python package would be constructed and what the dependencies are
└── tests                - Test folder
    └── unit             - ALl the unit tests

You can take time to read and understand the source code, but esentially what this code does is to create two CloudFormation templates: "hello-cdk-1" and "hello-cdk-2", and deploy them to "us-east-1" and "us-west-1" regions. Each template includes:

  • IAM user with predefined policies
  • AWS sqs queue
  • AWS sns topic and subscribe to the sqs queue
  • Few AWS S3 buckets

4. Give a read of the file, you can ignore the Python virtual env part, as we are already using a virtual env. As the suggest, we going to install all required Python packages and run some unit tests.
(cdk-venv) $ pip install -r requirements.txt
Collecting pytest (from -r requirements.txt (line 2))
  Downloading (221kB)
    100% |████████████████████████████████| 225kB 4.5MB/s 

(cdk-venv) $ pytest
================================================================================================================= test session starts
platform darwin -- Python 3.7.1, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /Users/txu/code/flashhop-dev
collected 1 item                                                                                                                                                                                                                                      

tests/unit/ . 

5. Generate AWS CloudFormation template
(cdk-venv) $ cdk synth
Successfully synthesized to /Users/txu/code/flashhop-dev/cdk.out
Supply a stack name (hello-cdk-1, hello-cdk-2) to display its template.
All the output files will be located in "cdk.out"

6. Deploy the generated templates to AWS account and validate the right resouces got created.
(cdk-venv) $ cdk deploy --profile YOU_AWS_PROFILE hello-cdk-1

7. To destroy the stack
(cdk-venv) $ cdk --profile flashhop-dev destroy hello-cdk-1
Are you sure you want to delete: hello-cdk-1 (y/n)? y
hello-cdk-1: destroying...

At this point, you learned how to use AWS CDK CLI to initialize a new Python project/app, how to synthesize hte Python code into AWS Cloudformation templates, deploy them, provision your infrastructure and destroy your infrastructure. You can learn more from AWS CDK developer documentations (

AWS CDK Python - How To Get Started

Assuming you are already familiar with the concept of "IaC" (Infrastructure As Code), well in AWS context, "IaC" means CloudFormation.

CloudFormation allows you to define your AWS infrastructure in JSON or YAML files that can be managed witin your source code repository (Git for example). You can do pull requests and code reviews. When everything looks good, you can use these files as input into an automated process (CI/CD) that deploys your infrastructure changes.

So AWS Python CDK builds on AWS CloudFormation and uses it as the engine for provisioning AWS resources. It allows you to compose new abstractions that hide details and simplify common use cases, then it packages the code up as a library in Python. This blog shows you how to get sarted with AWS Python CDK.


  • Node.js (>= 8.11.x): Why? AWS CDK is developed in TypeScript and transpiled to JavaScript. Bindings for Python make use of the AWS CDK back-end running on Node.js, as does the cdk command-line tool.
  • Your AWS profile and credentials
  • Python >= 3.6

Install AWS Python CDK

1. Check npm version
$ npm -v 6.10.0
2. Install AWS CDK
$ npm install -g aws-cdk
/usr/local/bin/cdk -> /usr/local/lib/node_modules/aws-cdk/bin/cdk > core-js@2.6.9 postinstall /usr/local/lib/node_modules/aws-cdk/node_modules/core-js > node scripts/postinstall || echo "ignore" Thank you for using core-js ( ) for polyfilling JavaScript standard library! The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: > > Also, the author of core-js ( ) is looking for a good job -) + aws-cdk@1.3.0
3. Check CDK version
$ cdk --version
1.3.0 (build bba9914)
4. Check Python version
$ python3.7 -V
Python 3.7.1

5. Create a Python virtual env
$ python3.7 -m venv  cdk-venv

$ source cdk-venv/bin/activate

(cdk-venv) $
6. Updating Python Dependencies
(cdk-venv) $ which pip


(cdk-venv) $ pip install --upgrade aws-cdk.cdk
Collecting aws-cdk.cdk
Collecting publication>=0.0.3 (from aws-cdk.cdk)
Installing collected packages: publication, typing-extensions, mypy-extensions, attrs, cattrs, six, python-dateutil, jsii, aws-cdk.cdk
Successfully installed attrs-19.1.0 aws-cdk.cdk-0.36.1 cattrs-0.9.0 jsii-0.13.4 mypy-extensions-0.4.1 publication-0.0.3 python-dateutil-2.8.0 six-1.12.0 typing-extensions-3.7.4
7. Create AWS profiles (If you don't have one already)
(cdk-venv) $ aws configure --profile test-dev
AWS Access Key ID [None]: xxxxx
AWS Secret Access Key [None]: xxxx
Default region name [None]: ca-central-1
Default output format [None]: json
8. Check CDK version
(cdk-venv) $ cdk --version
1.3.0 (build bba9914)
At this point, you have successfully installed AWS CDK for Python and connected your Python with AWS CDK.

In our next blog, we will create a example app stack.

Sunday, July 21, 2019

AWS EC2 - Install and Configure virtualenvwrapper with Python3

virtualenvwrapper is a very nice wrapper of Python virtualenv tool. It includes wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies. I really like that how easily it allows you to switch between different projects and virtual environments, saves you a lot of time. This blog shows you how to install and configure it on a AWS EC2 instance, using Python3

Install Python3

$ sudo yum install python3
$ pip3 -V
$ pip3 -V
pip 9.0.3 from /usr/lib/python3.7/site-packages (python 3.7)

Install virtualenvwrapper globally

$ sudo pip3 install virtualenvwrapper
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting virtualenvwrapper
Installing collected packages: virtualenv, virtualenv-clone, pbr, six, stevedore, virtualenvwrapper
  Running install for virtualenvwrapper ... done
Successfully installed pbr-5.4.1 six-1.12.0 stevedore-1.30.1 virtualenv-16.6.2 virtualenv-clone-0.5.3 virtualenvwrapper-4.8.4

Create project folder and virtual environment folder

$ ll /home/ec2-user/
total 8
drwxrwxr-x 2 ec2-user ec2-user   6 Jul 21 20:27 code
drwxrwxr-x 2 ec2-user ec2-user   6 Jul 21 20:27 venvs

Locates ""

$ which

Update bash profile

$ vim ~/.bash_profile
Append the following:
# Configure virtualenvwrapper
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/
export WORKON_HOME="/home/ec2-user/venvs"
export PROJECT_HOME="/home/ec2-user/code"

Resource bash profile

$ source ~/.bash_profile
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/premkproject
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postmkproject
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/initialize
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/premkvirtualenv
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postmkvirtualenv
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/prermvirtualenv
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postrmvirtualenv
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/predeactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postdeactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/preactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/get_env_details

Create a new project

$ mkproject new_proj
Using base prefix '/usr'
  No LICENSE.txt / LICENSE found in source
New python executable in /home/ec2-user/venvs/new_proj/bin/python3
Also creating executable in /home/ec2-user/venvs/new_proj/bin/python
Installing setuptools, pip, wheel...

virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/preactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/postactivate
virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/get_env_details
Creating /home/ec2-user/code/new_proj
Setting project for new_proj to /home/ec2-user/code/new_proj
(new_proj) [ec2-user@ip-172-31-36-216 new_proj]$ ls ~/venvs/
(new_proj) [ec2-user@ip-172-31-36-216 new_proj]$ ls ~/code/

You can see that a "new_proj" folder inside "~/code" and a new virtual environment "new_proj" inside "~/venvs" are created automatically and the virtual environment of "new_proj" automatically activated. Let's create a second project and show how easily we can switch between projects and their virtual environments.
$ mkproject new_proj2
Creating /home/ec2-user/code/new_proj2
Setting project for new_proj2 to /home/ec2-user/code/new_proj2

Switch to "new_proj" and its virtual env:

(new_proj2) [ec2-user@hostname new_proj2]$ workon new_proj
(new_proj) [ec2-user@hostname new_proj]$
You can see how easily you are able to switch between projects without deactivate, reactivate, switch folder, ..etc.!

You can list all the exisitng virtual environments:

# List environments
$ workon

More information, checkout

Sunday, May 05, 2019

Datadog- Artifactory JMX Monitoring

Datadog JMX integration collects metrics from applications that expose JMX metrics. A lightweight Java plugin named "JMXFetch" is called by the Datadog Agent to connect to the MBean server and to collect metrics, it can also send service checks that report on the status of your monitored instances.

This plugin sends metrics to the Datadog Agent using DogStatsD server running within the Agent. In the blog we will show you how to collect and send metrics from Artifactory application.


  1. You have a running Artifactory application with JMX enabled (
  2. You have a running Datadog agent which has access to your Artifactory JMX port


You need to enable JMX monitoring in Datadog by creating a file "/etc/datadog-agent/conf.d/jmx.d/conf.yaml" with the following configuration:

  new_gc_metrics: true

  - host: localhost # If DD agent is running on the same host
    port: 9010
    user: <USER_NAME>
    password: <PASSWORD>
    name: artifactory-app
      env: pord
      - include:
        domain: org.jfrog.artifactory
          - org.jfrog.artifactory:instance=Artifactory, type=Storage,prop=Binary Storage
          - org.jfrog.artifactory:instance=Artifactory, type=Repositories,prop=test-repo
          # Ginary storage size
          - Size
          # Repo
          - ArtifactsTotalSize
          - ArtifactsCount

      - include:
        domain: Catalina
          - Catalina:type=ThreadPool,name="http-nio-8081"
          - maxThreads
          - currentThreadCount
          - currentThreadBusy

This following graph shows you a visual mapping between Mbeans and the actual Datadog configuration.

You can also use the new way of MBean configuration, as described following (

    - include:
        domain: org.apache.cassandra.db
          - BloomFilterDiskSpaceUsed
          - BloomFilterFalsePositives
          - BloomFilterFalseRatio
          - Capacity
          - CompressionRatio
          - CompletedTasks
          - ExceptionCount
          - Hits
          - RecentHitRate

But I couldn't get the new confiuration to work as the old one yet, the new one looks much easier to config, but it didn't work for me, for some reason it is not sending metrics to Datadog platform.

You could use "datadog-agent jmx list collected" command to check what JMX metrics that you are collecting.


To run more than one JMX check, create configuration files with the format jmx_<INDEX>.d/conf.yaml (e.g. jmx_1.d/conf.yaml, jmx_2.d/conf.yaml, etc). Each folder should be stored in the conf.d directory. Include the is_jmx option set to true in those configuration files.

The 350 metric limit

Due to the nature of these integrations, it is possible to submit an extremely high number of metrics directly to Datadog. Many customers agree that some of these metrics are not needed. Therefore, Datadog has set the limit at 350 metrics.

Sunday, April 28, 2019

Logstash - AWS S3 Bucket As Data Input

You can use the "S3 input plugin" to stream events from files from a AWS S3 bucket. Each line from each file generates an event. Files ending in ".gz" are handled as gzip'ed files. Glacier files will be skipped.

Logstash version: 6.7.1

Here is a basic configuration for streaming data:

input {
  s3 {
    "access_key_id" => "1234"
    "secret_access_key" => "secret"
    "bucket" => "logstash-test-aws-s3-bucket"
    "additional_settings" => {
      "force_path_style" => true
      "follow_redirects" => false
    "region" => "us-east-1"
    "prefix" => "logstash-"
    "type" => "s3"

output {
  elasticsearch {
    cacert => "/path/to/cert"
    hosts => ""
    index => "test-index-%{+YYYY.MM}"
    user => "logstash"
    password => "logstash"

Files in this "logstash-test-aws-s3-bucket" AWS S3 bucket start with "logstash-" will match (including folders).

Start your logstash with "logstash -f confg/s3-input-logstash.conf", you should start seeing data coming into your Elasticsearch cluster.

Logstash - Send Data To Multiple Elastic Clusters

It is possible to send same data from one Logstash server to multiple Elasticsearch clusters. It is easy to configure, you just need to have multiple outputs definitions for the same events. However, it is not easy to find a clear example from the internet, so I've decided to make this blog.

My example Logstash configuration file:

input {
  file {
    path => ["/path/to/json/file"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    exclude => "*.gz"

filter {
  mutate {
    replace => [ "message", "%{message}" ]
    gsub => [ 'message','\n','']
  if [message] =~ /^{.*}$/ {
    json { source => message }

output {
  elasticsearch {
    cacert => "/path/to/cert"
    hosts => ""
    index => "test-index-%{+YYYY.MM}"
    user => "logstash"
    password => "logstash"

  elasticsearch {
    cacert => "/path/to/cert"
    hosts => ""
    index => "test-index-%{+YYYY.MM}"
    user => "logstash"
    password => "logstash"

Example of JSON file:

{"foo":"bar", "bar": "foo"}
{"hello":"world", "goodnight": "moon"}

Note the JSON file content need to be in one line.

This setup is an all or nothing, if one of the output is down, the second will not work. You should think do you really need this setup at the first place, now you have two copy of data that you need to keep in sync, maybe output to one place, and have the role based control is a better option.

Sunday, March 03, 2019

Octopus Deploy - How To Install Octopus Deploy In AWS EC2

"Octopu Deploy" is an industry leading automated deployment and release management tool used by leading continuous delivery teams worldwide. It is designed to simplify deployment of .NET applications, Windows Services and databases.

This blog shows you how to install the lastest LTS version of "Octopus Deploy" in AWS EC2.


  • Octopus Deploy: 2018.10.5 LTS
  • EC2: t2.large
  • OS: "Windows Server 2019 Datacenter"

Installation Steps:

1. Launch a Windows EC2 instance

2. Select "General purpose", "t2.large"

3. Configure Storage

4. Configure tags

5. Configure security group

6. Once the instance is up and running, RDP into the server, you can get the connection details from "Connect" button.

7. Open up IE in the remote server, download "Octopus Deploy 2018.10.5 LTS"
Note: by default, the stupid "Internet Explorer Enhanced Security Configuration" is on, you need to turn it off so you can download things.

8. Double click to install "Octopus.2018.10.5"

9. Complete the Octopus Deploy Server Setup Wizard

Note: You will need SQL Database for Octopus Deploy, if you don't have one, you can download the "SQL Server Express (free)" from Octopus "Database" installation window.

10. Install SQL server express basic

11. Connect to local "SQLEXPRESS" service, and create a "Octopus" database

12. Finally, install.

13. Congratulations, now you have successfully install "Octopu Deploy" in AWS Ec2 instance!

Now login into your Octopus Deploy and start play around with it!

Sunday, February 24, 2019

ElasticSearch - Part 1 - How To Deploy Single Node Cluster On AWS EC2

This blog shows you how to deploy and manage your own ElasticSeacrh cluster on AWS EC2.


  • Instance: t2.2xlarge (32GB Mem, 8vCPUs)
  • ElasticSearch version: 6.6.1
Assuming you have all your AWS environment setup, you are ready to launch a EC2 instance and have ssh access to it.

Provision AWS EC2 instance:

Elasticsearch runs on various operating systems such as CentOS, Redhat, Ubuntu, and Amazon Linux. We suggest using the latest Amazon Linux AMI.

Choose "t2.2xlarge" instance type, which provides 8vCPUs, 32 GB of memory and EBS volume for data, which is a reasonable starting point. Go ahead and start up the instance. Two thing to notice:
  • 1. The security group open ports:
    • - port 22: SSH
    • - port 9200: ElasticSearch requests
  • 2. The storage:
    • - 30GB; You can expand later

Install ElasticSearch

Once the EC2 instance is up and running, we can start ElasticSearch installation.

1. Log into EC2 instance:
$ ssh -i ~/.ssh/tony-aws.pem
$ sudo su -
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
tmpfs            16G  472K   16G   1% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/xvda1       30G  1.2G   29G   4% /
tmpfs           3.2G     0  3.2G   0% /run/user/0
tmpfs           3.2G     0  3.2G   0% /run/user/1000

2. Install Javav1.8.0
$ yum install java-1.8.0-openjdk
  java-1.8.0-openjdk.x86_64 1:

3. Download ElasticSearch 6.6.1
$ wget
--2019-02-25 01:02:20--
Resolving (, 
Connecting to (|
|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 114067654 (109M) [application/octet-stream]
Saving to: ‘elasticsearch-6.6.1.rpm’

2019-02-25 01:02:22 (115 MB/s) - ‘elasticsearch-6.6.1.rpm’ 
saved [114067654/114067654]

4. Install the Elasticsearch RPM package on each EC2 instance as instructed below.
$ rpm -iv elasticsearch-6.6.1.rpm
warning: elasticsearch-6.6.1.rpm: Header V4 RSA/SHA512 Signature, 
key ID d88e42b4: NOKEY
Preparing packages...
Creating elasticsearch group... OK
Creating elasticsearch user... OK

### NOT starting on installation, please execute the following 
statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service

### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch

5. By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the "--quiet" option must be removed from the ExecStart command line in the elasticsearch.service file.
$ vim /usr/lib/systemd/system/elasticsearch.service
# Remove --quiet by Tony
#ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/ --quiet
ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/ --quiet

6. Configure Elasticsearch
Elasticsearch defaults to using /etc/elasticsearch for runtime configuration. Elasticsearch loads its configuration from the /etc/elasticsearch/elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch (

Update the bind host:
$ vim /etc/elasticsearch/elasticsearch.yml
locate "", and update it to

This opens up ElasticSearch to listen on traffic from all hosts.

Update the cluster name:
$ vim /etc/elasticsearch/elasticsearch.yml
locate " my-application", and update it to tony-es-cluster

Setting the heap size:
By default, Elasticsearch tells the JVM to use a heap with a minimum and maximum size of 1 GB. However, it is important to configure heap size to ensure that Elasticsearch has enough heap available. Elasticsearch will assign the entire heap specified in jvm.options via the Xms (minimum heap size) and Xmx (maximum heap size) settings.
The value for these setting depends on the amount of RAM available on the instance, a rule of thumb is "Set Xmx to no more than 50% of your physical RAM, to ensure that there is enough physical RAM left for kernel file system caches." In our case, the value is 16GB.
$ vim /etc/elasticsearch/jvm.options
Locate "-Xms1g -Xmx1g", and update it to

6. Start ElasticSearch
$ systemctl start elasticsearch.service
$ systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; 
disabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-02-25 01:32:29 UTC; 2s ago
 Main PID: 13803 (java)
   CGroup: /system.slice/elasticsearch.service
           └─13803 /bin/java -Xms16g -Xmx16g -XX:+UseConcMarkSweepGC 
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
-Des.networkaddress.cache.ttl=60 -Des.networkaddr.

7. Verify from API request
$ curl -X GET http://54.1xx.1xx.1xx:9200/
  "name" : "ZAvN4SU",
  "cluster_name" : "tony-es-cluster",
  "cluster_uuid" : "bYSZ8nkqS-mnI8x2F3eHhQ",
  "version" : {
    "number" : "6.6.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "1fd8f69",
    "build_date" : "2019-02-13T17:10:04.160291Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  "tagline" : "You Know, for Search"

Since we enabled the journal entries for ElasticSeach, you can list the entries now:
$ journalctl --unit elasticsearch
-- Logs begin at Mon 2019-02-25 00:58:47 UTC, end at Mon 2019-02-25 01:32:29 UTC. --

Feb 25 01:26:55 ip-172-31-88-104.ec2.internal systemd[1]: Started Elasticsearch.
Feb 25 01:26:55 ip-172-31-88-104.ec2.internal systemd[1]: Starting Elasticsearch...
Feb 25 01:28:13 ip-172-31-88-104.ec2.internal systemd[1]: Stopping Elasticsearch...
Feb 25 01:28:14 ip-172-31-88-104.ec2.internal systemd[1]: Stopped Elasticsearch.
Feb 25 01:29:38 ip-172-31-88-104.ec2.internal systemd[1]: Started Elasticsearch.
Feb 25 01:29:38 ip-172-31-88-104.ec2.internal systemd[1]: Starting Elasticsearch...
Feb 25 01:31:01 ip-172-31-88-104.ec2.internal systemd[1]: Stopping Elasticsearch...
Feb 25 01:31:01 ip-172-31-88-104.ec2.internal systemd[1]: Stopped Elasticsearch.
Feb 25 01:32:29 ip-172-31-88-104.ec2.internal systemd[1]: Started Elasticsearch.
Feb 25 01:32:29 ip-172-31-88-104.ec2.internal systemd[1]: Starting Elasticsearch...

At this point, you have a running ElasticSearch 6.6.1 single node cluster.

ECE - Small Trick: Query Zookeeper Statistics

ECE stands for "Elastic Cloud Enterprise". It shares most of its codebase with Elastic Cloud. The key tenets of the architecture are:
  • Service-oriented architecture
  • Containerization using Docker
  • Deployment state coordination using ZooKeeper
  • Easy access through the Cloud UI

Since ECE is a service-oriented architecture, it makes scale the platform very easy. Different services can have different reliability and performance requirements, as each service can be scaled separately. This also hides some of technical details from you, especially administrators that curious how it works inside. Sometimes it could be really hard to get information out of a vendor container.

In our case, we have ECE 1.x and 2.x both installed in our environment, one of challenges we had was how to monitor the Zookeeper status. The Zookeeper status from the admin console doesn't count :). Fortunately, ECE Zookeeper container exposes its port to "" (i.e,>2192/tcp), this means you can can query some of whitelisted information through the host IP and exposed port.

For example, say one of your Zookeeper role exposes port "2192". If you would like to output a list of variables that could be used for monitoring the health of the cluster. You could query the underline host that runs Zookeeper role like following:
$ echo mntr | nc 2192

zk_version  3.4.0
zk_avg_latency  0
zk_max_latency  0
zk_min_latency  0
zk_packets_received 70
zk_packets_sent 69
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count   4
zk_watch_count  0
zk_ephemerals_count 0
zk_approximate_data_size    27
zk_followers    4                   - only exposed by the Leader
zk_synced_followers 4               - only exposed by the Leader
zk_pending_syncs    0               - only exposed by the Leader
zk_open_file_descriptor_count 23    - only available on Unix platforms
zk_max_file_descriptor_count 1024   - only available on Unix platforms

Then, if you really want to go fancy, you could send the output to a monitoring and alerting system like "Datadog" peoridically (I might make another post for this). But for information query, this is a nice and easy way.

Our Zookeeper version is v3.5.3, and a few useful commands:

  • conf: rint details about serving configuration (not in the whitelist).
  • cons: List full connection/session details for all clients connected to this server. Includes information on numbers of packets received/sent, session id, operation latencies, last operation performed, etc... (not in the whitelist).
  • dump: Lists the outstanding sessions and ephemeral nodes. This only works on the leader (not in the whitelist).
  • envi: Print details about serving environment (not in the whitelist).
  • ruok: Tests if server is running in a non-error state. The server will respond with imok if it is running. Otherwise it will not respond at all.
  • A response of "imok" does not necessarily indicate that the server has joined the quorum, just that the server process is active and bound to the specified client port. Use "stat" for details on state wrt quorum and client connection information.
  • srvr: Lists full details for the server.
  • stat: Lists brief details for the server and connected clients.
  • wchs: Lists brief information on watches for the server (not in the whitelist).
  • wchc: Lists detailed information on watches for the server, by session. This outputs a list of sessions(connections) with associated watches (paths). Note, depending on the number of watches this operation may be expensive (ie impact server performance), use it carefully (not in the whitelist).
  • wchp: Lists detailed information on watches for the server, by path. This outputs a list of paths (znodes) with associated sessions. Note, depending on the number of watches this operation may be expensive (ie impact server performance), use it carefully.
  • mntr: Outputs a list of variables that could be used for monitoring the health of the cluster.