Ansible is the simplest way to automate apps and IT infrastructure. Application Deployment + Configuration Management + Continuous Delivery. We use Ansible heavily for our AWS deployment stack.
When working with Ansible template, sometimes you will need to split a string and use part of its value. For example, you have a application template which setup the application base url, like follow:
<?xml version='1.1' encoding='UTF-8'?>
<app.model.AppLocationConfiguration>
<adminAddress>admin@app.com</adminAddress>
<appUrl>https://{{ app.split("-")[0] | lower }}-{{ env | lower }}.app.com/</appUrl>
</app.model.AppLocationConfiguration>
We have a variable "app" containing the string "asia-101.test.org" and want to extract the geolocation part of this string, we could just use "{{ app.split("-")[0] | lower }}"
The constructed url will be "https://asia-prod.app.com".
Ansible use Jinja2 for formatting, so you can apply the split filter (https://jinja.palletsprojects.com/en/2.10.x/templates/#filters).
Friday, January 24, 2020
Sunday, August 04, 2019
AWS CDK Python - Sample Project
This blog shows you how to create a sample project using AWS CDK Python. If you don't have the environment setup yet, check out my previous blog: https://www.lixu.ca/2019/08/aws-cdk-python-how-to-get-started.html
$ source cdk-env/bin/activate
(cdk-venv) $ mkdir sample-dev
(cdk-venv) $ cd sample-dev
(cdk-venv) $ cdk init
Available templates:
* app: Template for a CDK Application
└─ cdk init app --language=[csharp|fsharp|java|javascript|python|typescript]
* lib: Template for a CDK Construct Library
└─ cdk init lib --language=typescript
* sample-app: Example CDK Application with some constructs
└─ cdk init sample-app --language=[javascript|python|typescript]
2. The return information from cdk is very self-explanatary, we can choose "sample-app" so by the creation of this app, we have some exisitng constructs
$ cdk init --language python sample-app
Applying project template sample-app for python
Initializing a new git repository...
Executing Creating virtualenv...
...
3. The entire sample project looks like this:
.
├── README.md - Readme file
├── app.py - Defines app stacks, also the "main" file
├── cdk.json - Configuration file for CDK that defines what executable CDK should run to generate CDK construct tree
├── hello
│ ├── __init__.py - To make a Python module
│ ├── hello.egg-info
│ │ ├── PKG-INFO
│ │ ├── SOURCES.txt
│ │ ├── dependency_links.txt
│ │ ├── requires.txt
│ │ └── top_level.txt
│ ├── hello_construct.py - A custom CDK construct defined for use in your CDK application.
│ └── hello_stack.py - A custom CDK stack construct for use in your CDK application
├── requirements.txt - Required Python modules
├── setup.py - Defines how this Python package would be constructed and what the dependencies are
└── tests - Test folder
├── __init__.py
└── unit - ALl the unit tests
├── __init__.py
└── test_hello_construct.py
You can take time to read and understand the source code, but esentially what this code does is to create two CloudFormation templates: "hello-cdk-1" and "hello-cdk-2", and deploy them to "us-east-1" and "us-west-1" regions. Each template includes:
4. Give a read of the README.md file, you can ignore the Python virtual env part, as we are already using a virtual env. As the README.md suggest, we going to install all required Python packages and run some unit tests.
(cdk-venv) $ pip install -r requirements.txt
Collecting pytest (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/69/1d/2430053122a3c6106f7fd1ff0bc68eb73e27db8f951db70fcd942da52c7b/pytest-5.0.1-py3-none-any.whl (221kB)
100% |████████████████████████████████| 225kB 4.5MB/s
(cdk-venv) $ pytest
================================================================================================================= test session starts
=================================================================================================================
platform darwin -- Python 3.7.1, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /Users/txu/code/flashhop-dev
collected 1 item
tests/unit/test_hello_construct.py .
5. Generate AWS CloudFormation template
(cdk-venv) $ cdk synth
Successfully synthesized to /Users/txu/code/flashhop-dev/cdk.out
Supply a stack name (hello-cdk-1, hello-cdk-2) to display its template.
All the output files will be located in "cdk.out"
6. Deploy the generated templates to AWS account and validate the right resouces got created.
(cdk-venv) $ cdk deploy --profile YOU_AWS_PROFILE hello-cdk-1
...
7. To destroy the stack
(cdk-venv) $ cdk --profile flashhop-dev destroy hello-cdk-1
Are you sure you want to delete: hello-cdk-1 (y/n)? y
hello-cdk-1: destroying...
At this point, you learned how to use AWS CDK CLI to initialize a new Python project/app, how to synthesize hte Python code into AWS Cloudformation templates, deploy them, provision your infrastructure and destroy your infrastructure. You can learn more from AWS CDK developer documentations (https://docs.aws.amazon.com/cdk/latest/guide/home.html)
Create the Python Application
1. Fire up you Python3 virtual environment, create a project directory contains your Python project source code$ source cdk-env/bin/activate
(cdk-venv) $ mkdir sample-dev
(cdk-venv) $ cd sample-dev
(cdk-venv) $ cdk init
Available templates:
* app: Template for a CDK Application
└─ cdk init app --language=[csharp|fsharp|java|javascript|python|typescript]
* lib: Template for a CDK Construct Library
└─ cdk init lib --language=typescript
* sample-app: Example CDK Application with some constructs
└─ cdk init sample-app --language=[javascript|python|typescript]
2. The return information from cdk is very self-explanatary, we can choose "sample-app" so by the creation of this app, we have some exisitng constructs
$ cdk init --language python sample-app
Applying project template sample-app for python
Initializing a new git repository...
Executing Creating virtualenv...
...
3. The entire sample project looks like this:
.
├── README.md - Readme file
├── app.py - Defines app stacks, also the "main" file
├── cdk.json - Configuration file for CDK that defines what executable CDK should run to generate CDK construct tree
├── hello
│ ├── __init__.py - To make a Python module
│ ├── hello.egg-info
│ │ ├── PKG-INFO
│ │ ├── SOURCES.txt
│ │ ├── dependency_links.txt
│ │ ├── requires.txt
│ │ └── top_level.txt
│ ├── hello_construct.py - A custom CDK construct defined for use in your CDK application.
│ └── hello_stack.py - A custom CDK stack construct for use in your CDK application
├── requirements.txt - Required Python modules
├── setup.py - Defines how this Python package would be constructed and what the dependencies are
└── tests - Test folder
├── __init__.py
└── unit - ALl the unit tests
├── __init__.py
└── test_hello_construct.py
You can take time to read and understand the source code, but esentially what this code does is to create two CloudFormation templates: "hello-cdk-1" and "hello-cdk-2", and deploy them to "us-east-1" and "us-west-1" regions. Each template includes:
- IAM user with predefined policies
- AWS sqs queue
- AWS sns topic and subscribe to the sqs queue
- Few AWS S3 buckets
4. Give a read of the README.md file, you can ignore the Python virtual env part, as we are already using a virtual env. As the README.md suggest, we going to install all required Python packages and run some unit tests.
(cdk-venv) $ pip install -r requirements.txt
Collecting pytest (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/69/1d/2430053122a3c6106f7fd1ff0bc68eb73e27db8f951db70fcd942da52c7b/pytest-5.0.1-py3-none-any.whl (221kB)
100% |████████████████████████████████| 225kB 4.5MB/s
(cdk-venv) $ pytest
================================================================================================================= test session starts
=================================================================================================================
platform darwin -- Python 3.7.1, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /Users/txu/code/flashhop-dev
collected 1 item
tests/unit/test_hello_construct.py .
5. Generate AWS CloudFormation template
(cdk-venv) $ cdk synth
Successfully synthesized to /Users/txu/code/flashhop-dev/cdk.out
Supply a stack name (hello-cdk-1, hello-cdk-2) to display its template.
All the output files will be located in "cdk.out"
6. Deploy the generated templates to AWS account and validate the right resouces got created.
(cdk-venv) $ cdk deploy --profile YOU_AWS_PROFILE hello-cdk-1
...
7. To destroy the stack
(cdk-venv) $ cdk --profile flashhop-dev destroy hello-cdk-1
Are you sure you want to delete: hello-cdk-1 (y/n)? y
hello-cdk-1: destroying...
At this point, you learned how to use AWS CDK CLI to initialize a new Python project/app, how to synthesize hte Python code into AWS Cloudformation templates, deploy them, provision your infrastructure and destroy your infrastructure. You can learn more from AWS CDK developer documentations (https://docs.aws.amazon.com/cdk/latest/guide/home.html)
AWS CDK Python - How To Get Started
Assuming you are already familiar with the concept of "IaC" (Infrastructure As Code), well in AWS context, "IaC" means CloudFormation.
CloudFormation allows you to define your AWS infrastructure in JSON or YAML files that can be managed witin your source code repository (Git for example). You can do pull requests and code reviews. When everything looks good, you can use these files as input into an automated process (CI/CD) that deploys your infrastructure changes.
So AWS Python CDK builds on AWS CloudFormation and uses it as the engine for provisioning AWS resources. It allows you to compose new abstractions that hide details and simplify common use cases, then it packages the code up as a library in Python. This blog shows you how to get sarted with AWS Python CDK.
CloudFormation allows you to define your AWS infrastructure in JSON or YAML files that can be managed witin your source code repository (Git for example). You can do pull requests and code reviews. When everything looks good, you can use these files as input into an automated process (CI/CD) that deploys your infrastructure changes.
So AWS Python CDK builds on AWS CloudFormation and uses it as the engine for provisioning AWS resources. It allows you to compose new abstractions that hide details and simplify common use cases, then it packages the code up as a library in Python. This blog shows you how to get sarted with AWS Python CDK.
Prerequisites
- Node.js (>= 8.11.x): Why? AWS CDK is developed in TypeScript and transpiled to JavaScript. Bindings for Python make use of the AWS CDK back-end running on Node.js, as does the cdk command-line tool.
- Your AWS profile and credentials
- Python >= 3.6
Install AWS Python CDK
1. Check npm version
2. Install AWS CDK$ npm -v 6.10.0
3. Check CDK version$ npm install -g aws-cdk/usr/local/bin/cdk -> /usr/local/lib/node_modules/aws-cdk/bin/cdk > core-js@2.6.9 postinstall /usr/local/lib/node_modules/aws-cdk/node_modules/core-js > node scripts/postinstall || echo "ignore" Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library! The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: > https://opencollective.com/core-js > https://www.patreon.com/zloirock Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -) + aws-cdk@1.3.0
$ cdk --version 1.3.0 (build bba9914)4. Check Python version
$ python3.7 -V Python 3.7.15. Create a Python virtual env
$ python3.7 -m venv cdk-venv $ source cdk-venv/bin/activate (cdk-venv) $6. Updating Python Dependencies
(cdk-venv) $ which pip /Users/txu/code/cdk-venv/bin/pip (cdk-venv) $ pip install --upgrade aws-cdk.cdk Collecting aws-cdk.cdk Downloading https://files.pythonhosted.org/packages/45/77/07f23d943aece234c91230844595f5495ae6ef5be668b2c9e818ee479ef3/aws_cdk.cdk-0.36.1-py3-none-any.whl Collecting publication>=0.0.3 (from aws-cdk.cdk) ... Installing collected packages: publication, typing-extensions, mypy-extensions, attrs, cattrs, six, python-dateutil, jsii, aws-cdk.cdk Successfully installed attrs-19.1.0 aws-cdk.cdk-0.36.1 cattrs-0.9.0 jsii-0.13.4 mypy-extensions-0.4.1 publication-0.0.3 python-dateutil-2.8.0 six-1.12.0 typing-extensions-3.7.4
7. Create AWS profiles (If you don't have one already)
(cdk-venv) $ aws configure --profile test-dev AWS Access Key ID [None]: xxxxx AWS Secret Access Key [None]: xxxx Default region name [None]: ca-central-1 Default output format [None]: json8. Check CDK version
(cdk-venv) $ cdk --version 1.3.0 (build bba9914)
At this point, you have successfully installed AWS CDK for Python and connected your Python with AWS CDK.
In our next blog, we will create a example app stack.
Sunday, July 21, 2019
AWS EC2 - Install and Configure virtualenvwrapper with Python3
virtualenvwrapper is a very nice wrapper of Python virtualenv tool. It includes wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies. I really like that how easily it allows you to switch between different projects and virtual environments, saves you a lot of time. This blog shows you how to install and configure it on a AWS EC2 instance, using Python3
You can see that a "new_proj" folder inside "~/code" and a new virtual environment "new_proj" inside "~/venvs" are created automatically and the virtual environment of "new_proj" automatically activated. Let's create a second project and show how easily we can switch between projects and their virtual environments.
More information, checkout https://virtualenvwrapper.readthedocs.io/en/latest/
Install Python3
$ sudo yum install python3 $ pip3 -V $ pip3 -V pip 9.0.3 from /usr/lib/python3.7/site-packages (python 3.7)
Install virtualenvwrapper globally
$ sudo pip3 install virtualenvwrapper WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead. Collecting virtualenvwrapper ... Installing collected packages: virtualenv, virtualenv-clone, pbr, six, stevedore, virtualenvwrapper Running setup.py install for virtualenvwrapper ... done Successfully installed pbr-5.4.1 six-1.12.0 stevedore-1.30.1 virtualenv-16.6.2 virtualenv-clone-0.5.3 virtualenvwrapper-4.8.4
Create project folder and virtual environment folder
$ ll /home/ec2-user/ total 8 drwxrwxr-x 2 ec2-user ec2-user 6 Jul 21 20:27 code drwxrwxr-x 2 ec2-user ec2-user 6 Jul 21 20:27 venvs
Locates "virtualenvwrapper.sh"
$ which virtualenvwrapper.sh /usr/local/bin/virtualenvwrapper.sh
Update bash profile
$ vim ~/.bash_profile Append the following: # Configure virtualenvwrapper export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3 source /usr/local/bin/virtualenvwrapper.sh export WORKON_HOME="/home/ec2-user/venvs" export PROJECT_HOME="/home/ec2-user/code"
Resource bash profile
$ source ~/.bash_profile virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/premkproject virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postmkproject virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/initialize virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/premkvirtualenv virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postmkvirtualenv virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/prermvirtualenv virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postrmvirtualenv virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/predeactivate virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postdeactivate virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/preactivate virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/postactivate virtualenvwrapper.user_scripts creating /home/ec2-user/.virtualenvs/get_env_details
Create a new project
$ mkproject new_proj Using base prefix '/usr' No LICENSE.txt / LICENSE found in source New python executable in /home/ec2-user/venvs/new_proj/bin/python3 Also creating executable in /home/ec2-user/venvs/new_proj/bin/python Installing setuptools, pip, wheel... done. virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/predeactivate virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/postdeactivate virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/preactivate virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/postactivate virtualenvwrapper.user_scripts creating /home/ec2-user/venvs/new_proj/bin/get_env_details Creating /home/ec2-user/code/new_proj Setting project for new_proj to /home/ec2-user/code/new_proj (new_proj) [ec2-user@ip-172-31-36-216 new_proj]$ ls ~/venvs/ new_proj (new_proj) [ec2-user@ip-172-31-36-216 new_proj]$ ls ~/code/ new_proj
You can see that a "new_proj" folder inside "~/code" and a new virtual environment "new_proj" inside "~/venvs" are created automatically and the virtual environment of "new_proj" automatically activated. Let's create a second project and show how easily we can switch between projects and their virtual environments.
$ mkproject new_proj2 ... Creating /home/ec2-user/code/new_proj2 Setting project for new_proj2 to /home/ec2-user/code/new_proj2
Switch to "new_proj" and its virtual env:
(new_proj2) [ec2-user@hostname new_proj2]$ workon new_proj (new_proj) [ec2-user@hostname new_proj]$You can see how easily you are able to switch between projects without deactivate, reactivate, switch folder, ..etc.!
You can list all the exisitng virtual environments:
# List environments $ workon new_proj2 new_proj
More information, checkout https://virtualenvwrapper.readthedocs.io/en/latest/
Sunday, May 05, 2019
Datadog- Artifactory JMX Monitoring
Datadog JMX integration collects metrics from applications that expose JMX metrics. A lightweight Java plugin named "JMXFetch" is called by the Datadog Agent to connect to the MBean server and to collect metrics, it can also send service checks that report on the status of your monitored instances.
This plugin sends metrics to the Datadog Agent using DogStatsD server running within the Agent. In the blog we will show you how to collect and send metrics from Artifactory application.
You can also use the new way of MBean configuration, as described following (https://docs.datadoghq.com/integrations/java/):
You could use "datadog-agent jmx list collected" command to check what JMX metrics that you are collecting.
This plugin sends metrics to the Datadog Agent using DogStatsD server running within the Agent. In the blog we will show you how to collect and send metrics from Artifactory application.
Prerequisites:
- You have a running Artifactory application with JMX enabled (https://jfrog.com/knowledge-base/how-do-i-use-jmx-remote-monitoring/)
- You have a running Datadog agent which has access to your Artifactory JMX port
Configuration:
You need to enable JMX monitoring in Datadog by creating a file "/etc/datadog-agent/conf.d/jmx.d/conf.yaml" with the following configuration:init_config:
new_gc_metrics: true
instances:
- host: localhost # If DD agent is running on the same host
port: 9010
user: <USER_NAME>
password: <PASSWORD>
name: artifactory-app
tags:
env: pord
conf:
- include:
domain: org.jfrog.artifactory
bean:
- org.jfrog.artifactory:instance=Artifactory, type=Storage,prop=Binary Storage
- org.jfrog.artifactory:instance=Artifactory, type=Repositories,prop=test-repo
attribute:
# Ginary storage size
- Size
# Repo
- ArtifactsTotalSize
- ArtifactsCount
- include:
domain: Catalina
bean:
- Catalina:type=ThreadPool,name="http-nio-8081"
attribute:
- maxThreads
- currentThreadCount
- currentThreadBusy
This following graph shows you a visual mapping between Mbeans and the actual Datadog configuration.
You can also use the new way of MBean configuration, as described following (https://docs.datadoghq.com/integrations/java/):
conf:
- include:
domain: org.apache.cassandra.db
attribute:
- BloomFilterDiskSpaceUsed
- BloomFilterFalsePositives
- BloomFilterFalseRatio
- Capacity
- CompressionRatio
- CompletedTasks
- ExceptionCount
- Hits
- RecentHitRate
But I couldn't get the new confiuration to work as the old one yet, the new one looks much easier to config, but it didn't work for me, for some reason it is not sending metrics to Datadog platform.You could use "datadog-agent jmx list collected" command to check what JMX metrics that you are collecting.
Note1:
To run more than one JMX check, create configuration files with the format jmx_<INDEX>.d/conf.yaml (e.g. jmx_1.d/conf.yaml, jmx_2.d/conf.yaml, etc). Each folder should be stored in the conf.d directory. Include the is_jmx option set to true in those configuration files.The 350 metric limit
Due to the nature of these integrations, it is possible to submit an extremely high number of metrics directly to Datadog. Many customers agree that some of these metrics are not needed. Therefore, Datadog has set the limit at 350 metrics.Sunday, April 28, 2019
Logstash - AWS S3 Bucket As Data Input
You can use the "S3 input plugin" to stream events from files from a AWS S3 bucket. Each line from each file generates an event. Files ending in ".gz" are handled as gzip'ed files. Glacier files will be skipped.
Logstash version: 6.7.1
Here is a basic configuration for streaming data:
Files in this "logstash-test-aws-s3-bucket" AWS S3 bucket start with "logstash-" will match (including folders).
Start your logstash with "logstash -f confg/s3-input-logstash.conf", you should start seeing data coming into your Elasticsearch cluster.
Logstash version: 6.7.1
Here is a basic configuration for streaming data:
input {
s3 {
"access_key_id" => "1234"
"secret_access_key" => "secret"
"bucket" => "logstash-test-aws-s3-bucket"
"additional_settings" => {
"force_path_style" => true
"follow_redirects" => false
}
"region" => "us-east-1"
"prefix" => "logstash-"
"type" => "s3"
}
}
output {
elasticsearch {
cacert => "/path/to/cert"
hosts => "https://elasticsearch1.com:9243"
index => "test-index-%{+YYYY.MM}"
user => "logstash"
password => "logstash"
}
}
Files in this "logstash-test-aws-s3-bucket" AWS S3 bucket start with "logstash-" will match (including folders).
Start your logstash with "logstash -f confg/s3-input-logstash.conf", you should start seeing data coming into your Elasticsearch cluster.
Logstash - Send Data To Multiple Elastic Clusters
It is possible to send same data from one Logstash server to multiple Elasticsearch clusters. It is easy to configure, you just need to have multiple outputs definitions for the same events. However, it is not easy to find a clear example from the internet, so I've decided to make this blog.
Note the JSON file content need to be in one line.
This setup is an all or nothing, if one of the output is down, the second will not work. You should think do you really need this setup at the first place, now you have two copy of data that you need to keep in sync, maybe output to one place, and have the role based control is a better option.
My example Logstash configuration file:
input {
file {
path => ["/path/to/json/file"]
start_position => "beginning"
sincedb_path => "/dev/null"
exclude => "*.gz"
}
}
filter {
mutate {
replace => [ "message", "%{message}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/ {
json { source => message }
}
}
output {
elasticsearch {
cacert => "/path/to/cert"
hosts => "https://elasticsearch1.com:9243"
index => "test-index-%{+YYYY.MM}"
user => "logstash"
password => "logstash"
}
elasticsearch {
cacert => "/path/to/cert"
hosts => "https://elasticsearch2.com:9243"
index => "test-index-%{+YYYY.MM}"
user => "logstash"
password => "logstash"
}
}
Example of JSON file:
{"foo":"bar", "bar": "foo"}
{"hello":"world", "goodnight": "moon"}
Note the JSON file content need to be in one line.
This setup is an all or nothing, if one of the output is down, the second will not work. You should think do you really need this setup at the first place, now you have two copy of data that you need to keep in sync, maybe output to one place, and have the role based control is a better option.
Sunday, March 03, 2019
Octopus Deploy - How To Install Octopus Deploy In AWS EC2
"Octopu Deploy" is an industry leading automated deployment and release management tool used by leading continuous delivery teams worldwide. It is designed to simplify deployment of .NET applications, Windows Services and databases.
This blog shows you how to install the lastest LTS version of "Octopus Deploy" in AWS EC2.
Environment:
- Octopus Deploy: 2018.10.5 LTS
- EC2: t2.large
- OS: "Windows Server 2019 Datacenter"
Installation Steps:
1. Launch a Windows EC2 instance
2. Select "General purpose", "t2.large"
3. Configure Storage
4. Configure tags
5. Configure security group
6. Once the instance is up and running, RDP into the server, you can get the connection details from "Connect" button.
7. Open up IE in the remote server, download "Octopus Deploy 2018.10.5 LTS"
Note: by default, the stupid "Internet Explorer Enhanced Security Configuration" is on, you need to turn it off so you can download things.
8. Double click to install "Octopus.2018.10.5"
9. Complete the Octopus Deploy Server Setup Wizard
Note: You will need SQL Database for Octopus Deploy, if you don't have one, you can download the "SQL Server Express (free)" from Octopus "Database" installation window.
10. Install SQL server express basic
11. Connect to local "SQLEXPRESS" service, and create a "Octopus" database
12. Finally, install.
13. Congratulations, now you have successfully install "Octopu Deploy" in AWS Ec2 instance!
Now login into your Octopus Deploy and start play around with it!
Sunday, February 24, 2019
ElasticSearch - Part 1 - How To Deploy Single Node Cluster On AWS EC2
This blog shows you how to deploy and manage your own ElasticSeacrh cluster on AWS EC2.
Choose "t2.2xlarge" instance type, which provides 8vCPUs, 32 GB of memory and EBS volume for data, which is a reasonable starting point. Go ahead and start up the instance. Two thing to notice:
1. Log into EC2 instance:
2. Install Javav1.8.0
3. Download ElasticSearch 6.6.1
4. Install the Elasticsearch RPM package on each EC2 instance as instructed below.
5. By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the "--quiet" option must be removed from the ExecStart command line in the elasticsearch.service file.
6. Configure Elasticsearch
Elasticsearch defaults to using /etc/elasticsearch for runtime configuration. Elasticsearch loads its configuration from the /etc/elasticsearch/elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch (https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html).
Update the bind host:
Update the cluster name:
Setting the heap size:
By default, Elasticsearch tells the JVM to use a heap with a minimum and maximum size of 1 GB. However, it is important to configure heap size to ensure that Elasticsearch has enough heap available. Elasticsearch will assign the entire heap specified in jvm.options via the Xms (minimum heap size) and Xmx (maximum heap size) settings.
The value for these setting depends on the amount of RAM available on the instance, a rule of thumb is "Set Xmx to no more than 50% of your physical RAM, to ensure that there is enough physical RAM left for kernel file system caches." In our case, the value is 16GB.
6. Start ElasticSearch
7. Verify from API request
Since we enabled the journal entries for ElasticSeach, you can list the entries now:
At this point, you have a running ElasticSearch 6.6.1 single node cluster.
Environment:
- Instance: t2.2xlarge (32GB Mem, 8vCPUs)
- ElasticSearch version: 6.6.1
Provision AWS EC2 instance:
Elasticsearch runs on various operating systems such as CentOS, Redhat, Ubuntu, and Amazon Linux. We suggest using the latest Amazon Linux AMI.Choose "t2.2xlarge" instance type, which provides 8vCPUs, 32 GB of memory and EBS volume for data, which is a reasonable starting point. Go ahead and start up the instance. Two thing to notice:
- 1. The security group open ports:
- - port 22: SSH
- - port 9200: ElasticSearch requests
- 2. The storage:
- - 30GB; You can expand later
Install ElasticSearch
Once the EC2 instance is up and running, we can start ElasticSearch installation.1. Log into EC2 instance:
$ ssh -i ~/.ssh/tony-aws.pem ec2-user@54.1xx.1xx.xxx $ sudo su - $ df -h
Filesystem Size Used Avail Use% Mounted on devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 16G 472K 16G 1% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/xvda1 30G 1.2G 29G 4% / tmpfs 3.2G 0 3.2G 0% /run/user/0 tmpfs 3.2G 0 3.2G 0% /run/user/1000
2. Install Javav1.8.0
$ yum install java-1.8.0-openjdk
Installed: java-1.8.0-openjdk.x86_64 1:1.8.0.191.b12-0.amzn2
3. Download ElasticSearch 6.6.1
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.1.rpm --2019-02-25 01:02:20-- https://artifacts.elastic.co/downloads/
elasticsearch/elasticsearch-6.6.1.rpm Resolving artifacts.elastic.co (artifacts.elastic.co)... 151.101.250.222,
2a04:4e42:3b::734 Connecting to artifacts.elastic.co (artifacts.elastic.co)|151.101.250.222
|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 114067654 (109M) [application/octet-stream] Saving to: ‘elasticsearch-6.6.1.rpm’ 2019-02-25 01:02:22 (115 MB/s) - ‘elasticsearch-6.6.1.rpm’
saved [114067654/114067654]
4. Install the Elasticsearch RPM package on each EC2 instance as instructed below.
$ rpm -iv elasticsearch-6.6.1.rpm warning: elasticsearch-6.6.1.rpm: Header V4 RSA/SHA512 Signature,
key ID d88e42b4: NOKEY Preparing packages... Creating elasticsearch group... OK Creating elasticsearch user... OK elasticsearch-0:6.6.1-1.noarch ### NOT starting on installation, please execute the following
statements to configure elasticsearch service to start automatically using systemd sudo systemctl daemon-reload sudo systemctl enable elasticsearch.service ### You can start elasticsearch service by executing sudo systemctl start elasticsearch.service Created elasticsearch keystore in /etc/elasticsearch
5. By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the "--quiet" option must be removed from the ExecStart command line in the elasticsearch.service file.
$ vim /usr/lib/systemd/system/elasticsearch.service
# Remove --quiet by Tony
#ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet
ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet
6. Configure Elasticsearch
Elasticsearch defaults to using /etc/elasticsearch for runtime configuration. Elasticsearch loads its configuration from the /etc/elasticsearch/elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch (https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html).
Update the bind host:
$ vim /etc/elasticsearch/elasticsearch.yml locate "network.host: 192.168.0.1", and update it to network.host: 0.0.0.0This opens up ElasticSearch to listen on traffic from all hosts.
Update the cluster name:
$ vim /etc/elasticsearch/elasticsearch.yml locate "cluster.name: my-application", and update it to cluster.name: tony-es-cluster
Setting the heap size:
By default, Elasticsearch tells the JVM to use a heap with a minimum and maximum size of 1 GB. However, it is important to configure heap size to ensure that Elasticsearch has enough heap available. Elasticsearch will assign the entire heap specified in jvm.options via the Xms (minimum heap size) and Xmx (maximum heap size) settings.
The value for these setting depends on the amount of RAM available on the instance, a rule of thumb is "Set Xmx to no more than 50% of your physical RAM, to ensure that there is enough physical RAM left for kernel file system caches." In our case, the value is 16GB.
$ vim /etc/elasticsearch/jvm.options Locate "-Xms1g -Xmx1g", and update it to -Xms16g -Xmx16g
6. Start ElasticSearch
$ systemctl start elasticsearch.service $ systemctl status elasticsearch.service ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
disabled; vendor preset: disabled) Active: active (running) since Mon 2019-02-25 01:32:29 UTC; 2s ago Docs: http://www.elastic.co Main PID: 13803 (java) CGroup: /system.slice/elasticsearch.service └─13803 /bin/java -Xms16g -Xmx16g -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-Des.networkaddress.cache.ttl=60 -Des.networkaddr.
7. Verify from API request
$ curl -X GET http://54.1xx.1xx.1xx:9200/
{
"name" : "ZAvN4SU",
"cluster_name" : "tony-es-cluster",
"cluster_uuid" : "bYSZ8nkqS-mnI8x2F3eHhQ",
"version" : {
"number" : "6.6.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "1fd8f69",
"build_date" : "2019-02-13T17:10:04.160291Z",
"build_snapshot" : false,
"lucene_version" : "7.6.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
Since we enabled the journal entries for ElasticSeach, you can list the entries now:
$ journalctl --unit elasticsearch -- Logs begin at Mon 2019-02-25 00:58:47 UTC, end at Mon 2019-02-25 01:32:29 UTC. -- Feb 25 01:26:55 ip-172-31-88-104.ec2.internal systemd[1]: Started Elasticsearch. Feb 25 01:26:55 ip-172-31-88-104.ec2.internal systemd[1]: Starting Elasticsearch... Feb 25 01:28:13 ip-172-31-88-104.ec2.internal systemd[1]: Stopping Elasticsearch... Feb 25 01:28:14 ip-172-31-88-104.ec2.internal systemd[1]: Stopped Elasticsearch. Feb 25 01:29:38 ip-172-31-88-104.ec2.internal systemd[1]: Started Elasticsearch. Feb 25 01:29:38 ip-172-31-88-104.ec2.internal systemd[1]: Starting Elasticsearch... Feb 25 01:31:01 ip-172-31-88-104.ec2.internal systemd[1]: Stopping Elasticsearch... Feb 25 01:31:01 ip-172-31-88-104.ec2.internal systemd[1]: Stopped Elasticsearch. Feb 25 01:32:29 ip-172-31-88-104.ec2.internal systemd[1]: Started Elasticsearch. Feb 25 01:32:29 ip-172-31-88-104.ec2.internal systemd[1]: Starting Elasticsearch...
At this point, you have a running ElasticSearch 6.6.1 single node cluster.
ECE - Small Trick: Query Zookeeper Statistics
ECE stands for "Elastic Cloud Enterprise". It shares most of its codebase with Elastic Cloud. The key tenets of the architecture are:
Since ECE is a service-oriented architecture, it makes scale the platform very easy. Different services can have different reliability and performance requirements, as each service can be scaled separately. This also hides some of technical details from you, especially administrators that curious how it works inside. Sometimes it could be really hard to get information out of a vendor container.
In our case, we have ECE 1.x and 2.x both installed in our environment, one of challenges we had was how to monitor the Zookeeper status. The Zookeeper status from the admin console doesn't count :). Fortunately, ECE Zookeeper container exposes its port to "0.0.0.0" (i.e, 0.0.0.0:2192->2192/tcp), this means you can can query some of whitelisted information through the host IP and exposed port.
For example, say one of your Zookeeper role exposes port "2192". If you would like to output a list of variables that could be used for monitoring the health of the cluster. You could query the underline host that runs Zookeeper role like following:
Then, if you really want to go fancy, you could send the output to a monitoring and alerting system like "Datadog" peoridically (I might make another post for this). But for information query, this is a nice and easy way.
- Service-oriented architecture
- Containerization using Docker
- Deployment state coordination using ZooKeeper
- Easy access through the Cloud UI
Since ECE is a service-oriented architecture, it makes scale the platform very easy. Different services can have different reliability and performance requirements, as each service can be scaled separately. This also hides some of technical details from you, especially administrators that curious how it works inside. Sometimes it could be really hard to get information out of a vendor container.
In our case, we have ECE 1.x and 2.x both installed in our environment, one of challenges we had was how to monitor the Zookeeper status. The Zookeeper status from the admin console doesn't count :). Fortunately, ECE Zookeeper container exposes its port to "0.0.0.0" (i.e, 0.0.0.0:2192->2192/tcp), this means you can can query some of whitelisted information through the host IP and exposed port.
For example, say one of your Zookeeper role exposes port "2192". If you would like to output a list of variables that could be used for monitoring the health of the cluster. You could query the underline host that runs Zookeeper role like following:
$ echo mntr | nc ecedc1h1.lixu.ca 2192 zk_version 3.4.0 zk_avg_latency 0 zk_max_latency 0 zk_min_latency 0 zk_packets_received 70 zk_packets_sent 69 zk_outstanding_requests 0 zk_server_state leader zk_znode_count 4 zk_watch_count 0 zk_ephemerals_count 0 zk_approximate_data_size 27 zk_followers 4 - only exposed by the Leader zk_synced_followers 4 - only exposed by the Leader zk_pending_syncs 0 - only exposed by the Leader zk_open_file_descriptor_count 23 - only available on Unix platforms zk_max_file_descriptor_count 1024 - only available on Unix platforms
Then, if you really want to go fancy, you could send the output to a monitoring and alerting system like "Datadog" peoridically (I might make another post for this). But for information query, this is a nice and easy way.
Our Zookeeper version is v3.5.3, and a few useful commands:
- conf: rint details about serving configuration (not in the whitelist).
- cons: List full connection/session details for all clients connected to this server. Includes information on numbers of packets received/sent, session id, operation latencies, last operation performed, etc... (not in the whitelist).
- dump: Lists the outstanding sessions and ephemeral nodes. This only works on the leader (not in the whitelist).
- envi: Print details about serving environment (not in the whitelist).
- ruok: Tests if server is running in a non-error state. The server will respond with imok if it is running. Otherwise it will not respond at all.
- A response of "imok" does not necessarily indicate that the server has joined the quorum, just that the server process is active and bound to the specified client port. Use "stat" for details on state wrt quorum and client connection information.
- srvr: Lists full details for the server.
- stat: Lists brief details for the server and connected clients.
- wchs: Lists brief information on watches for the server (not in the whitelist).
- wchc: Lists detailed information on watches for the server, by session. This outputs a list of sessions(connections) with associated watches (paths). Note, depending on the number of watches this operation may be expensive (ie impact server performance), use it carefully (not in the whitelist).
- wchp: Lists detailed information on watches for the server, by path. This outputs a list of paths (znodes) with associated sessions. Note, depending on the number of watches this operation may be expensive (ie impact server performance), use it carefully.
- mntr: Outputs a list of variables that could be used for monitoring the health of the cluster.
Subscribe to:
Posts (Atom)