AWS - Configure Rabbitmq Cluster With Auto-scaling Discovery Plugin

RabbitMQ is an open source message broker software or in simple terms, a queue-manager. Applications or services connect to RabbitMQ, they can then, either add messages/tasks to the queue(publisher), or fetch from the queue(subscriber). A communication platform for microservices, you can say.

Each running an instance of the RabbitMQ application is a node. A RabbitMQ cluster (or broker) can consist of multiple nodes. ll of the nodes will share configuration information, such as users, exchanges, and queues.

In this blog,  I will show you how to configure RabbitMQ clustering in AWS EC2 with Auto-scaling discovery by using "rabbitmq_peer_discovery_aws" plugin (https://github.com/rabbitmq/rabbitmq-peer-discovery-aws). There are not many good instructions out there for how to configure auto-cluster discovery, so I decided to make one.

We will be using Ansible for Rabbitmq server installation, configuration and cluster discovery, so you will need to be familiar with Ansible (https://www.ansible.com/).

I've created a Github repo called ansible-aws (https://github.com/tonylixu/ansible-aws), if you are familiar with Ansible already, you can go ahead and check out how I configure RabbitMQ auto clustering with Ansible. The code itself is pretty self-explanatory, so I will not describe the details, go ahead and read the source code.

Important things:

• plugins: You will need rabbitmq_aws, rabbitmq_peer_discovery_aws and rabbitmq_autocluster plugins
• Erlang cookie: You need to use the same erlang cookie for all of your RabbitMQ nodes.
• Queue mirroring: By default, incoming, ondemand, index and parser queues are mirrored, it is easy to configure mirror for other queues, checkout "create mirror policy" task.

To run:

Simply pre-deploy the rabbit role to your EC2 instance, then do a "ansible-playbook -f main.yml" run. When a new RabbitMQ node is initialized, it will auto join the cluster.

Example:

Clustering:

Mirroring: