Running distributed Erlang & Elixir applications on Docker

by Oleg Tarasenko

The docker platform has been around for a couple of years already, so let’s briefly review which benefits it provides, and why you should consider using it:

An easy means to boot your production services on the desktop

I still remember how complex it was to bring up MySql on OS X in the past. But it would be even worse if you consider a scenario where your production environment is running some specific version of MySQL, especially built on non OS X.

How would you bring that up? Luckily this problem just goes away when you’re using docker, as you can bring up any version of program as easily as executing the command docker run mysql:5.3.

The ability to work against a realistic testing environment, without having to implement service mocks

It was always an issue. If you wanted to test your software properly, you had to bring up (or mock) a lot of related services. And it always created issues, as a mock never really matches the characteristics of a complete running service.

And with docker you can just plug any software that your application relies upon, and have it right there as a part of your testing environment.

Creating realistic cloud infrastructure, using containers

Finally, docker containers can be spun up by tools like Apache Mesos forming full featured clouds, deployable in the effortless and clear manner, and incredibly stable!

Interested? Lets see which benefits the technology can give you when it comes to the Erlang and Elixir worlds.

Testing distributed erlang applications with docker

Let’s consider a case when we need to create and test a distributed erlang application, running on the same machine, and let’s say we want to have it running, but it relies upon some extra services.

This kind of setup can be described in the following docker-compose file:

version: "3"services:c1:image: erlang:latest# We're using the following command to bring up erlang shell# for the example purposes, but in the other case the command# will describe a running containercommand: erl -noshell -name [email protected] -setcookie cookiecontainer_name: host1.comnetworks:- net1

c2:image: erlang:17.5command: erl -noshell -name [email protected] -setcookie cookiecontainer_name: host2.comnetworks:- net1

ejabberd:image: rroemhild/ejabberdports:- 5222:5222- 5269:5269- 5280:5280container_name: host3.comenvironment:- [email protected] XMPP_DOMAIN=test.io- ERLANG_COOKIE=cookie- [email protected] [email protected]:admin- EJABBERD_SKIP_MODULES_UPDATE=truenetworks:- net1

# We're using custom network setup, as it ships with a DNS# system which allows containers to communicate by hostnames.networks:net1:driver: bridge

Now let’s boot this setup and take a look at how we can facilitate communication between these containers:

1. Bring up every container described in the file: docker-compose up -d
2. Connect to one of the hosts: docker exec -it host2.com erl -name [email protected] -setcookie cookie -remsh [email protected]
3. Verify that host connectivity is functioning correctly:

([email protected])1> net_adm:ping('[email protected]').pong([email protected])2> net_adm:ping('[email protected]').pong([email protected])4> net_adm:ping('[email protected]').pong([email protected])5> nodes().['[email protected]','[email protected]','[email protected]']

In this simple example we’ve brought up several containers and connected them using the standard erlang distribution. This is possible because all of the containers are running in the same docker network, and can access one anothers epmd daemons…

But how can we connect docker containers running on different hosts?

Connecting beams running on separate machines

Finally if you want to go to production, and to create a real cloud, running multiple hosting machines, the setup above will not work, as containers will not know how to connect to each other.

Indeed, hosts will not be aware about each others docker networks (in reality some work to interconnect docker containers using docker swarm or amazon ec2 containers service is being done, but we could not bring it up reliably running), and we would have to organise a more complex connection mechanism.

Actually the idea of creating alternative erlang distribution protocol came out of this amazing article, Erlang (and Elixir) distribution without epmd, we just decided to make few steps forward to create a pluggable application, which would just “Find them all, chain them all and… make sure all of them are always connected.”

The epmdless approach

As it’s described by Magnus Henoch, in his blog post, the epmd is not really needed in the dockerised setup, as docker environment is already forcing to expose particular ports for external access. And because of that there could be only one port used by erlang distribution (as other ports just will not be exposed).

On the other hand it’s quite easy to store mapping between hosts and ports inside the Erlang application. And now we already have the application for that: EpmdLess (credits go to Dmitry Mazurin for building it’s initial version).

Lets now see how it works:

• Clone out Erlang Distribution in Docker SandBox Applucation: git clone [https://github.com/oltarasenko/erlang_distribution_in_docker.git](https://github.com/oltarasenko/erlang_distribution_in_docker.git)
• Checkout to epmdless_example branch: git checkout epmdless_example
• Build docker image for the application: cd erlang_distribution_in_docker && docker-compose build
• Bring up the containers described in docker-compose.yml: docker-compose up
• Let’s check that epmd is not really running on the nodes:

docker-compose exec app1 lsof -i -n -PCOMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEbeam.smp 1 root 13u IPv4 251627 0t0 TCP *:17012 (LISTEN)

• Login to one of the containers: docker-compose exec app3 _build/default/rel/sample_app/bin/sample_app console

• Connect to another node using epmdless:

  epmdless_dist:add_node(‘[email protected]’, 17012). % Add node into epmdless database> net_adm:ping(‘[email protected]’). % Do ping> nodes(). % Node is discovered

Here is the short video showing steps above executed:

Watch: EPMDLess NewExample

Great, as you can see we’re able to connect nodes this way. Hovewer it looks like it would be quite hard to discover nodes this way, as we would need to initiate epmdless_dist:add_node and net_adm:ping calls for every node. That’s why we also want to suggest a convenient way (at least we hope you will like it) which allows you to also enable automatic node discovery for docker platform…

Nodes discovery on a distributed cluster

In order to be able to organise node discovery process we’ve released the Erlang Node Discovery application which automatically establishes connections between different nodes, as it’s specified in the application configuration file.

Besides this, it provides a simple API that allows you to extend the node discovery mechanism and to consume data from any other node/port database (local file, redis, database, etc).

{ erlang_node_discovery, [% epmdless_dist as our database{db_callback, epmdless_dist},{hosts, ["host1.com", "host2.com", "host3.com"]},% query following appnames/ports{node_ports, [{'app1', 17012},{'app2', 17013},{'app3', 17014}]}]}

This application will now spawn a process per possible pair of {appname@hostname, port} and will try to organise a fully connected cluster (and will do the reconnection).

In our case (as defined in configuration file above) it will spawn processes for the following possible nodes:

[email protected]:17012 [email protected]:17013 [email protected]:17014 [email protected]:17012 [email protected]:17013 [email protected]:17014 [email protected]:17012 [email protected]:17013 [email protected]:17014

Let’s give it a try:

1. Clone out Erlang Distribution in Docker SandBox Applucation: git clone [https://github.com/oltarasenko/erlang_distribution_in_docker.git](https://github.com/oltarasenko/erlang_distribution_in_docker.git)
2. Checkout to master branch (needed in case you’ve already cloned everything on previous step): git checkout master
3. Build docker image for the application: cd erlang_distribution_in_docker && docker-compose build
4. Bring up the containers described in docker-compose.yml: docker-compose up

And again, here is a video summarising the steps above:

Watch: ErlangNodeDiscovery

Ok, but what about Elixir?

Actually there is no problem re-using the same erlang libs to run a connected elixir cluster. The only trick is to show elixir where to find erlang libs :). This time I’ve added a Makefile to demonstrate it.

start:@ERL_LIBS=_build/dev/lib/ iex --name ${NODE_NAME} \--erl "+K true" \--erl "-config config/sys.config" \--erl "-proto_dist epmdless_proto" \--erl "-start_epmd false" \--erl "-epmd_module epmdless_client" \-S mix

Let’s now review the Elixir example:

1. git clone https://github.com/oltarasenko/epmdless.git
2. docker-compose build
3. docker-compose up

That’s all. Here is the short video demonstration!

Watch: ElixirEpmdLess

Originally published at www.erlang-solutions.com.