In this article I will try to explain the best practises to follow in the implementation of docker containers running in cluster mode (swarm, kubernetes or something different) or not.

I’m very passionate of the docker world because there are a lot of systems and network aspects very interesting to explore, manage and deep inside like namespaces, ipvs, vxlan, device mapper storage driver, natting, etc.. Old linux concepts assembled together for providing all the docker features.

I will try to share my experience in the docker container management matured in this last two years with the goal to trace some rule to follow in the architecture design of a based solution docker in a production environment.

Is it right to dockerize a database or a queue server? Where do I have put my static configuration? What should be the characteristics of a container? These are the questions I will try to answer.

Le’t start to speak about the features that docker containers must have, and then, starting from these principles, I will try to provide how a docker solution can be designed.

Docker Container design

Docker software implementation provides a lot of benefits in term of faster and easier deployment giving a useful answer to the demands of an increasingly demanding business. But not everything must be to docker if doesn’t bring practical advantages.

In this context, the big question that every developer or system administrator have to ask is if it’s correct or not to docker its software. The benefits to docker some software must be clear and obvious respect the old scenarios that continue to always be valid.

For answering to this question, thanks to the experience gained so far, I will list a serie of features that a container must have. My analysis considers the possibility to run the software also in a cluster environment orchestrated by docker swarm or kubernetes. This scenario must be considered because the cluster is the best solution for running container for having an environment high available, scalable and robust. If you don’t use cluster now, most likely you will use it next.

I will describe the features that every container must have. If one of this cannot be respected, you should really value the possibility to use the old approach.

Following the seven quality that every container should have:

1. Light. Every container should be as small as possible. This simplifies portability from one environment to another. It’s good approach to start in the Dockerfile from small distribution like alpine in Linux or user tecnologies like spring boot for building the containers. It’s important to have one only process per container running with pid equal to 1. In this context, it’s possible to manage the process sending to it signal like HUP for configuration reload.  If the service must fork more than one process, like apache, the container’s main process must be responsible for managing all processes and proxying all the signal received to managed process. The haproxy script start (https://github.com/docker-library/haproxy/blob/7837715e428efe0943053aae0130c709d017fd81/1.8/docker-entrypoint.sh) is one optimum example to see how to start a process: the exec form permits to have the main docker process with pid equal to 1. Other solutions for overcoming this problem, use process supervisor like dumb-init (https://github.com/Yelp/dumb-init).
2. Resilient. It means robust. If some requirement is not satisfied, like a queue service not reachable or database service down, the container should be able to manage these exceptions and returning to work when the requirements becomes respected. If this is not possible, it’s better to exit with error. In this case the orchestration software like swarm or kubernetes will retry to restart it. A container that is not working and doesn’t exit with error is a bad solution: it would be a misleading information.
3. Stateless. It means that doesn’t matter if a container goes down. One of great features of docker in cluster environment is the possibility to scale up and down the containers. This process is manual, but it can be easily automatized or well managed by orchestration container software like swarm or kubernetes. This is only possible if the container is stateless, otherwise the scaling up could not have any immediatle benefits, and the scaling down would cause outages.
4. Replicable. The big benefit of container orchestrator is the capacity to replicate a service creating copy of a container to run into other nodes of the cluster. This is true in swarm and kubernetes. The replicas of pods (kubernetes) or containers (called task in swarm) are balanced automatically by internal load balancing. It creates a lot of problems to containers that manage data like database, queue service, in memory key value store, etc. What should the data happen? Should the data be recreated in new volumes? If yes, how should this be done? In other side, every data server has its best practises for high availability: think to replication of sql db, the replica set of mongo, the clustering of redis and memcached. How will these aspects be related to replications? Very difficult to answer to all these questions even if some approach like ReplicaSet in Kubernetes is born for this reason. My advice is not to use docker for every software that stores data. This at least in Production where the databases should be configured in high availability following the best practises of its owners.
5. Fast: Every container should run very fastly for permit to make the most of the benefits of scaling and rolling update in a cluster environment like kubernetes or swarm.
6. Monitorable. Every software, dockerized or not, must be monitorable, but in a docker container scenario is even more true. A cluster composed by hundreds of containers can become hard to monitor if every container doesn’t expose the right metric in order to monitor its service level. This can be implemented, for example, by a ELK stack (logstash, kibana, elasticsearch) or using Prometeus/Graphana for collecting and visualizing the metric exposed.
7. Secure. Group in isolated environments all the containers that provide a particular service or logically similar by advanced concepts like stack, in docker swarm, or namespace in kubernetes. In the yml configuration file, don’t put any password, but for that it’s possible to use the swarm or kubernetes secret. Expose the port to external only if necessary, and use a modern reverse proxy and load balancer in front to front end containers because this permits a centralized way for all the security aspects like ssl configuration, access uri configuration, etc. I would also use these front end reverse proxy for caching, hosting the static sites and for load balancers. A external load balancing is necessary because the native load balancing provided by docker orchestrator (swarm or kubernetes) are not fully functional.

Inspired by this basic principles, following a picture that shows a possible solution architecture:

Docker Architectural Infrastructure

Conclusion

These tips are the result of several years of work in the management of microservices in environments managed by orchestrators such as swarm or kubernetes from point of view of the operation.

I discussed about how to developer and delivery containers in high critical environment  leaving the aspects related to way to integrate them for synchronous and asynchronous services in a next article.

Don’t hesitate to contact me for any question.