In this article I will try to explain the best practises to follow in the implementation of docker containers running in cluster mode (swarm, kubernetes or something different) or not.
I’m very passionate of the docker world because there are a lot of systems and network aspects very interesting to explore, manage and deep inside like namespaces, ipvs, vxlan, device mapper storage driver, natting, etc. Old linux concepts assembled together for providing all the docker features.
I will try to share my experience in the docker container management matured in this last two years with the goal to trace some rule to follow in the architecture design of a based solution docker in a production environment.
Is it right to dockerize a database or a queue server? Where do I have put my static configuration? What should be the characteristics of a container? These are the questions I will try to answer.
Le’t start to speak about the features that docker containers must have, and then, starting from these principles, I will try to provide how a docker solution can be designed.
Docker Container design
Docker software implementation provides a lot of benefits in term of faster and easier deployment giving a useful answer to the demands of an increasingly demanding business. But not everything must be to docker if doesn’t bring practical advantages.
In this context, the big question that every developer or system administrator have to ask is if it’s correct or not to docker its software. The benefits to docker some software must be clear and obvious respect the old scenarios that continue to always be valid.
For answering to this question, thanks to the experience gained so far, I will list a serie of features that a container must have. My analysis considers the possibility to run the software also in a cluster environment orchestrated by docker swarm or kubernetes. This scenario must be considered because the cluster is the best solution for running container in an environment high available, scalable and robust. If you don’t use cluster now, most likely you will use it next.
I will describe the features that every container must have. If one of this cannot be respected, you should really value the possibility to use the old approach.
Following the seven quality that every container should have:
- Light. Every container should be as small as possible. This simplifies portability from one environment to another. It’s good approach to start in the Dockerfile from small distribution like alpine in Linux or user tecnologies like spring boot for building the containers. It’s important to have one only process per container running with pid equal to 1. In this context, it’s possible to manage the process sending to it signal like HUP for configuration reload. If the service must fork more than one process, like apache, the container’s main process must be responsible for managing all processes and proxying all the signal received to managed process. The haproxy script start (https://github.com/docker-library/haproxy/blob/7837715e428efe0943053aae0130c709d017fd81/1.8/docker-entrypoint.sh) is one optimum example to see how to start a process: the exec form permits to have the main docker process with pid equal to 1. Other solutions for overcoming this problem, use process supervisor like dumb-init (https://github.com/Yelp/dumb-init).
- Resilient. Every container cannot stop if some other dependent container is not still started. The container should be able to wait other container dependencies without stopping. It’s true that a container orchestrator like swarm of kubernetes permits to schedule the container re-running, but in standalone environment this is not possible and then it’s good practise to add this resiliency inside the container. Infact the “depends_on” docker parameter waits that the container is running, not ready (https://docs.docker.com/compose/startup-order/).
- Stateless. One of great features of docker in cluster environment is the possibility to scale up and down the containers of a service in function of service load. This process is manual, but it can be easily automatized. All that is possible if the container is stateless, otherwise the scaling up could not have any immediatle benefits, and the scaling down would cause sicurely outages. Design stateful container is simply be beautiful without practical utility. The biggest mistake not to commit.
- Replicable. The big benefit of container orchestrator is the capacity to replicate a service creating copy of a container to run into other nodes of the cluster. This is true in swarm and kubernetes. The replicates of pods (kubernetes) or containers (called task in swarm) are balanced automatically by internal load balancing. It creates a lot of problems to containers that manage data like database, queue service, in memory key value store. What should the data happen? Should the data be recreated in new volumes? If yes, how should this be done? In other side, every data server has its best practises for high availability: think to replication of sql db, the replica set of mongo, cluster of redis and memcached. How will these aspects be related to replications? Very difficult to answer to all these questions even if some approach like ReplicaSet in Kubernetes is born for this reason. My advice is not to use docker for every software that stores data. This at least in Production where the databases should be configured in high availability following the best practises of its owners. This is still true in the cloud where the data store services are offered as managed service high available, robust and scalable ready to use from containers and not.
- Fast: Every container should run very fastly for permit to make the most of the benefits of scaling and rolling update in a cluster environment like kubernetes or swarm. Scaling a service composed by container very fast to run means no to have any outage during this process. The same concept can be applied during the rolling update process implemented by kuberentes or swarm. The big issue of ths orchestrators is not have any check live strument in order to decide if a container is ready to go live. This justifies even more the reason to design container fast, very fast to be ready.
- Monitorable. Every software, dockerized or not, must be monitorable, but in a docker container scenario is even more true. A cluster composed by hundreds of containers can become hard to monitor if every container doesn’t expose the right metric in order to monitor its service level. This can be implemented, for example, by a ELK stack (logstash, kibana, elasticsearch) or using Prometeus/Graphana for collecting and visualizing the metric exposed.
- Secure. Group in isolated environments all the containers that provide a particular service or logically similar by advanced concepts like stack, in docker swarm, or namespace in kubernertes. In the yml configuration file, don’t put any password, but for that it’s possible to use the swarm or kubernetes secret. Expose the port to external only if necessary, and use a modern reverse proxy and load balancer in front to front end containers because this permit a centralized way for all the security aspects like ssl configuration, access uri configuration, etc. I would also use these front end reverse proxy for caching, hosting the static sites and for load balancers. A external load balancing is necessary because the native load balancing provided by docker orchestrator (swarm or kubernetes) are non fully functional.
Inspiring to this basic principles, following a picture that shows a possible architectural solution:
Work in Progress