Docker

From Wildsong
Jump to navigation Jump to search

General information

Docker is a container platform. They leverage more of the host's operating system so they are lightweight compared to virtual machines, and far easier to administer.

Docker lets you build a container to run a service.

Docker Compose lets you combine several containers to build up more complex services, for example a database and a web front end.

Docker Swarm lets you deploy containers across multiple servers and keeps them running.

Data for the service can be baked into the container, in a Docker volume, or in a filesystem shared with the host. Docker containers can share Docker volumes.

Networks for containers can be isolated or shared. By default, all the containers in a Docker Compose setup share their own isolated network.

Docker Hub is an archive of thousands of ready-to-go images so that you can spin up a service with nearly no effort.

Go to the docker.com site and look for the Community edition. For Debian, try https://www.docker.com/docker-debian

Docker for Windows

I think this is worth a separate page: Docker for Windows also known as Docker I can use at work

I am trying out Podman on Windows now.

Docker Compose

I've made it far enough with Docker to now see the advantages of Compose. I can use one or several containers together with compose and the command line goes from multiple commands like

docker run -d --name=geoserver -e ENVSETTING1=secret -e ENVSETTING2=anothersecret -p "8888:80" --restart=unless-stopped -v geoserver_files:/geoserver geoceg/geoserver

to

docker-compose up

Also Compose takes care of 99% of networking issues for me. It was working swimmingly until I decided to add more flexible configuration options. I started thinking back on the Vastra Ansible project and how sweet that was, for configuring actual physical servers. So now I turn to ansible-container to try generating Dockerfiles from templates.

Ansible-Container

My current need is to push configurations down into an nginx container so that it can act as a proxy for geoserver's components. I want to bake the configuration into the container; my previous solution was to put the template engine (just 'sed' in this case) into a script that runs when the container starts to generate the nginx config.

I am currently running through the intro in this book (via Safari): Containerization with Ansible 2 from Packt. I had to install both vagrant and ansible on Bellman to get things rolling, this surprised me, I must have done all my Ansible development work for Vastra on other computers.

Ansible Container Cheatsheet

Docker Swarm

Having mastered (more or less) Docker Compose I now want to learn about orchestration with Docker Swarm. Kubernetes is breathing down my neck too but later for that.

I was going to use Dart as a worker but I decided I did not want to pay the electric bill to spin all those drives up anymore, so I sold Dart and now it lives in Newberg. I am going to use Tern instead. Should be fine for testing.

bellman> docker swarm init
Swarm initialized: current node (isk0jocx0rb37yonoafstyvoj) is now a manager.

To add a worker to this swarm, run the following command:

   docker swarm join --token SWMTKN-1-5b81dywl9xkis6769fxnsvjahfy361w2kxkz69nc35bz3nxt6s-43jxeopl6inw8xur1vpcl23w7 192.168.123.2:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
tern> docker swarm join --token SWMTKN-1-5b81dywl9xkis6769fxnsvjahfy361w2kxkz69nc35bz3nxt6s-43jxeopl6inw8xur1vpcl23w7 192.168.123.2:2377
This node joined a swarm as a worker.
bellman> docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE  VERSION
isk0jocx0rb37yonoafstyvoj *   bellman             Ready               Active              Leader              19.03.5
vjbx2h8n8280ecib2btzkwcxw     tern                Ready               Active                                  18.09.1
bellman> docker network create -d overlay --attachable proxy_net

The "attachable" option is for containers not yet running in swarm.

Now ordinarily I'd use Docker Compose to start a proxy running consisting of two components. One is the nginx reverse proxy and one is the letsencrypt docker. Here are the rules to start them swarm style using Docker Stack.

How about before trying all that I spin up simple web server? This works, and the proxy works normally too. It creates 4 replicas.

docker service create --name web --replicas=4 \
   -p 80:80 -e NETWORK_ACCESS=internal \
   -e VIRTUAL_HOST=solr.wildsong.biz -e VIRTUAL_PORT=80 \
   -e LETSENCRYPT_HOST=solr.wildsong.biz -e LETSENCRYPT_MAIL=brian@wildsong.biz \
   nginx:latest

This did not work, it's just something I tried. Possibly it's the volume settings? How do volumes work in a swarm?

docker service create --name proxy \
 -p 80:80 -p 443:443 \
 -e DHPARAM_GENERATION="false" \
 -v /var/run/docker.sock:/tmp/docker.sock:ro \
 -v ./network_internal.conf:/etc/nginx/network_internal.conf \
 -v ./vhost.d:/etc/nginx/vhost.d \
 -v proxy_html:/usr/share/nginx/html \
 -v proxy_dhparam:/etc/nginx/dhparam \
 -v proxy_certs:/etc/nginx/certs:ro \
 jwilder/nginx-proxy:alpine

Docker monitoring with Prometheus

See the change on Bellman's Docker Engine below.

Get my IP address (DIAMOL is wrong)

hostIP=$(ip route get 1 | awk '{print $7}')

NOTE I had to block port 9090 on Squeezebox to allow this to run here.

docker container run -e DOCKER_HOST=$(ip route get 1 | awk '{print $7}') -d -p 9090:9090 diamol/prometheus:2.13.1

Docker on Bellman

2020-08-20 Adding monitoring, /etc/docker/daemon.json looks like this now. Try it, http://bellman:9323/metrics

{
   "data-root": "/var/lib/docker",
   "storage-driver": "overlay2",
   "dns" : ["127.0.0.11", "1.0.0.1", "1.1.1.1"],
   "dns-search":["wildsong.biz"],
   "metrics-addr" : "0.0.0.0:9323",
   "experimental" : true
}

2019-10-03 Upgrading to Debian Buster broke two things: there's a complaint about a missing module, and networking.

The first issue was not horrendous, I had to switch back to the apt packages which also meant dropping back a version in docker-compose.yml files. This is not so bad since it makes them more compatible out in the github world. The commands I used to switch were:

sudo pip uninstall docker docker-compose
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install docker docker-compose

The network issue required installing a legacy package

Some of the services on Bellman

A year or so ago I migrated services into Dockers on Bellman, including GeoServer, a web proxy, UniFi, mariadb, and the Logitech media server (Squeezebox).

I used to run Owncloud in Docker Compose but I'm using a Synology Diskstation named Wenda now.


Docker as a service

Don't screw around with making containers behave in systemd. It's a waste of time. Use the "restart" option.

After getting a container running the way you want, set it to restart after reboots with the "restart" option. You can issue this option as part of the "run" command but if you are testing the container and the config is broken the container will endlessly crash and restart. Test first, then use update like this:

docker update --restart unless-stopped 'CONTAINER'

Don't use "--restart always" because if you do then you have to change the restart setting before stopping it. With "always" issuing 'stop' alone is not enough, if you do that the docker daemon detects the container stopped and just restarts it. This is the workaround for that situation:

docker update --restart no 'CONTAINER'
docker stop 'CONTAINER'

(or you could say "unless-stopped" instead of "no"....)

Docker ArcGIS Enterprise

I worked on this Docker project putting ArcGIS Enterprise into containers. It sort of works. Licensing is a pain. The biggest pain is that it wants to start fresh each time I restart the containers; something is not being persisted. I created volumes but I am guessing some file tucked away is inside the container and gets wiped on restarts.

When my developer license expired, I stopped all work and moved over to open source.

If you want to experiment the project is preserved in Github, you can find my work in this repository: docker-arcgis-enterprise

Development on Mac

Install Docker for Mac (unless your Mac is too old, boo hoo like Stellar) or Kitematic. Kitematic installs VirtualBox. Docker for Mac is lighter weight. If you already use Vagrant (for MySQL for example) then you already need VirtualBox so Kitematic is fine.

Asking to download Kitematic takes me to the Docker Toolbox download page. The Docker Toolbox includes Kitematic. Kitematic installs VirtualBox if you don't already have it.

After installing the prerequisites, the Docker Toolbox installer took me to the Docker warehouse, where I selected the "official" Solr container and started it, all in one go, without actually needing to know anything at all about Docker or Virtualbox. I now have a running Solr instance.

Deployment on Linux

For Debian make sure you are using docker.com docker-ce package not docker-engine

https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-storage-driver-option

I am using btrfs as a compromise for speed and stability

My /etc/docker/daemon.json file on laysan contains this, it works on Ubuntu but not on Debian Jessie

{                                                                               
   "storage-driver":"btrfs",                                                                
   "graph":"/volumes/ssd_machines/docker",                                     
                                                                               
   "dns":["192.168.123.2"],                                                    
   "dns-search":["arcgis.net"]                                                 
}

See https://docs.docker.com/v1.11/engine/reference/commandline/daemon/#daemon-configuration-file

Just a quick test

You should now be able to download and run a pre-built image from the Docker hub, get a shell prompt, and quit. After you exit from the shell, you can see that the container was running with the "docker ps -a" command.

# Download the Ubuntu Server LTS (Long Term Support) release
docker pull ubuntu:16.04
# Run it interactively with a shell prompt, so we can look around
docker run -it ubuntu /bin/bash
root@a6f99cc58685:/# 
docker ps -a
CONTAINER ID    IMAGE    COMMAND       CREATED           STATUS                      PORTS  NAMES
a6f99cc58685    ubuntu   "/bin/bash"   21 seconds ago    Exited (0) 10 seconds ago          furious_kowalevski

Samba in Docker

https://hub.docker.com/r/dperson/samba/

docker run -it --name samba -p 139:139 -p 445:445 -v /home/bwilson:/bwilson -d dperson/samba

Docker Hub

https://0-proquest.safaribooksonline.com.marinet.lib.ca.us/book/operating-systems-and-server-administration/virtualization/9781789616606/docker-hub/b39d70eb_75a7_4d25_9b21_21474b7fbf01_xhtml

Use case: Version 2.16.0 of geoserver came out. I need to update.

  1. Edit the Dockerfile in the Wildsong/docker-geoserver repository to the new version
  2. git commit -m 'updated version' -a
  3. git tag 2.16.0
  4. git push

In the Docker Hub Builds page for the project, click Configure Autonmated Builds create a new configuration and attach it to the new tag.

docker push wildsong/geoserver:2.16.0

Shortcuts

See all images available on this machine:

docker images

See all running containers and get an id:

docker ps

See all containers, running or not:

docker ps -a

Look at a container in detail

docker inspect id

See also /var/lib/docker/containers/id

Docker networking

Check ifconfig docker0 to see what the bridge address is on your docker server then as needed create a route on your router so that traffic can flow from your local computers into the docker instances running on the docker server.

If you have more than one docker server you have to make sure they don't both pick the same network ip range. If routes are already set up in the router docker should be able to find an unused subnet.

docker pull nginx

cd static-html-directory cat <<EOF > nginx.docker FROM nginx COPY static-html-directory /usr/share/nginx/html EOF

Place this file in the same directory as your directory of content ("static-html-directory"), run ., then start your container:

docker run --name some-nginx -d some-content-nginx

Docker on Windows

My big problem is the path for a volume has a space in it and it blows Docker Compose out of the water.

Okay, actually I just saw this comment and I am wondering if the space in the path is really my biggest problem or just an excuse to not use Windows. My biggest problem is that my Windows machine is so puny that I am afraid to even try to run Docker on it. I've been remoting into Bellman. Bellman is my favorite puny computer right now. It seems 8 slow processor cores + 32GB RAM + fast SSD > 4 slow cores.

Anyway - regards the space in

/c/Program Files/

Docker sees two arguments, with the space as a delimiter

I am trying to find out where it's happening in Compose.

I tried all the variations I could think of to quote it.

compose.parallel.feed_queue: Pending: set([])
compose.parallel.parallel_execute_iter: Finished processing: <Service: datastore>
compose.parallel.feed_queue: Pending: set([])
ERROR: for server  Cannot create container for service server: Invalid bind mount spec "39aab1cb131c79756827d7d4c41fb05dde57bd451011592106e0a0db0aeeb708:Files/ESRI/License10.5/sysgen]:rw": Invalid volume destination path: 'Files/ESRI/License10.5/sysgen]' mount path must be absolute.
ERROR: compose.cli.main.main: Encountered errors while bringing up the project.

Resources

I am reading "Docker Deployment" on Safari and "Learn Docker in a Month of Lunches" by Elton Stoneman.