Docker Compose Quick Start

Posted on | Edited on | In

Docker Compose vs Docker Swarm The difference lies mostly in the backend, where docker-compose deploys container on a single Docker host, Docker Swarm deploys it across multiple nodes.

现在想想，当时组员在本地搭建DataStage也应该用docker compose, it is especially good for web development:

  • accelerate onboarding
  • eliminate app conflicts
  • environment consistency
  • ship software faster

Install

https://docs.docker.com/compose/install/ For Mac docker-compose is self-contained with Docker desktop:

1
2
3
4
## check location
which docker-compose
## show version
docker-compose version

For Linux, first install Docker Engine, then download docker-compose to executable path:

1
2
3
## 1.26.2 is current stable version
sudo curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

or download binary from github release directly: https://github.com/docker/compose/releases

Uninstall docker-compose, remove binary:

1
sudo rm -f /usr/local/bin/docker-compose

Getting Started

https://docs.docker.com/compose/gettingstarted/ If you know how to write yaml file for Kubernetes, then quick easy to understand the docker-compose.yml.

Some basic commands:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
## build or rebuild service images
## if not specify service name, will build all images
docker-compose build [--no-cache] [service name]

## run services
## will build images if not yet
## -d: detch
## --no-deps: just bring up specified service
docker-compose up [-d | --no-deps] [service name]

## check logs
docker-compose logs [-f] <service name>

## see what is running
docker-compose ps

## one-off command in container
docker-compose run <service name> <commands>

## start/stop service, will not remove container
docker-compose start
docker-compose stop

## remove stopped containers
docker-compose rm [service name]

## shutdown, similar to docker rm -f ...
## -v,--volumes: remove the mounted data volume
## --rmi all: remove all images
docker-compose down [-v] [--rmi all]

Command Completion

https://docs.docker.com/compose/completion/#install-command-completion for oh-my-zsh, add docker and docker-compose to plugins list in ~/.zshrc:

1
plugins=(... docker docker-compose)

Config File

遇到没见过的指令，查阅这里, see left sidebar version 3.

Define services relationship in docker-compose.yml file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
## docker-compose.yml can parse env variables in current running environment
## 通过这些环境变量控制读取的文件类型,比如development, staging, production
## export APP_ENV=development

## 这里假设下面的文件夹和文件都存在，比如dockerfile 等等

## docker-compuse version
version: "3.7"
services:
  web:
    container_name: web
    build:
      ## relative path is base on docker-compose.yml directory
      ## can be url
      context: .
      ## args used in dockerfile, must declaure using ARG in dockerfile
      ## only seen in build process!
      args:
      - buildno=1
      ## relative path to context
      dockerfile: ./web/web.dockerfile
    ## specify built image and tag
    ## if no image here, docker-compose will use it's name convension
    image: webapp:v1
    
    ## dependencies
    ## docker-compose will start mongo and redis first
    depends_on:
    - mongo
    - redis

    ## host: container
    ## 外界访问采用host port
    ## container 之间互相访问port 不用在这里expose
    ports:
    - "80:80"
    - "443:443"
    - "9090-9100:8080-8100"
    
    ## add env vars from a file
    env_file:
    - web.env/app.${APP_ENV}.env
    - /opt/runtime_opts.env
    
    ## other form of env vars
    environment:
      RACK_ENV: development
      ## boolean muct be quoted
      SHOW: 'true'
      ## value is from current running environment
      SESSION_SECRET:

    ## short syntax
    ## https://docs.docker.com/compose/compose-file/#short-syntax-3
    volumes:
    ## host path, relative to docker-compose.yml
    - "./web:/opt/web"
    ## named volume
    - "mydata:/opt/data"

    ## overwrite WORKDIR in Dockerfile
    working_dir: /opt/web

    ## containers in the same network are reachable by others
    networks:
    - nodeapp-network
  
  mongo:
    container_name: mongo
    build:
      context: .
      dockerfile: mongo/mongo-dockerfile
    ports:
    - "27017"
    env_file:
    - mongo.env/mongo.${APP_ENV}.env
    networks:
     nodeapp-network:
       ## can be accessed by `mongo` or `db` in nodeapp-network
       aliases:
         - db

  redis:
    container_name: redis
    ## pull image from other places
    image: redis:latest

    ## long syntax
    ## https://docs.docker.com/compose/compose-file/#long-syntax-3
    volumes:
    - type: volume
      source: dbdata
      target: /data
      volume:
        nocopy: true

    ports:
    - "6379"
    env_file:
    - redis.env/redis.${APP_ENV}.env
    networks:
    - nodeapp-network

networks:
 nodeapp-network:
   ## docker defaults to use bridge driver in single host
   driver: bridge

## named volumes
volumes:
  ## default use 'local' driver
  mydata:
  dbdata:

More about environement variables: By default, the docker-compose command will look for a file named .env in the directory you run the command. By passing the file as an argument, you can store it anywhere and name it appropriately, for example, .env.ci, .env.dev, .env.prod. Passing the file path is done using the --env-file option:

1
docker-compose --env-file ./config/.env.dev up

.env contains key=value format equaltions.

Storage

Image is set of read-only layers (shared), whereas container has its unique thin read write layer but it is ephemeral.

关于storage的讲解: https://docs.docker.com/storage/ 这里主要弄清楚volumes, bind mounts and tmpfs的区别和使用:

如果docker host的文件系统和docker container使用的不一样，bind mounts如何处理呢? 并且内外user, group都不一样，如果在docker container中新建一个文件，bind mounts 在host中如何映射呢?

Docker Compose Volume: https://docs.docker.com/compose/compose-file/#volumes 这里讲了如何mount host path or named volumes.

Volumes

https://docs.docker.com/storage/volumes/ 类似于K8s的volume claim, general preferred.

Stored in a part of the host filesystem which is managed by Docker (/var/lib/docker/volumes/ on Linux). Non-Docker processes should not modify this part of the filesystem. Volumes are the best way to persist data in Docker.

A given volume can be mounted into multiple containers simultaneously. When no running container is using a volume, the volume is still available to Docker and is not removed automatically.

1
2
3
4
5
6
docker volume create <volume name>
docker volume ls
docker volume rm <volume name>
docker volume inspect
## remove unused volumes
docker volume prune

When you mount a volume, it may be named or anonymous.

Volumes also support the use of volume drivers, which allow you to store your data on remote hosts or cloud providers, among other possibilities.

If you need to specify volume driver options, you must use --mount, -v 的表示比较局限, 这里只是一个简单的例子, 实际上用--mount的配置选项很多:

1
2
3
4
5
6
7
8
9
10
11
## docker will create volume myvol2 automatically if it does exist
docker run -d \
  --name devtest \
  -v myvol2:/app[:ro] \
  nginx:latest

## or
docker run -d \
  --name devtest \
  --mount source=myvol2,target=/app[,readonly] \
  nginx:latest

If the container has files or directories in the directory to be mounted (such as /app/ above), the directory’s contents are copied into the volume, other containers which use the volume also have access to the pre-populated content.

Bind Mounts

https://docs.docker.com/storage/bind-mounts/ 类似于K8s的hostpath, use case for example, mount source code for development.

May be stored anywhere on the host system. They may even be important system files or directories. Non-Docker processes on the Docker host or a Docker container can modify them at any time.

The file or directory does not need to exist on the Docker host already. It is created on demand if it does not yet exist.

Bind mounts are very performant, but they rely on the host machine’s filesystem having a specific directory structure available.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
docker run -d \
  -it \
  --name devtest \
  -v "$(pwd)"/target:/app[:ro] \
  nginx:latest
## or
docker run -d \
  -it \
  --name devtest \
  --mount type=bind,source="$(pwd)"/target,target=/app[,readonly] \
  nginx:latest

## check Mounts section
docker inspect devtest

If you bind-mount into a non-empty directory on the container, the directory’s existing contents are obscured by the bind mount.

tmpfs

https://docs.docker.com/storage/tmpfs/ Only Linux has this option, useful to temporarily store sensitive files. Stored in the host system’s memory only, and are never written to the host system’s filesystem.

1
2
3
4
5
6
7
8
9
10
11
docker run -d \
  -it \
  --name tmptest \
  --tmpfs /app \
  nginx:latest
## or
docker run -d \
  -it \
  --name tmptest \
  --mount type=tmpfs,destination=/app,tmpfs-mode=1770,tmpfs-size=1024 \
  nginx:latest

Network

这个章节讲到了所有docker network的类型，作用，区别: https://docs.docker.com/network/

Docker network labs: https://github.com/docker/labs/tree/master/networking

From docker’s perspective, Steps to create a container network:

  1. create a custom bridge network
1
2
3
4
5
6
7
8
9
## create a bridge network isolated_network
docker network create --driver bridge isolated_network

## see created network, the driver is bridge
## you can see host and none are there
docker network ls

## inspect 
docker network inspect isolated_network
  1. run containers in the network and ping each other by container name
1
2
3
4
5
6
7
8
9
10
## create 2 busybox in the same network
docker run -d --name=test1  --net=isolated_network --entrypoint=/bin/sh busybox -c "tail -f /dev/null"
docker run -d --name=test2  --net=isolated_network --entrypoint=/bin/sh busybox -c "tail -f /dev/null"

## you can see containers in this network
docker network inspect isolated_network

## ping each other
docker exec test1 ping -c 5 test2
docker exec test2 ping -c 5 test1
  1. remove network created
1
2
docker network rm isolated_network
docker network ls

同样的思路，可以用docker command查看docker-compose中建立的network的信息。

列出了docker compose中top-level networks 创建时的options, after creating top-level networks, they can be referenced by service-level to use: https://docs.docker.com/compose/compose-file/#network-configuration-reference

这篇文章说得很明白, docker compose中network是如何作为的: https://docs.docker.com/compose/networking/

这个例子很有意思，把frontend, backend的网络分开了, only app can reach both networks: https://docs.docker.com/compose/networking/#specify-custom-networks

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
version: "3"
services:

  proxy:
    build: ./proxy
    networks:
      - frontend
  app:
    build: ./app
    networks:
      - frontend
      - backend
  db:
    image: postgres
    networks:
      - backend

networks:
  frontend:
    # Use a custom driver
    driver: custom-driver-1
  backend:
    # Use a custom driver which takes special options
    driver: custom-driver-2
    driver_opts:
      foo: "1"
      bar: "2"

Resource

类似于K8s, 也有quota的配置在deploy key下面，但是docker compose file v3 并不支持: ignored by docker-compose up and docker-compose run，虽然可以转换成v2，比如:

1
docker-compose --compatibility up

但是是best effort , 见这里讨论: How to specify Memory & CPU limit in docker compose version 3

Docker Compose file version 2是支持quote设置的: https://docs.docker.com/compose/compose-file/compose-file-v2/#cpu-and-other-resources

0%