Using Docker Health Checks to Wait for Development Environment Services

At PMG we use Docker and Docker Compose to spin up backing services for local development and continuous integration — think things like databases, cache, or localstack.

One challenge, especially in a CI environment, is making sure that the services are up and available before starting test runs.

Enter HEALTHCHECK

Dockerfiles can contain a HEALTHCHECK stanza, and similarly docker run also takes a --health-cmd flag. And, of course, docker compose also allows healthcheck to be defined per service.

The HEALTHCHECK or --health-cmd runs inside the container to see if, as the name implies, the container is up and healthy. This status becomes available via docker inspect.

Healthcheck in Docker Compose

Let’s use PostgreSQL as an example here. There is a pg_isready command that can check if postgres up and accepting connections.

services:
  postgres:
    image: postgres:16
    environment:
      - POSTGRES_USER=app
      - POSTGRES_PASSWORD=password
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready"]
      retries: 1

Now we can combine a little docker inspect with docker compose ps -q {servicename} to pull the status:

docker inspect --format "{{.State.Health.Status}}" $(docker compose ps -q postgres)

And throw that into a loop to wait a bit and keep checking, this is best combined into a wait_for_service script that can be re-used for any service in the compose file:

#!/usr/bin/env bash

set -e

container="$1"
if [ -z "$container" ]; then
    echo "usage: $0 service"
    exit 1
fi

count=0
alive="no"

while [ "$count" -lt 6 ]; do
    health=$(docker inspect --format "{{.State.Health.Status}}" "$(docker-compose ps -q $container)")

    if [ "$health" == "healthy" ]; then
        alive="yes"
        break
    fi

    echo "waiting for $container: $health"
    sleep 10
    count=$((count+1))
done

if [ "$alive" = "yes" ]; then
    exit 0
else
    echo "$container did not start up in time"
    exit 1
fi

Now waiting for postgres to be up in a CI environment is a ./wait_for_service postgres away.

Another Healthcheck Example: Localstack

Not everything comes with a handy pg_isready equivalent. Localstack, for instance, spins up very quickly, but can take a bit to run it’s initialization hooks. For that use case, I will often have a 999-ready.sh script in my localstack ready.d that runs last and creates a /tmp/localstack_ready file:

#!/usr/bin/env bash

echo 'yes' > /tmp/localstack_ready

The healthcheck command can check for this file and mark the container as healthy:

services:
  localstack:
    image: localstack/localstack:3
    ports:
      - "4566:4566"
    volumes:
      - ./docker/localstack/ready.d:/etc/localstack/init/ready.d
    healthcheck:
      test: ["CMD", "test", "-f", "/var/lib/localstack/localstack_ready"]
      retries: 1

And again ./wait_for_service localstack is all it takes to make sure the backing services and resources are up and ready for the engineer or a CI server.