Add a new Cloud Worker Pool

Introduction

To support spikes in work requests, debusine needs to be able to dynamically make use of compute resources in clouds. Static workers may still be useful for various reasons, such as:

locality
confidentiality
supporting an expected base load
customers who are sensitive to making use of artifacts built in clouds they don’t control
support for exotic architectures
in some cases, the ability to launch virtual machines in the worker

However, dynamic workers offer more flexibility and may be cheaper overall.

Initial Configuration

Select a supported cloud provider (currently Amazon EC2 or Hetzner Cloud). This howto will use Hetzner Cloud, but the process is very similar for EC2.
Sign up for an account with the provider and create an API key.
Ensure that your Debusine server is visible to cloud instances within the cloud provider at its configured DEBUSINE_FQDN URL. If it’s a private instance, not on the Internet, then you may need to run it within the cloud, or provide a tunnel from your private network in the cloud to the Debusine server.

Create a YAML file (hetzner-account.yaml) describing a debusine:cloud-provider-account asset:

provider_type: hetzner
name: hetzner-acct
configuration:
  region_name: eu-central
credentials:
  api_token: LMg2iuGg47q6jzZ9aCE2ZchzOnRSeBQum5zAvhwaLi6GAKakJP

Create an asset containing this account data:

$ debusine-admin asset create debusine:cloud-provider-account \
  --data hetzner-account.yaml

Create a YAML file (specifications.yaml) describing the worker pool. See Dynamic Worker Pools for all the configuration details.

provider_type: hetzner
server_type: cx22
image_name: debian-12
ssh_keys:
  - my-ssh-key
labels:
  role: debusine-worker
debian_release: bookworm
debusine_install_source: backports

Create a worker pool named hetzner-cloud:

$ debusine-admin worker_pool create --provider-account hetzner-acct \
  --architectures amd64 --specifications specifications.yaml --instance-wide \
  hetzner-cloud

Launch an instance manually to verify settings:

$ debusine-admin worker_pool launch hetzner-cloud 1

A new worker hetzner-cloud-001 should appear in the workers list, and after a couple of minutes should connect and show as “Idle”.
If necessary, find the instance IP address in the Hetzner Cloud Console, and ssh into the instance to debug its startup. Instances are provisioned using cloud-init.