Getting started with debusine

In this tutorial you will discover two of debusine’s building blocks: work requests and artifacts. The former lets you schedule the execution of different kind of tasks on remote workers, while the latter is a simple way to represent a set of files managed by debusine.

For the purpose of this tutorial, we will use debusine’s command line interface to:

  • create a build environment

  • upload a source package

  • build binary packages out of the uploaded source package and in the build chroot that got created

Pre-requisites

You need to have access to a working debusine instance:

  • If you are a Debian developer, you can use debusine.debian.net. You can login there with your salsa.debian.org account, and it will automatically create your debusine account.

  • Otherwise, please follow the instructions from Install your first debusine instance. Once completed, the debusine instance will be available under the debusine.internal hostname and this is the name that we will continue to use here.

Install the client and configure it

You should first install the debusine-client package (if needed configure APT with one of the supported package repositories):

$ sudo apt install debusine-client

Now it’s time to create yourself a token that the client will use to connect to the server. For this open your web browser and visit the URL of your debusine instance (e.g. http://debusine.internal or https://debusine.debian.net). Then login with your user credentials (or with the salsa.debian.org authentication) and open the user menu (in the top right corner, or behind the “hamburger” menu) and click on the Tokens page.

Click the link create a new token and input a comment describing the token’s intended usage (for example “On my laptop”) and click the Create button. You will receive the token, which you should copy before exiting the page as the token will only be shown once.

Finally, we can create debusine-client’s configuration file. It indicates the server URL and the token to use (please adjust the DEBUSINE_URL to match your specific case and paste the value of the token in the TOKEN variable):

$ DEBUSINE_URL="http://debusine.internal" # or DEBUSINE_URL="https://debusine.debian.net"
$ TOKEN="1e81fb5f00a489344ddfe4e34588c84d202a97ca87cc136d15d17cb3d1eb4e82"
$ DEBUSINE_FQDN=$(basename $DEBUSINE_URL)
$ SCOPE=debusine
$ mkdir -p ~/.config/debusine/client
$ cat >~/.config/debusine/client/config.ini <<END
[General]
default-server = $DEBUSINE_FQDN

[server:$DEBUSINE_FQDN]
api-url = $DEBUSINE_URL/api
scope = $SCOPE
token = $TOKEN
END

More information about the debusine command line interface is available in The debusine command.

Create a build environment

If you’re using debusine.debian.net, you can skip this step, as suitable build environments have already been created for you.

Otherwise, start by creating a debian:environments collection on your server:

$ sudo -u debusine-server debusine-admin create_collection \
    debian debian:environments </dev/null

You’ll also need to have the necessary permissions to create workflow templates. We don’t yet have a good interface for granting this permission, so for now you’ll need to insert this into the database by hand:

$ sudo -u debusine-server debusine-admin shell
from django.contrib.auth import get_user_model
from django.contrib.auth.models import Permission
User = get_user_model()
user = User.objects.get(username="YOUR-USER-NAME")
permission = Permission.objects.get_by_natural_key(
    "add_workflowtemplate", "db", "workflowtemplate"
)
user.user_permissions.add(permission)

Then create an update_environments workflow template and start it running:

$ debusine create-workflow-template \
    update-debian-environments update_environments <<END
vendor: "debian"
targets:
- codenames: ["trixie"]
  architectures: ["amd64"]
  backends: ["unshare"]
  mmdebstrap_template:
    bootstrap_options:
      variant: "minbase"
    bootstrap_repositories:
    - mirror: "http://deb.debian.org/debian"
      components: ["main"]
- codenames: ["trixie"]
  architectures: ["amd64"]
  variants: ["sbuild"]
  backends: ["unshare"]
  mmdebstrap_template:
    bootstrap_options:
      variant: "buildd"
    bootstrap_repositories:
    - mirror: "http://deb.debian.org/debian"
      components: ["main"]
END

$ debusine create-workflow update-debian-environments <<END
{}
END

Once this workflow finishes (which will take a few minutes), you should have a debian:environments collection populated with some useful base tarballs for trixie/amd64 that can be used with the unshare backend: a default variant containing only essential and required packages, and an sbuild variant that also contains build-essential packages. These can be looked up by name. If you wish, you can vary the targets dictionary to build different environments, or automate this workflow to run regularly.

Create an artifact by uploading a source package

The low level debusine create-artifact command can be used to create any arbitrary artifact but when it comes to Debian source packages (.dsc) or Debian uploads (.changes), debusine offers a more convenient interface with debusine import-debian-artifact [FILE|URL]. You can refer to a local file or to a remote URL.

For instance, you can create and upload an artifact for the “hello” source package with:

$ debusine import-debian-artifact http://deb.debian.org/debian/pool/main/h/hello/hello_2.10-3.dsc
[...]
message: New artifact created in http://debusine.internal/api in workspace System with id 536.
artifact_id: 536

Artifacts can be provided as input to some debusine tasks. We will shortly use this artifact to schedule a package build.

Submit a work request to build binary packages

Submitting a work request is asking the debusine server to schedule some specific task on one of its available workers. Debusine can execute many different tasks.

Among the available tasks, the sbuild one can build binary packages out of a source package (provided through the parameter source_artifact below the input key) and a build chroot (provided through the parameter environment).

Each task requires a certain number of key-value parameters that are fed to debusine create-work-request TASKNAME as YAML data on the standard input. Try this, taking care to refer to the ID of the artifact that we created in the previous step (536 for the source package in this example):

$ debusine create-work-request sbuild << END
build_components:
- any
- all
host_architecture: amd64
input:
  source_artifact: 536
environment: debian/match:codename=trixie:variant=sbuild
END

This outputs some YAML structured information:

result: success
message: Work request registered on http://debusine.internal/api with id 315.
work_request_id: 315

At this point, the task has not been executed yet but it has been accepted and will be processed as soon as a worker becomes available. You can follow the status of the work request through the web interface (click on Work requests in the top menu) or with debusine show-work-request WORK_REQUEST_ID.

$ debusine show-work-request 315
id: 315
created_at: 2024-01-24 17:02:31.855184+00:00
started_at: 2024-01-24 17:02:31.937001+00:00
completed_at: null
duration: null
status: running
result: ''
worker: 1
task_name: sbuild
task_data:
[…]
artifacts: []
workspace: System

In the above example, the task is currently running. Note that if the status is pending it indicates that no debusine-worker is currently available to run the task. Check that the debusine-worker is connected, has been approved and has mmdebstrap installed.

Once it has completed (as indicated by status: completed), you will see different values: the result field will be either success, failure (task ran but returned a failure) or error (something unexpected happened), and more information will be available in the artifacts key in particular:

$ debusine show-work-request 315
id: 315
created_at: 2024-01-24 17:02:31.855184+00:00
started_at: 2024-01-24 17:02:31.937001+00:00
completed_at: 2024-01-24 17:05:48.457239+00:00
duration: 196
status: completed
result: success
[…]
artifacts:
- id: 537
  workspace: System
  category: debian:package-build-log
[…]
- id: 538
  workspace: System
  category: debian:binary-package
[…]
- id: 541
  workspace: System
  category: debusine:work-request-debug-logs
[…]

The artifacts listed correspond to sets of files generated by the task, and each task will typically generate the same kind of files as output. In the case of the sbuild task, you will get debian:upload, debian:binary-packages and debian:package-build-log artifacts. You will also get a debusine:work-request-debug-logs artifact containing various files generated by debusine to help troubleshoot issues with the task.

The generated artifacts can be browsed and downloaded from the web interface. But they can also be downloaded from the command line with the debusine client:

$ debusine download-artifact 538
Downloading artifact and uncompressing into /home/debian
hello_2.10-3_amd64.deb

On top of the contained files, the artifact category also defines the structure of the metadata that is associated to the artifact. You can inspect those metadata and the file listing on the web interface or on the command line with debusine show-artifact ARTIFACT_ID (here 540 is the artifact ID of the debian:upload artifact created by the task):

$ debusine show-artifact 540
id: 540
workspace: System
category: debian:upload
created_at: '2024-01-24T17:05:04.975882+00:00'
data:
  type: dpkg
  changes_fields:
    Date: Mon, 26 Dec 2022 16:30:00 +0100
[…]
download_tar_gz_url: http://debusine.internal/artifact/539/?archive=tar.gz
files_to_upload: []
expire_at: null
files:
  hello_2.10-3_amd64.buildinfo:
    size: 5511
    checksums:
      sha256: 422aef340c827d2ed2b38c353f660b70e754509bc0ddb0952975090d9f25caaa
    type: file
    url: http://debusine.internal/artifact/539/hello_2.10-3_amd64.buildinfo
  hello_2.10-3_amd64.changes:
    size: 1889
    checksums:
      sha256: d5d694b42b94587d38a5f883fe1fc5d44368ffe974ac3d506d55bcbef0ab0767
    type: file
    url: http://debusine.internal/artifact/539/hello_2.10-3_amd64.changes
  hello_2.10-3_amd64.deb:
    size: 53084
    checksums:
      sha256: 069754b87d7a546253554813252dacbd7a53e959845cc9f6e8f4c1c8fe3746c5
    type: file
    url: http://debusine.internal/artifact/539/hello_2.10-3_amd64.deb
  hello-dbgsym_2.10-3_amd64.deb:
    size: 35096
    checksums:
      sha256: 1550fcd93105a3cf8fddfc776fda0fbebb51dd7c2d2286eeabc43cb37896ad1e
    type: file
    url: http://debusine.internal/artifact/539/hello-dbgsym_2.10-3_amd64.deb