Debian pipeline

We want to provide a workflow coordinating all the steps that are typically run to build and test an upload to Debian, similar to the Salsa CI pipeline but (eventually) with more distribution-wide testing and the ability to handle the task of performing the upload.

This builds on the existing sbuild workflow.

CheckInstallability task

This is a server-side task that checks whether the uninstallability count in a suite increases as a result of adding packages to it, along the lines of the installability regression tests performed by britney.

The task_data for this task may contain the following keys:

• suite (Single lookup, required): the debian:suite collection to check installability against

• binary_artifacts (Multiple lookup, required): a list of debian:binary-package, debian:binary-packages, or debian:upload artifacts to check

Todo

Check whether it’s feasible to implement this in debusine itself as a server-side task. If not, we’ll need to make it a worker task and consider what environment it should run in.

Todo

Define the output. It should probably be a new artifact category, produced only if the task fails, containing the list of newly-uninstallable packages.

Workflow lintian

This workflow schedules Lintian checks for a single source package and its binaries on a set of architectures.

• task_data:

  • source_artifact (Single lookup, required): see Lintian task

  • binary_artifacts (Multiple lookup, required): see Lintian task

  • vendor (string, required): the distribution vendor on which to run tests

  • codename (string, required): the distribution codename on which to run tests

  • backend (string, optional): see Lintian task

  • architectures (list of strings, optional): if set, only run on any of these architecture names

  • arch_all_host_architecture (string, defaults to amd64): concrete architecture on which to run tasks for Architecture: all packages

  • output, include_tags, exclude_tags, fail_on_severity: see Lintian task

Lintian will be run on the intersection of the provided list of architectures (if any) and the architectures provided in binary_artifacts, in each case grouping source + arch-all + arch-any together for the best test coverage. If only Architecture: all binary packages are provided in binary_artifacts, then Lintian will be run once for source + arch-all on {arch_all_host_architecture}.

The workflow creates a Lintian task for each concrete architecture, with task data:

• input.source_artifact: {source_artifact}

• input.binary_artifacts: the subset of {binary_artifacts} that are for the concrete architecture or all

• host_architecture: the concrete architecture, or {arch_all_host_architecture} if only Architecture: all binary packages are being checked by this task

• environment: {vendor}/match:codename={codename}

• backend: {backend}

• output, include_tags, exclude_tags, fail_on_severity: copied from workflow task data parameters of the same names

Any of the lookups in input.source_artifact or input.binary_artifacts may result in promises, and in that case the workflow adds corresponding dependencies. Binary promises must include an architecture field in their data.

Workflow piuparts

This workflow schedules piuparts checks for binaries built by a single source package on a set of architectures.

• task_data:

  • binary_artifacts (Multiple lookup, required): see Piuparts task

  • vendor (string, required): the distribution vendor on which to run tests

  • codename (string, required): the distribution codename on which to run tests

  • backend (string, optional): see Piuparts task

  • architectures (list of strings, optional): if set, only run on any of these architecture names

  • arch_all_host_architecture (string, defaults to amd64): concrete architecture on which to run tasks for Architecture: all packages

piuparts will be run on the intersection of the provided list of architectures (if any) and the architectures provided in binary_artifacts, in each case grouping arch-all + arch-any together. If only Architecture: all binary packages are provided in binary_artifacts, then piuparts will be run once for arch-all on {arch_all_host_architecture}.

The workflow creates a Piuparts task for each concrete architecture, with task data:

• input.binary_artifacts: the subset of {binary_artifacts} that are for the concrete architecture or all

• host_architecture: the concrete architecture, or {arch_all_host_architecture} if only Architecture: all binary packages are being checked by this task

• environment: {vendor}/match:codename={codename}

• base_tgz: {vendor}/match:codename={codename}

• backend: {backend}

Any of the lookups in input.binary_artifacts may result in promises, and in that case the workflow adds corresponding dependencies. Binary promises must include an architecture field in their data.

Todo

It would be useful to have a mechanism to control multiarch tests, such as testing i386 packages on an amd64 testbed.

Todo

It would be useful to be able to set base_tgz separately from environment.

Workflow qa

• task_data:

  • source_artifact (Single lookup, required): the debian:source-package or debian:upload artifact representing the source package to test

  • binary_artifacts (Multiple lookup, required): the debian:binary-packages or debian:upload artifacts representing the binary packages to test

  • vendor (string, required): the distribution vendor on which to run tests

  • codename (string, required): the distribution codename on which to run tests

  • architectures (list of strings, optional): if set, only run on any of these architecture names

  • architectures_allowlist (list of strings, optional, either concrete architecture names or all): if set, only run on any of these architecture names; while architectures is intended to be supplied by users or passed down from a higher-level workflow, this field is intended to be provided via Task configuration

  • architectures_denylist (list of strings, optional, either concrete architecture names or all): if set, do not run on any of these architecture names; this field is intended to be provided via Task configuration

  • arch_all_host_architecture (string, defaults to amd64): concrete architecture on which to run tasks for Architecture: all packages

  • enable_check_installability (boolean, defaults to True): whether to include installability-checking tasks

  • check_installability_suite (Single lookup, required if enable_check_installability is True): the debian:suite collection to check installability against; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

  • enable_autopkgtest (boolean, defaults to True): whether to include autopkgtest tasks

  • autopkgtest_backend (string, optional): see Autopkgtest task

  • enable_reverse_dependencies_autopkgtest (boolean, defaults to True): whether to include autopkgtest tasks for reverse-dependencies

  • reverse_dependencies_autopkgtest_suite (Single lookup, required if enable_reverse_dependencies_autopkgtest is True): the debian:suite collection to search for reverse-dependencies; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

  • enable_lintian (boolean, defaults to True): whether to include lintian tasks

  • lintian_backend (string, optional): see Lintian task

  • lintian_fail_on_severity (string, optional): see Lintian task

  • enable_piuparts (boolean, defaults to True): whether to include piuparts tasks

  • piuparts_backend (string, optional): see Piuparts task

Any of the lookups in source_artifact or binary_artifacts may result in promises, and in that case the workflow adds corresponding dependencies. Binary promises must include an architecture field in their data.

The list of architectures to run on is the list of architectures from binary_artifacts, intersecting {architectures} if set, intersecting {architectures_allowlist} if set, and subtracting {architectures_denylist} if set.

The workflow creates sub-workflows and tasks as follows, with substitutions based on its own task data:

• if enable_check_installability is set, a single CheckInstallability task, with task data:

  • suite: {check_installability_suite}

  • binary_artifacts: the subset of the lookup in this workflow’s binary_artifacts for each available architecture

• if enable_autopkgtest is set, an autopkgtest sub-workflow, with task data:

  • source_artifact: {source_artifact}

  • binary_artifacts: the subset of the lookup in this workflow’s binary_artifacts for each of all and the concrete architecture in question that exist

  • vendor: {vendor}

  • codename: {codename}

  • backend: {autopkgtest_backend}

  • architectures: {architectures}

  • arch_all_host_architecture: {arch_all_host_architecture}

• if enable_reverse_dependencies_autopkgtest is set, a reverse_dependencies_autopkgtest sub-workflow, with task data:

  • binary_artifacts: the subset of the lookup in this workflow’s binary_artifacts for each of all and the concrete architecture in question that exist

  • suite_collection: {reverse_dependencies_autopkgtest_suite}

  • vendor: {vendor}

  • codename: {codename}

  • backend: {autopkgtest_backend}

  • architectures: {architectures}

  • arch_all_host_architecture: {arch_all_host_architecture}

• if enable_lintian is set, a lintian sub-workflow, with task data:

  • input.source_artifact: {source_artifact}

  • input.binary_artifacts: the subset of the lookup in this workflow’s binary_artifacts for each of all and the concrete architecture in question that exist

  • vendor: {vendor}

  • codename: {codename}

  • backend: {lintian_backend}

  • architectures: {architectures}

  • arch_all_host_architecture: {arch_all_host_architecture}

  • fail_on_severity: {lintian_fail_on_severity}

• if enable_piuparts is set, a piuparts sub-workflow, with task data:

  • input.binary_artifacts: the subset of the lookup in this workflow’s binary_artifacts for each of all and the concrete architecture in question that exist

  • vendor: {vendor}

  • codename: {codename}

  • backend: {piuparts_backend}

  • architectures: {architectures}

  • arch_all_host_architecture: {arch_all_host_architecture}

Confirm task

This wait task blocks until the user who created it says that it may proceed. It is equivalent to a confirmation prompt in a traditional user interface. It has no task data.

Workflow make_signed_source

This workflow produces a source package with signed contents from a template package and some binary packages.

• task_data:

  • binary_artifacts (Multiple lookup, required): the debian:binary-packages or debian:upload artifacts representing the binary packages to test

  • signing_template_artifacts (Multiple lookup, required): the debian:binary-package artifacts representing the binary packages to test

  • vendor (string, required): the distribution vendor on which to sign

  • codename (string, required): the distribution codename on which to sign

  • architectures (list of strings, optional): if set, only run on any of these architecture names

  • purpose (string, required): the purpose of the key to sign with; see Sign task

  • key (Single lookup, required): the debusine:signing-key artifact to sign with; must match purpose

  • sbuild_backend (string, optional): see Task PackageBuild

  • sbuild_build_logs_collection (Single lookup with default category debian:package-build-logs, optional): collection where build logs should be retained

Any of the lookups in binary_artifacts or signing_template_artifacts may result in promises, and in that case the workflow adds corresponding dependencies. Promises must include an architecture field in their data.

The list of architectures to run on is the list of architectures that are present in both binary_artifacts and signing_template_artifacts, intersected with architectures if it is set.

For each architecture, the workflow creates sub-workflows and tasks as follows, with substitutions based on its own task data. Each one has a dependency on the previous one in sequence, using event reactions to store output in the workflow’s internal collection for use by later tasks:

• an ExtractForSigning task, with task data:

  • input.template_artifact: the subset of the lookup in this workflow’s signing_template_artifacts for the concrete architecture in question

  • input.binary_artifacts: the subset of the lookup in this workflow’s binary_artifacts for each of all and the concrete architecture in question that exist

  • environment: {vendor}/match:codename={codename}

• a Sign task, with task data:

  • purpose: {purpose}

  • unsigned: the output of the previous task, from the workflow’s internal collection

  • key: {key}

• an AssembleSignedSource task, with task data:

  • environment: {vendor}/match:codename={codename}

  • template: the subset of the lookup in this workflow’s signing_template_artifacts for the concrete architecture in question

  • signed: the output of the previous task, from the workflow’s internal collection

• an sbuild sub-workflow, with task data:

  • prefix: signed-source/

  • input.source_artifact: the output of the previous task, from the workflow’s internal collection

  • target_distribution: {vendor}:{codename}

  • backend: {sbuild_backend}

  • architectures: if {architectures} is set, then {architectures} plus all

  • build_logs_collection: {sbuild_build_logs_collection}, if set

Todo

We may need to use different keys for different architectures. For example, a UEFI signing key is only useful on architectures that use UEFI, and some architectures have other firmware signing arrangements.

Workflow debian_pipeline

This is a configurable workflow aiming to support all the things that are typically run to build and test an upload to Debian.

• task_data:

  • source_artifact (Single lookup, required): the debian:source-package or debian:upload artifact representing the source package to test

  • vendor (string, required): the distribution vendor on which to run tests

  • codename (string, required): the distribution codename on which to run tests

  • architectures (list of strings, optional): if set, only run on any of these architecture names

  • architectures_allowlist (list of strings, optional, either concrete architecture names or all): if set, only run on any of these architecture names; while architectures is intended to be supplied by users, this field is intended to be provided via Task configuration

  • architectures_denylist (list of strings, optional, either concrete architecture names or all): if set, do not run on any of these architecture names; this field is intended to be provided via Task configuration

  • arch_all_host_architecture (string, defaults to amd64): concrete architecture on which to run tasks for Architecture: all packages

  • signing_template_names (dictionary, optional): mapping from architecture to name of binary package that should be used as signing templates by the make_signed_source sub-workflow

  • sbuild_backend (string, optional): see Task PackageBuild

  • sbuild_build_logs_collection (Single lookup with default category debian:package-build-logs, optional): collection where build logs should be retained

  • sbuild_environment_variant (string, optional): variant of the environment to build on, e.g. buildd

  • enable_check_installability (boolean, defaults to True): whether to include installability-checking tasks

  • check_installability_suite (Single lookup, required if enable_check_installability is True): the debian:suite collection to check installability against; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

  • enable_autopkgtest (boolean, defaults to True): whether to include autopkgtest tasks

  • autopkgtest_backend (string, optional): see Autopkgtest task

  • enable_reverse_dependencies_autopkgtest (boolean, defaults to True): whether to include autopkgtest tasks for reverse-dependencies

  • reverse_dependencies_autopkgtest_suite (Single lookup, required if enable_reverse_dependencies_autopkgtest is True): the debian:suite collection to search for reverse-dependencies; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

  • enable_lintian (boolean, defaults to True): whether to include lintian tasks

  • lintian_backend (string, optional): see Lintian task

  • lintian_fail_on_severity (string, optional): see Lintian task

  • enable_piuparts (boolean, defaults to True): whether to include piuparts tasks

  • piuparts_backend (string, optional): see Piuparts task

  • enable_make_signed_source (boolean, defaults to False): whether to sign the contents of builds and make a signed source package

  • make_signed_source_purpose (string, required): the purpose of the key to sign with; see Sign task

  • make_signed_source_key (Single lookup, required): the debusine:signing-key artifact to sign with; must match purpose

  • enable_upload (boolean, defaults to False): whether to upload to an upload queue

  • upload_include_binaries (boolean, defaults to False): include binaries with the upload

  • upload_since_version (string, optional): if source_artifact is a debian:source-package, include changelog information from all versions strictly later than this version in the .changes file; the default is to include only the topmost changelog entry

  • upload_target_distribution (string, optional): if source_artifact is a debian:source-package, override the target Distribution field in the .changes file to this value; the default is to use the distribution from the topmost changelog entry

  • upload_target (string, defaults to ftp://anonymous@ftp.upload.debian.org/pub/UploadQueue/): the upload queue, as an ftp:// or sftp:// URL

The effective set of architectures is {architectures} (defaulting to all architectures supported by this debusine instance and the {vendor}:{codename} suite, plus all), intersecting {architectures_allowlist} if set, and subtracting {architectures_denylist} if set.

The workflow creates sub-workflows and tasks as follows, with substitutions based on its own task data:

• an sbuild sub-workflow, with task data:

  • input.source_artifact: {source_artifact}

  • target_distribution: {vendor}:{codename}

  • backend: {sbuild_backend}

  • architectures: the effective set of architectures

  • arch_all_host_architecture: {arch_all_host_architecture}, if set

  • build_logs_collection: {sbuild_build_logs_collection}, if set

  • environment_variant: {sbuild_environment_variant}, if set

  • signing_template_names: {signing_template_names}, if set

• if any of enable_check_installability, enable_autopkgtest, enable_lintian, and enable_piuparts are True, a qa sub-workflow, with task data copied from the items of the same name in this workflow’s task data, plus:

  • binary_artifacts: internal@collections/name:build-{architecture}, for each available architecture

  • architectures: the effective set of architectures

• a Confirm task

• if enable_make_signed_source and signing_template_names are set, a make_signed_source sub-workflow, with task data:

  • binary_artifacts: internal@collections/name:build-{architecture}, for each available architecture

  • signing_template_artifacts: internal@collections/name:signing-template-{architecture}-{binary_package_name}, for each architecture and binary package name from signing_template_names

  • vendor: {vendor}

  • codename: {codename}

  • architectures: the effective set of architectures

  • purpose: {make_signed_source_purpose}

  • key: {make_signed_source_key}

  • sbuild_backend: {sbuild_backend}

  • sbuild_build_logs_collection: {sbuild_build_logs_collection}

• if enable_upload is set, a package_upload sub-workflow configured to require a signature from the developer, with task data:

  • source_artifact: {source_artifact}

  • include_binaries: {upload_include_binaries}

  • merge_uploads: True

  • since_version: {upload_since_version}

  • target_distribution: {upload_target_distribution}

  • require_signature: True

  • target: {upload_target}

The first work request for each architecture in the make_signed_source sub-workflow and the first work request in the package_upload sub-workflow depend on the Confirm task above.

Todo

There should also be an option to add the results to a debian:suite collection rather than uploading it to an external queue. However, this isn’t very useful until debusine has its own repository hosting, and once it does, we’ll need to be able to apply consistency checks to uploads rather than just adding them to suites in an unconstrained way. This will probably involve a new workflow yet to be designed.

API changes

A new /api/1.0/work-request/<int:work_request_id>/confirm/ view allows handling the new Confirm wait task. The view works as follows:

• check that the work request is WAIT/Confirm and is running, and otherwise return HTTP 400

• check that the request user is the same as the user who created the work request, and otherwise return HTTP 403

• mark the work request completed

• return HTTP 200

UI changes

When showing a blocked Confirm work request, the web UI shows a “Confirm” button which does the equivalent of the confirm API view above.