Workflow debian_pipeline

We want to provide a workflow coordinating all the steps that are typically run to build and test an upload to Debian, similar to the Salsa CI pipeline but (eventually) with more distribution-wide testing and the ability to handle the task of performing the upload.

This builds on the existing sbuild workflow.

  • task_data:

    • source_artifact (Single lookup, required): the debian:source-package or debian:upload artifact representing the source package to test

    • vendor (string, required): the distribution vendor on which to run tests

    • codename (string, required): the distribution codename on which to run tests

    • architectures (list of strings, optional): if set, only run on any of these architecture names

    • architectures_allowlist (list of strings, optional, either concrete architecture names or all): if set, only run on any of these architecture names; while architectures is intended to be supplied by users, this field is intended to be provided via Task configuration

    • architectures_denylist (list of strings, optional, either concrete architecture names or all): if set, do not run on any of these architecture names; this field is intended to be provided via Task configuration

    • arch_all_host_architecture (string, defaults to amd64): concrete architecture on which to run tasks for Architecture: all packages

    • signing_template_names (dictionary, optional): mapping from architecture to list of names of binary packages that should be used as signing templates by the make_signed_source sub-workflow

    • sbuild_backend (string, optional): see Task PackageBuild

    • sbuild_environment_variant (string, optional): variant of the environment to build on, e.g. buildd

    • enable_check_installability (boolean, defaults to True): whether to include installability-checking tasks

    • check_installability_suite (Single lookup, required if enable_check_installability is True): the debian:suite collection to check installability against; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

    • enable_autopkgtest (boolean, defaults to True): whether to include autopkgtest tasks

    • autopkgtest_backend (string, optional): see Autopkgtest task

    • enable_reverse_dependencies_autopkgtest (boolean, defaults to True): whether to include autopkgtest tasks for reverse-dependencies

    • reverse_dependencies_autopkgtest_suite (Single lookup, required if enable_reverse_dependencies_autopkgtest is True): the debian:suite collection to search for reverse-dependencies; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

    • enable_lintian (boolean, defaults to True): whether to include lintian tasks

    • lintian_backend (string, optional): see Lintian task

    • lintian_fail_on_severity (string, optional): see Lintian task

    • enable_piuparts (boolean, defaults to True): whether to include piuparts tasks

    • piuparts_backend (string, optional): see Piuparts task

    • enable_make_signed_source (boolean, defaults to False): whether to sign the contents of builds and make a signed source package

    • make_signed_source_purpose (string, required only if enable_make_signed_source is True): the purpose of the key to sign with; see Sign task

    • make_signed_source_key (Single lookup, required only if enable_make_signed_source is True): the debusine:signing-key artifact to sign with; must match purpose

    • enable_confirmation (boolean, defaults to False): whether the generated workflow includes a confirmation step asking the user to double check what was built before the upload

    • enable_upload (boolean, defaults to False): whether to upload to an upload queue

    • upload_key (Single lookup, optional): key used to sign the uploads. If not set and if upload_require_signature is True, then the user will have to remotely sign the files.

    • upload_require_signature (boolean, defaults to True): whether the uploads must be signed

    • upload_include_source (boolean, defaults to True): include source with the upload

    • upload_include_binaries (boolean, defaults to True): include binaries with the upload

    • upload_merge_uploads (boolean, defaults to True): if True, merge the uploads and create a single PackageUpload task to upload them all together; if False, create a separate PackageUpload task for each upload

    • upload_since_version (string, optional): if source_artifact is a debian:source-package, include changelog information from all versions strictly later than this version in the .changes file; the default is to include only the topmost changelog entry

    • upload_target_distribution (string, optional): if source_artifact is a debian:source-package, override the target Distribution field in the .changes file to this value; the default is to use the distribution from the topmost changelog entry

    • upload_target (string, defaults to ftp://anonymous@ftp.upload.debian.org/pub/UploadQueue/): the upload queue, as an ftp:// or sftp:// URL

    • upload_delayed_days (integer, optional): the number of days to delay the upload; this assumes that the upload queue implements Debian’s convention of uploading delayed uploads to a DELAYED/{n}-day queue

The effective set of architectures is {architectures} (defaulting to all architectures supported by this debusine instance and the {vendor}:{codename} suite, plus all), intersecting {architectures_allowlist} if set, and subtracting {architectures_denylist} if set.

The workflow creates sub-workflows and tasks as follows, with substitutions based on its own task data:

  • an sbuild sub-workflow, with task data:

    • input.source_artifact: {source_artifact}

    • target_distribution: {vendor}:{codename}

    • backend: {sbuild_backend}

    • architectures: the effective set of architectures

    • arch_all_host_architecture: {arch_all_host_architecture}, if set

    • environment_variant: {sbuild_environment_variant}, if set

    • signing_template_names: {signing_template_names}, if set

  • if any of enable_check_installability, enable_autopkgtest, enable_lintian, and enable_piuparts are True, a qa sub-workflow, with task data copied from the items of the same name in this workflow’s task data, plus:

    • binary_artifacts: internal@collections/name:build-{architecture}, for each available architecture

    • architectures: the effective set of architectures

  • if enable_confirmation is set, a Confirm task

  • if enable_make_signed_source and signing_template_names are set, a make_signed_source sub-workflow, with task data:

    • binary_artifacts: internal@collections/name:build-{architecture}, for each available architecture

    • signing_template_artifacts: internal@collections/name:signing-template-{architecture}-{binary_package_name}, for each architecture and binary package name from signing_template_names

    • vendor: {vendor}

    • codename: {codename}

    • architectures: the effective set of architectures

    • purpose: {make_signed_source_purpose}

    • key: {make_signed_source_key}

    • sbuild_backend: {sbuild_backend}

  • if enable_upload is set, a package_upload sub-workflow configured to require a signature from the developer, with task data:

    • source_artifact: {source_artifact} (left unset if upload_include_source is False)

    • binary_artifacts: internal@collections/name:build-{architecture}, for each available architecture

    • merge_uploads: {upload_merge_uploads}

    • since_version: {upload_since_version}

    • target_distribution: {upload_target_distribution}

    • key: {upload_key}

    • require_signature: {upload_require_signature}

    • target: {upload_target}

    • vendor: {vendor}

    • codename: {codename}

    • delayed_days: {upload_delayed_days}

The first work request for each architecture in the make_signed_source sub-workflow and the first work request in the package_upload sub-workflow depend on the Confirm task above.

Todo

Not implemented: enable_check_installability, check_installability_suite, enable_reverse_dependencies_autopkgtest, reverse_dependencies_autopkgtest_suite and enable_confirmation. See the relevant blueprints for task installability, reverse dependencies autopkgtest or enable confirmation.

Todo

There should also be an option to add the results to a debian:suite collection rather than uploading it to an external queue. However, this isn’t very useful until debusine has its own repository hosting, and once it does, we’ll need to be able to apply consistency checks to uploads rather than just adding them to suites in an unconstrained way. This will probably involve a new workflow yet to be designed.

Todo

The pipeline should also include the ability to schedule a debdiff against a baseline suite (either directly or in a sub-workflow).