debian_pipeline

Workflow debian_pipeline

We want to provide a workflow coordinating all the steps that are typically run to build and test an upload to Debian, similar to the Salsa CI pipeline but (eventually) with more distribution-wide testing and the ability to handle the task of performing the upload.

This builds on the existing sbuild workflow.

  • task_data:

    • source_artifact (Single lookup, required): the debian:source-package or debian:upload artifact representing the source package to test

    • vendor (string, required): the distribution vendor on which to run tests

    • codename (string, required): the distribution codename on which to run tests

    • extra_repositories (optional): see PackageBuild

    • architectures (list of strings, optional): if set, only run on any of these architecture names

    • architectures_allowlist (list of strings, optional, either concrete architecture names or all): if set, only run on any of these architecture names; while architectures is intended to be supplied by users, this field is intended to be provided via Task configuration

    • architectures_denylist (list of strings, optional, either concrete architecture names or all): if set, do not run on any of these architecture names; this field is intended to be provided via Task configuration

    • arch_all_host_architecture (string, defaults to amd64): concrete architecture on which to run tasks for Architecture: all packages

    • signing_template_names (dictionary, optional): mapping from architecture to list of names of binary packages that should be used as signing templates by the make_signed_source sub-workflow

    • sbuild_backend (string, optional): see PackageBuild

    • sbuild_environment_variant (string, optional): variant of the environment to build on, e.g. buildd

    • qa_suite (Single lookup, required if enable_reverse_dependencies_autopkgtest or enable_debdiff is True): the debian:suite collection to search for reverse-dependencies or debdiff original packages, or to use to generate reference QA results to detect regressions. Once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

    • enable_regression_tracking (boolean, defaults to False): configure the QA workflow to detect and display regressions in QA results

    • regression_tracking_qa_results (Single lookup, required if enable_regression_tracking is True): the debian:qa-results collection that contains the reference results of QA tasks to use to detect regressions

    • qa_failure_policy (string, defaults to ignore): the policy to apply if the qa workflow fails. Allowed values are ignore, fail, confirm.

    • enable_check_installability (boolean, defaults to True): whether to include installability-checking tasks

    • check_installability_suite (Single lookup, required if enable_check_installability is True): the debian:suite collection to check installability against; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

    • enable_autopkgtest (boolean, defaults to True): whether to include autopkgtest tasks

    • autopkgtest_backend (string, optional): see Autopkgtest

    • enable_reverse_dependencies_autopkgtest (boolean, defaults to False): whether to include autopkgtest tasks for reverse-dependencies

    • enable_lintian (boolean, defaults to True): whether to include lintian tasks

    • lintian_backend (string, optional): see Lintian

    • lintian_fail_on_severity (string, optional): see Lintian

    • enable_piuparts (boolean, defaults to True): whether to include piuparts tasks

    • piuparts_backend (string, optional): see Piuparts

    • piuparts_environment (string, optional): the environment to run piuparts in

    • enable_debdiff (boolean, defaults to False): whether to include debdiff for source and binary packages, comparing the supplied source package and the built binary packages against the packages available in the distribution identified by qa_suite.

    • enable_blhc (boolean, defaults to True): whether to include blhc tasks for the logs associated to the builds

    • enable_confirmation (boolean, defaults to False): whether the generated workflow includes a confirmation step asking the user to double check what was built before the upload

    • enable_make_signed_source (boolean, defaults to False): whether to sign the contents of builds and make a signed source package

    • make_signed_source_purpose (string, required only if enable_make_signed_source is True): the purpose of the key to sign with; see Sign

    • make_signed_source_key (string, required only if enable_make_signed_source is True): the fingerprint to sign with; must match purpose

    • enable_upload (boolean, defaults to False): whether to upload to an upload queue

    • upload_key (Single lookup, optional): key used to sign uploads

    • upload_binary_key (Single lookup, optional): key used to sign binary uploads

    • upload_require_signature (boolean, defaults to True): whether uploads must be signed; if this is set and neither upload_key nor (if applicable) upload_binary_key is set, then the user will have to remotely sign the files

    • upload_include_source (boolean, defaults to True): include source with the upload

    • upload_include_binaries (boolean, defaults to True): include binaries with the upload

    • upload_merge_uploads (boolean, defaults to True): if True, merge the uploads for each source package and its binaries that are being uploaded, and create one PackageUpload task per source package to upload them all together; if False, create a separate PackageUpload task for each source and binary upload

    • upload_since_version (string, optional): if source_artifact is a debian:source-package, include changelog information from all versions strictly later than this version in the .changes file; the default is to include only the topmost changelog entry

    • upload_target_distribution (string, optional): if source_artifact is a debian:source-package, override the target Distribution field in the .changes file to this value; the default is to use the distribution from the topmost changelog entry

    • upload_target (string, defaults to ftp://anonymous@ftp.upload.debian.org/pub/UploadQueue/): the upload queue, as an ftp:// or sftp:// URL

    • upload_delayed_days (integer, optional): the number of days to delay the upload; this assumes that the upload queue implements Debian’s convention of uploading delayed uploads to a DELAYED/{n}-day queue

    • publish_target (Single lookup, optional): a debian:suite collection to publish packages to

    • publish_replace (boolean, defaults to False): if True, replace existing similar items, if the target suite allows it

The workflow computes dynamic metadata as:

  • subject: package name of source_artifact

The effective set of architectures is {architectures} (defaulting to all architectures supported by this Debusine instance and the {vendor}:{codename} suite, plus all), intersecting {architectures_allowlist} if set, and subtracting {architectures_denylist} if set.

The workflow creates sub-workflows and tasks as follows, with substitutions based on its own task data:

  • an sbuild sub-workflow, with task data:

    • input.source_artifact: {source_artifact}

    • target_distribution: {vendor}:{codename}

    • backend: {sbuild_backend}

    • architectures: the effective set of architectures

    • arch_all_host_architecture: {arch_all_host_architecture}, if set

    • environment_variant: {sbuild_environment_variant}, if set

    • signing_template_names: {signing_template_names}, if set

  • if any of enable_check_installability, enable_autopkgtest, enable_lintian, enable_piuparts and enable_debdiff are True, a qa sub-workflow, with task data copied from the items of the same name in this workflow’s task data, plus:

    • binary_artifacts: internal@collections/name:build-{architecture}, for each available architecture

    • architectures: the effective set of architectures

  • if enable_confirmation is set, a Confirm

  • if enable_make_signed_source and signing_template_names are set, a make_signed_source sub-workflow, with task data:

    • binary_artifacts: internal@collections/name:build-{architecture}, for each available architecture

    • signing_template_artifacts: internal@collections/name:signing-template-{architecture}-{binary_package_name}, for each architecture and binary package name from signing_template_names

    • vendor: {vendor}

    • codename: {codename}

    • architectures: the effective set of architectures

    • purpose: {make_signed_source_purpose}

    • key: {make_signed_source_key}

    • sbuild_backend: {sbuild_backend}

  • if enable_upload is set, a package_upload sub-workflow for each source package being uploaded (at least the top-level source_artifact, but also each assembled signed source package from the make_signed_source sub-workflow if one exists), configured to require a signature from the developer, with task data:

    • source_artifact: the source artifact to upload (or unset if this upload is for the top-level source artifact and upload_include_source is False)

    • binary_artifacts: internal@collections/name:{prefix}build-{architecture}, for each available architecture (or empty if upload_include_binaries is False), where prefix is empty if this upload is for the top-level source artifact or signed-source-{architecture}-{binary_package_name}| if this upload is for an assembled signed source package

    • merge_uploads: {upload_merge_uploads}

    • since_version: {upload_since_version}

    • target_distribution: {upload_target_distribution}

    • key: {upload_key}

    • require_signature: {upload_require_signature}

    • target: {upload_target}

    • vendor: {vendor}

    • codename: {codename}

    • arch_all_host_architecture: {arch_all_host_architecture}

    • delayed_days: {upload_delayed_days}

  • if publish_target is set, a package_publish sub-workflow for each source package being published (at least the top-level source_artifact, but also each assembled signed source package from the make_signed_source sub-workflow if one exists), with task data:

    • source_artifact: the source artifact to publish

    • binary_artifacts: internal@collections/name:{prefix}build-{architecture}, for each available architecture, where prefix is empty if publishing the top-level source artifact or signed-source-{architecture}-{binary_package_name}| if publishing an assembled signed source package

    • target_suite: {publish_target}

    • replace: {publish_replace}

The dependencies between these sub-workflows differ based on the value of qa_failure_policy:

  • ignore (default): The qa sub-workflow is scheduled but nothing depends on it. If enable_confirmation is set, then the make_signed_source, package_upload, and package_publish sub-workflows depend on the Confirm task.

  • fail: The make_signed_source, package_upload, and package_publish sub-workflows depend on the qa sub-workflow, so they are not executed before it finishes and are aborted if anything fails.

  • confirm: The qa sub-workflow is scheduled with allow_failure: true in its workflow data, and a Confirm task is scheduled between it and the remaining parts of the workflow. If the qa sub-workflow succeeds, then that task is confirmed automatically; otherwise, it waits for the user to decide whether to continue the workflow despite the QA failures.

When enable_regression_tracking is set

  • the normal qa workflow is run with:

    • enable_regression_tracking set to True

    • reference_qa_results set with the value of regression_tracking_qa_results

    • reference_prefix set to reference-qa-result|

  • an extra qa workflow is run with:

    • enable_regression_tracking set to False

    • reference_qa_results set with the value of regression_tracking_qa_results

    • update_qa_results set to True

    • prefix set to reference-qa-result|

    • source_artifact and binary_artifacts pointing to the corresponding artifacts in the debian:suite collection listed by qa_suite

    • fail_on set to never

    • enable_debdiff set to False (since debdiff doesn’t contribute any QA results)

Todo

Not implemented: