debian_pipeline

Workflow debian_pipeline

We want to provide a workflow coordinating all the steps that are typically run to build and test an upload to Debian, similar to the Salsa CI pipeline but (eventually) with more distribution-wide testing and the ability to handle the task of performing the upload.

This builds on the existing sbuild workflow.

Giving contributors access to run this workflow implies giving them access to publish to any value of suite in the same workspace allowed by the workflow template’s static_parameters and runtime_parameters.

  • task_data:

    • source_artifact (Single lookup, required): the debian:source-package or debian:upload artifact representing the source package to test

    • suite (Single lookup, optional): a debian:suite collection to publish packages to; if set, this suite is also added to the start of extra_repositories, and must be in the same workspace as the workflow

    • vendor (string, required): the distribution vendor on which to run tests

    • codename (string, required): the distribution codename on which to run tests

    • extra_repositories (optional): see PackageBuild

    • architectures (list of strings, optional): if set, only run on any of these architecture names

    • architectures_allowlist (list of strings, optional, either concrete architecture names or all): if set, only run on any of these architecture names; while architectures is intended to be supplied by users, this field is intended to be provided via Task configuration

    • architectures_denylist (list of strings, optional, either concrete architecture names or all): if set, do not run on any of these architecture names; this field is intended to be provided via Task configuration

    • arch_all_build_architecture (string, defaults to amd64): concrete architecture on which to run tasks for Architecture: all packages

    • signing_template_names (dictionary, optional): mapping from architecture to list of names of binary packages that should be used as signing templates by the make_signed_source sub-workflow

    • sbuild_backend (string, optional): see PackageBuild

    • sbuild_environment_variant (string, optional): variant of the environment to build on, e.g. buildd

    • qa_suite (Single lookup, required if enable_reverse_dependencies_autopkgtest or enable_debdiff is True): the debian:suite collection to search for reverse-dependencies or debdiff original packages, or to use to generate reference QA results to detect regressions. Once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

    • enable_regression_tracking (boolean, defaults to False): configure the QA workflow to detect and display regressions in QA results

    • regression_tracking_qa_results (Single lookup, required if enable_regression_tracking is True): the debian:qa-results collection that contains the reference results of QA tasks to use to detect regressions

    • qa_failure_policy (string, defaults to ignore): the policy to apply if the qa workflow fails. Allowed values are ignore, fail, confirm. Note that if enable_regression_tracking is set, the qa workflow only fails if one of the QA results is a regression compared to the reference results.

    • enable_check_installability (boolean, defaults to True): whether to include installability-checking tasks

    • check_installability_suite (Single lookup, required if enable_check_installability is True): the debian:suite collection to check installability against; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

    • enable_autopkgtest (boolean, defaults to True): whether to include autopkgtest tasks

    • autopkgtest_backend (string, optional): see Autopkgtest

    • enable_reverse_dependencies_autopkgtest (boolean, defaults to False): whether to include autopkgtest tasks for reverse-dependencies

    • enable_lintian (boolean, defaults to True): whether to include lintian tasks

    • lintian_backend (string, optional): see Lintian

    • lintian_fail_on_severity (string, optional): see Lintian

    • enable_piuparts (boolean, defaults to True): whether to include piuparts tasks

    • piuparts_backend (string, optional): see Piuparts

    • piuparts_environment (string, optional): the environment to run piuparts in

    • enable_debdiff (boolean, defaults to False): whether to include debdiff for source and binary packages, comparing the supplied source package and the built binary packages against the packages available in the distribution identified by qa_suite.

    • enable_blhc (boolean, defaults to True): whether to include blhc tasks for the logs associated to the builds

    • arch_all_expected_installable (list of strings, optional): see qa

    • enable_confirmation (boolean, defaults to False): whether the generated workflow includes a confirmation step asking the user to double check what was built before the upload

    • enable_make_signed_source (boolean, defaults to False): whether to sign the contents of builds and make a signed source package

    • make_signed_source_purpose (string, required only if enable_make_signed_source is True): the purpose of the key to sign with; see Sign

    • make_signed_source_key (string, required only if enable_make_signed_source is True): the fingerprint to sign with; must match purpose

    • enable_upload (boolean, defaults to False): whether to upload to an upload queue

    • upload_key (Single lookup, optional): key used to sign uploads

    • upload_binary_key (Single lookup, optional): key used to sign binary uploads

    • upload_require_signature (boolean, defaults to True): whether uploads must be signed; if this is set and neither upload_key nor (if applicable) upload_binary_key is set, then the user will have to remotely sign the files

    • upload_include_source (boolean, defaults to True): include source with the upload

    • upload_include_binaries (boolean, defaults to True): include binaries with the upload

    • upload_merge_uploads (boolean, defaults to True): if True, merge the uploads for each source package and its binaries that are being uploaded, and create one PackageUpload task per source package to upload them all together; if False, create a separate PackageUpload task for each source and binary upload

    • upload_since_version (string, optional): if source_artifact is a debian:source-package, include changelog information from all versions strictly later than this version in the .changes file; the default is to include only the topmost changelog entry

    • upload_target_distribution (string, optional): if source_artifact is a debian:source-package, override the target Distribution field in the .changes file to this value; the default is to use the distribution from the topmost changelog entry

    • upload_target (string, defaults to ftp://anonymous@ftp.upload.debian.org/pub/UploadQueue/): the upload queue, as an ftp:// or sftp:// URL

    • upload_delayed_days (integer, optional): the number of days to delay the upload; this assumes that the upload queue implements Debian’s convention of uploading delayed uploads to a DELAYED/{n}-day queue

    • publish_replace (boolean, defaults to False): if True, replace existing similar items, if the target suite allows it

The workflow computes dynamic metadata as:

  • subject: package name of source_artifact

  • source_artifact_id: id of the source artifact

The effective set of architectures is {architectures} (defaulting to all architectures supported by this Debusine instance and the {vendor}:{codename} suite, plus all), intersecting {architectures_allowlist} if set, and subtracting {architectures_denylist} if set.

The workflow creates sub-workflows and tasks as follows, with substitutions based on its own task data:

  • an sbuild sub-workflow, with task data:

    • input.source_artifact: {source_artifact}

    • target_distribution: {vendor}:{codename}

    • backend: {sbuild_backend}

    • architectures: the effective set of architectures

    • arch_all_build_architecture: {arch_all_build_architecture}, if set

    • environment_variant: {sbuild_environment_variant}, if set

    • signing_template_names: {signing_template_names}, if set

  • if any of enable_check_installability, enable_autopkgtest, enable_lintian, enable_piuparts and enable_debdiff are True, a qa sub-workflow, with task data copied from the items of the same name in this workflow’s task data, plus:

    • binary_artifacts: internal@collections/name:build-{architecture}, for each available architecture

    • architectures: the effective set of architectures

  • if enable_confirmation is set, a Confirm task will block any signing and upload sub-workflows until it has been approved

  • if enable_make_signed_source and signing_template_names are set, a make_signed_source sub-workflow, with task data:

    • binary_artifacts: internal@collections/name:build-{architecture}, for each available architecture

    • signing_template_artifacts: internal@collections/name:signing-template-{architecture}-{binary_package_name}, for each architecture and binary package name from signing_template_names

    • vendor: {vendor}

    • codename: {codename}

    • architectures: the effective set of architectures

    • purpose: {make_signed_source_purpose}

    • key: {make_signed_source_key}

    • sbuild_backend: {sbuild_backend}

  • if enable_upload is set, a package_upload sub-workflow for each source package being uploaded (at least the top-level source_artifact, but also each assembled signed source package from the make_signed_source sub-workflow if one exists), configured to require a signature from the developer, with task data:

    • source_artifact: the source artifact to upload (or unset if this upload is for the top-level source artifact and upload_include_source is False)

    • binary_artifacts: internal@collections/name:{prefix}build-{architecture}, for each available architecture (or empty if upload_include_binaries is False), where prefix is empty if this upload is for the top-level source artifact or signed-source-{architecture}-{binary_package_name}| if this upload is for an assembled signed source package

    • merge_uploads: {upload_merge_uploads}

    • since_version: {upload_since_version}

    • target_distribution: {upload_target_distribution}

    • key: {upload_key}

    • require_signature: {upload_require_signature}

    • target: {upload_target}

    • vendor: {vendor}

    • codename: {codename}

    • arch_all_build_architecture: {arch_all_build_architecture}

    • delayed_days: {upload_delayed_days}

  • if suite is set, a package_publish sub-workflow for each source package being published (at least the top-level source_artifact, but also each assembled signed source package from the make_signed_source sub-workflow if one exists), with task data:

    • source_artifact: the source artifact to publish

    • binary_artifacts: internal@collections/name:{prefix}build-{architecture}, for each available architecture, where prefix is empty if publishing the top-level source artifact or signed-source-{architecture}-{binary_package_name}| if publishing an assembled signed source package

    • target_suite: {suite}

    • replace: {publish_replace}

The dependencies between these sub-workflows differ based on the value of qa_failure_policy:

  • ignore (default): The qa sub-workflow is scheduled but nothing depends on it. If enable_confirmation is set, then the make_signed_source, package_upload, and package_publish sub-workflows depend on the Confirm task.

  • fail: The make_signed_source, package_upload, and package_publish sub-workflows depend on the qa sub-workflow, so they are not executed before it finishes and are aborted if anything fails.

  • confirm: The qa sub-workflow is scheduled with allow_failure: true in its workflow data, and a Confirm task is scheduled between it and the remaining parts of the workflow. If the qa sub-workflow succeeds, then that task is confirmed automatically; otherwise, it waits for the user to decide whether to continue the workflow despite the QA failures.

When enable_regression_tracking is set

  • the normal qa workflow is run with:

    • enable_regression_tracking set to True

    • reference_qa_results set with the value of regression_tracking_qa_results

    • reference_prefix set to reference-qa-result|

    • fail_on set to regression, so that this sub-workflow fails only if one of the QA results is a regression compared to the reference result

  • an extra qa workflow is run with:

    • enable_regression_tracking set to False

    • reference_qa_results set with the value of regression_tracking_qa_results

    • update_qa_results set to yes

    • prefix set to reference-qa-result|

    • source_artifact and binary_artifacts pointing to the corresponding artifacts in the debian:suite collection listed by qa_suite

    • fail_on set to never (since failures will be analyzed separately)

    • enable_debdiff set to False (since debdiff doesn’t contribute any QA results)

Todo

Not implemented: