debian_pipeline

Workflow debian_pipeline

We want to provide a workflow coordinating all the steps that are typically run to build and test an upload to Debian, similar to the Salsa CI pipeline but (eventually) with more distribution-wide testing and the ability to handle the task of performing the upload.

This builds on the existing sbuild workflow.

• task_data:

  • source_artifact (Single lookup, required): the debian:source-package or debian:upload artifact representing the source package to test

  • vendor (string, required): the distribution vendor on which to run tests

  • codename (string, required): the distribution codename on which to run tests

  • extra_repositories (optional): see PackageBuild

  • architectures (list of strings, optional): if set, only run on any of these architecture names

  • architectures_allowlist (list of strings, optional, either concrete architecture names or all): if set, only run on any of these architecture names; while architectures is intended to be supplied by users, this field is intended to be provided via Task configuration

  • architectures_denylist (list of strings, optional, either concrete architecture names or all): if set, do not run on any of these architecture names; this field is intended to be provided via Task configuration

  • arch_all_host_architecture (string, defaults to amd64): concrete architecture on which to run tasks for Architecture: all packages

  • signing_template_names (dictionary, optional): mapping from architecture to list of names of binary packages that should be used as signing templates by the make_signed_source sub-workflow

  • sbuild_backend (string, optional): see PackageBuild

  • sbuild_environment_variant (string, optional): variant of the environment to build on, e.g. buildd

  • qa_suite (Single lookup, required if enable_reverse_dependencies_autopkgtest or enable_debdiff is True): the debian:suite collection to search for reverse-dependencies or debdiff original packages. Once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

  • enable_check_installability (boolean, defaults to True): whether to include installability-checking tasks

  • check_installability_suite (Single lookup, required if enable_check_installability is True): the debian:suite collection to check installability against; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

  • enable_autopkgtest (boolean, defaults to True): whether to include autopkgtest tasks

  • autopkgtest_backend (string, optional): see Autopkgtest

  • enable_reverse_dependencies_autopkgtest (boolean, defaults to False): whether to include autopkgtest tasks for reverse-dependencies

  • enable_lintian (boolean, defaults to True): whether to include lintian tasks

  • lintian_backend (string, optional): see Lintian

  • lintian_fail_on_severity (string, optional): see Lintian

  • enable_piuparts (boolean, defaults to True): whether to include piuparts tasks

  • piuparts_backend (string, optional): see Piuparts

  • piuparts_environment (string, optional): the environment to run piuparts in

  • enable_debdiff (boolean, defaults to False): whether to include debdiff for source and binary packages, comparing the supplied source package and the built binary packages against the packages available in the distribution identified by qa_suite.

  • enable_blhc (boolean, defaults to True): whether to include blhc tasks for the logs associated to the builds

  • enable_make_signed_source (boolean, defaults to False): whether to sign the contents of builds and make a signed source package

  • make_signed_source_purpose (string, required only if enable_make_signed_source is True): the purpose of the key to sign with; see Sign

  • make_signed_source_key (string, required only if enable_make_signed_source is True): the fingerprint to sign with; must match purpose

  • enable_confirmation (boolean, defaults to False): whether the generated workflow includes a confirmation step asking the user to double check what was built before the upload

  • enable_upload (boolean, defaults to False): whether to upload to an upload queue

  • upload_key (Single lookup, optional): key used to sign uploads

  • upload_binary_key (Single lookup, optional): key used to sign binary uploads

  • upload_require_signature (boolean, defaults to True): whether uploads must be signed; if this is set and neither upload_key nor (if applicable) upload_binary_key is set, then the user will have to remotely sign the files

  • upload_include_source (boolean, defaults to True): include source with the upload

  • upload_include_binaries (boolean, defaults to True): include binaries with the upload

  • upload_merge_uploads (boolean, defaults to True): if True, merge the uploads for each source package and its binaries that are being uploaded, and create one PackageUpload task per source package to upload them all together; if False, create a separate PackageUpload task for each source and binary upload

  • upload_since_version (string, optional): if source_artifact is a debian:source-package, include changelog information from all versions strictly later than this version in the .changes file; the default is to include only the topmost changelog entry

  • upload_target_distribution (string, optional): if source_artifact is a debian:source-package, override the target Distribution field in the .changes file to this value; the default is to use the distribution from the topmost changelog entry

  • upload_target (string, defaults to ftp://anonymous@ftp.upload.debian.org/pub/UploadQueue/): the upload queue, as an ftp:// or sftp:// URL

  • upload_delayed_days (integer, optional): the number of days to delay the upload; this assumes that the upload queue implements Debian’s convention of uploading delayed uploads to a DELAYED/{n}-day queue

  • publish_target (Single lookup, optional): a debian:suite collection to publish packages to

  • publish_replace (boolean, defaults to False): if True, replace existing similar items, if the target suite allows it

The workflow computes dynamic metadata as:

• subject: package name of source_artifact

The effective set of architectures is {architectures} (defaulting to all architectures supported by this Debusine instance and the {vendor}:{codename} suite, plus all), intersecting {architectures_allowlist} if set, and subtracting {architectures_denylist} if set.

The workflow creates sub-workflows and tasks as follows, with substitutions based on its own task data:

• an sbuild sub-workflow, with task data:

  • input.source_artifact: {source_artifact}

  • target_distribution: {vendor}:{codename}

  • backend: {sbuild_backend}

  • architectures: the effective set of architectures

  • arch_all_host_architecture: {arch_all_host_architecture}, if set

  • environment_variant: {sbuild_environment_variant}, if set

  • signing_template_names: {signing_template_names}, if set

• if any of enable_check_installability, enable_autopkgtest, enable_lintian, enable_piuparts and enable_debdiff are True, a qa sub-workflow, with task data copied from the items of the same name in this workflow’s task data, plus:

  • binary_artifacts: internal@collections/name:build-{architecture}, for each available architecture

  • architectures: the effective set of architectures

• if enable_confirmation is set, a Confirm

• if enable_make_signed_source and signing_template_names are set, a make_signed_source sub-workflow, with task data:

  • binary_artifacts: internal@collections/name:build-{architecture}, for each available architecture

  • signing_template_artifacts: internal@collections/name:signing-template-{architecture}-{binary_package_name}, for each architecture and binary package name from signing_template_names

  • vendor: {vendor}

  • codename: {codename}

  • architectures: the effective set of architectures

  • purpose: {make_signed_source_purpose}

  • key: {make_signed_source_key}

  • sbuild_backend: {sbuild_backend}

• if enable_upload is set, a package_upload sub-workflow for each source package being uploaded (at least the top-level source_artifact, but also each assembled signed source package from the make_signed_source sub-workflow if one exists), configured to require a signature from the developer, with task data:

  • source_artifact: the source artifact to upload (or unset if this upload is for the top-level source artifact and upload_include_source is False)

  • binary_artifacts: internal@collections/name:{prefix}build-{architecture}, for each available architecture (or empty if upload_include_binaries is False), where prefix is empty if this upload is for the top-level source artifact or signed-source-{architecture}-{binary_package_name}| if this upload is for an assembled signed source package

  • merge_uploads: {upload_merge_uploads}

  • since_version: {upload_since_version}

  • target_distribution: {upload_target_distribution}

  • key: {upload_key}

  • require_signature: {upload_require_signature}

  • target: {upload_target}

  • vendor: {vendor}

  • codename: {codename}

  • arch_all_host_architecture: {arch_all_host_architecture}

  • delayed_days: {upload_delayed_days}

• if publish_target is set, a package_publish sub-workflow for each source package being uploaded (at least the top-level source_artifact, but also each assembled signed source package from the make_signed_source sub-workflow if one exists), with task data:

  • source_artifact: the source artifact to upload (or unset if this upload is for the top-level source artifact and upload_include_source is False)

  • binary_artifacts: internal@collections/name:{prefix}build-{architecture}, for each available architecture (or empty if upload_include_binaries is False), where prefix is empty if this upload is for the top-level source artifact or signed-source-{architecture}-{binary_package_name}| if this upload is for an assembled signed source package

  • target_suite: {publish_target}

  • replace: {publish_replace}

The first work request for each architecture in the make_signed_source sub-workflow depends on the Confirm task above, as do the first work requests in each of the package_upload and package_publish sub-workflows.

Todo

Not implemented:

• enable_check_installability, check_installability_suite (see CheckInstallability)

• enable_confirmation (see Confirm)

• publish_target, publish_replace