.. workflow:: debian_pipeline Workflow ``debian_pipeline`` ============================ We want to provide a workflow coordinating all the steps that are typically run to build and test an upload to Debian, similar to the `Salsa CI pipeline `_ but (eventually) with more distribution-wide testing and the ability to handle the task of performing the upload. This builds on the existing :workflow:`sbuild` workflow. * ``task_data``: * ``source_artifact`` (:ref:`lookup-single`, required): the :artifact:`debian:source-package` or :artifact:`debian:upload` artifact representing the source package to test * ``vendor`` (string, required): the distribution vendor on which to run tests * ``codename`` (string, required): the distribution codename on which to run tests * ``extra_repositories`` (optional): see :task:`PackageBuild` * ``architectures`` (list of strings, optional): if set, only run on any of these architecture names * ``architectures_allowlist`` (list of strings, optional, either concrete architecture names or ``all``): if set, only run on any of these architecture names; while ``architectures`` is intended to be supplied by users, this field is intended to be provided via :ref:`task-configuration` * ``architectures_denylist`` (list of strings, optional, either concrete architecture names or ``all``): if set, do not run on any of these architecture names; this field is intended to be provided via :ref:`task-configuration` * ``arch_all_host_architecture`` (string, defaults to ``amd64``): concrete architecture on which to run tasks for ``Architecture: all`` packages * ``signing_template_names`` (dictionary, optional): mapping from architecture to list of names of binary packages that should be used as signing templates by the :workflow:`make_signed_source` sub-workflow * ``sbuild_backend`` (string, optional): see :task:`PackageBuild` * ``sbuild_environment_variant`` (string, optional): variant of the environment to build on, e.g. ``buildd`` * ``qa_suite`` (:ref:`lookup-single`, required if ``enable_reverse_dependencies_autopkgtest`` or ``enable_debdiff`` is True): the :collection:`debian:suite` collection to search for reverse-dependencies or debdiff original packages. Once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so * ``enable_check_installability`` (boolean, defaults to True): whether to include installability-checking tasks * ``check_installability_suite`` (:ref:`lookup-single`, required if ``enable_check_installability`` is True): the :collection:`debian:suite` collection to check installability against; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so * ``enable_autopkgtest`` (boolean, defaults to True): whether to include autopkgtest tasks * ``autopkgtest_backend`` (string, optional): see :task:`Autopkgtest` * ``enable_reverse_dependencies_autopkgtest`` (boolean, defaults to False): whether to include autopkgtest tasks for reverse-dependencies * ``enable_lintian`` (boolean, defaults to True): whether to include lintian tasks * ``lintian_backend`` (string, optional): see :task:`Lintian` * ``lintian_fail_on_severity`` (string, optional): see :task:`Lintian` * ``enable_piuparts`` (boolean, defaults to True): whether to include piuparts tasks * ``piuparts_backend`` (string, optional): see :task:`Piuparts` * ``piuparts_environment`` (string, optional): the environment to run piuparts in * ``enable_debdiff`` (boolean, defaults to False): whether to include debdiff for source and binary packages, comparing the supplied source package and the built binary packages against the packages available in the distribution identified by ``qa_suite``. * ``enable_blhc`` (boolean, defaults to True): whether to include ``blhc`` tasks for the logs associated to the builds * ``enable_make_signed_source`` (boolean, defaults to False): whether to sign the contents of builds and make a signed source package * ``make_signed_source_purpose`` (string, required only if ``enable_make_signed_source`` is True): the purpose of the key to sign with; see :task:`Sign` * ``make_signed_source_key`` (string, required only if ``enable_make_signed_source`` is True): the fingerprint to sign with; must match ``purpose`` * ``enable_confirmation`` (boolean, defaults to False): whether the generated workflow includes a confirmation step asking the user to double check what was built before the upload * ``enable_upload`` (boolean, defaults to False): whether to upload to an upload queue * ``upload_key`` (:ref:`lookup-single`, optional): key used to sign uploads * ``upload_binary_key`` (:ref:`lookup-single`, optional): key used to sign binary uploads * ``upload_require_signature`` (boolean, defaults to True): whether uploads must be signed; if this is set and neither ``upload_key`` nor (if applicable) ``upload_binary_key`` is set, then the user will have to remotely sign the files * ``upload_include_source`` (boolean, defaults to True): include source with the upload * ``upload_include_binaries`` (boolean, defaults to True): include binaries with the upload * ``upload_merge_uploads`` (boolean, defaults to True): if True, merge the uploads for each source package and its binaries that are being uploaded, and create one PackageUpload task per source package to upload them all together; if False, create a separate PackageUpload task for each source and binary upload * ``upload_since_version`` (string, optional): if ``source_artifact`` is a :artifact:`debian:source-package`, include changelog information from all versions strictly later than this version in the ``.changes`` file; the default is to include only the topmost changelog entry * ``upload_target_distribution`` (string, optional): if ``source_artifact`` is a :artifact:`debian:source-package`, override the target ``Distribution`` field in the ``.changes`` file to this value; the default is to use the distribution from the topmost changelog entry * ``upload_target`` (string, defaults to ``ftp://anonymous@ftp.upload.debian.org/pub/UploadQueue/``): the upload queue, as an ``ftp://`` or ``sftp://`` URL * ``upload_delayed_days`` (integer, optional): the number of days to delay the upload; this assumes that the upload queue implements Debian's convention of uploading delayed uploads to a ``DELAYED/{n}-day`` queue * ``publish_target`` (:ref:`lookup-single`, optional): a :collection:`debian:suite` collection to publish packages to * ``publish_replace`` (boolean, defaults to False): if True, replace existing similar items, if the target suite allows it The workflow computes dynamic metadata as: .. dynamic_data:: :method: debusine.server.workflows.debian_pipeline::DebianPipelineWorkflow.build_dynamic_data The effective set of architectures is ``{architectures}`` (defaulting to all architectures supported by this Debusine instance and the ``{vendor}:{codename}`` suite, plus ``all``), intersecting ``{architectures_allowlist}`` if set, and subtracting ``{architectures_denylist}`` if set. The workflow creates sub-workflows and tasks as follows, with substitutions based on its own task data: * an :workflow:`sbuild` sub-workflow, with task data: * ``input.source_artifact``: ``{source_artifact}`` * ``target_distribution``: ``{vendor}:{codename}`` * ``backend``: ``{sbuild_backend}`` * ``architectures``: the effective set of architectures * ``arch_all_host_architecture``: ``{arch_all_host_architecture}``, if set * ``environment_variant``: ``{sbuild_environment_variant}``, if set * ``signing_template_names``: ``{signing_template_names}``, if set * if any of ``enable_check_installability``, ``enable_autopkgtest``, ``enable_lintian``, ``enable_piuparts`` and ``enable_debdiff`` are True, a :workflow:`qa` sub-workflow, with task data copied from the items of the same name in this workflow's task data, plus: * ``binary_artifacts``: ``internal@collections/name:build-{architecture}``, for each available architecture * ``architectures``: the effective set of architectures * if ``enable_confirmation`` is set, a :task:`Confirm` * if ``enable_make_signed_source`` and ``signing_template_names`` are set, a :workflow:`make_signed_source` sub-workflow, with task data: * ``binary_artifacts``: ``internal@collections/name:build-{architecture}``, for each available architecture * ``signing_template_artifacts``: ``internal@collections/name:signing-template-{architecture}-{binary_package_name}``, for each architecture and binary package name from ``signing_template_names`` * ``vendor``: ``{vendor}`` * ``codename``: ``{codename}`` * ``architectures``: the effective set of architectures * ``purpose``: ``{make_signed_source_purpose}`` * ``key``: ``{make_signed_source_key}`` * ``sbuild_backend``: ``{sbuild_backend}`` * if ``enable_upload`` is set, a :workflow:`package_upload` sub-workflow for each source package being uploaded (at least the top-level ``source_artifact``, but also each assembled signed source package from the :workflow:`make_signed_source` sub-workflow if one exists), configured to require a signature from the developer, with task data: * ``source_artifact``: the source artifact to upload (or unset if this upload is for the top-level source artifact and ``upload_include_source`` is False) * ``binary_artifacts``: ``internal@collections/name:{prefix}build-{architecture}``, for each available architecture (or empty if ``upload_include_binaries`` is False), where ``prefix`` is empty if this upload is for the top-level source artifact or ``signed-source-{architecture}-{binary_package_name}|`` if this upload is for an assembled signed source package * ``merge_uploads``: ``{upload_merge_uploads}`` * ``since_version``: ``{upload_since_version}`` * ``target_distribution``: ``{upload_target_distribution}`` * ``key``: ``{upload_key}`` * ``require_signature``: ``{upload_require_signature}`` * ``target``: ``{upload_target}`` * ``vendor``: ``{vendor}`` * ``codename``: ``{codename}`` * ``arch_all_host_architecture``: ``{arch_all_host_architecture}`` * ``delayed_days``: ``{upload_delayed_days}`` * if ``publish_target`` is set, a :workflow:`package_publish` sub-workflow for each source package being published (at least the top-level ``source_artifact``, but also each assembled signed source package from the :workflow:`make_signed_source` sub-workflow if one exists), with task data: * ``source_artifact``: the source artifact to publish * ``binary_artifacts``: ``internal@collections/name:{prefix}build-{architecture}``, for each available architecture, where ``prefix`` is empty if publishing the top-level source artifact or ``signed-source-{architecture}-{binary_package_name}|`` if publishing an assembled signed source package * ``target_suite``: ``{publish_target}`` * ``replace``: ``{publish_replace}`` The first work request for each architecture in the :workflow:`make_signed_source` sub-workflow depends on the :task:`Confirm` task above, as do the first work requests in each of the :workflow:`package_upload` and :workflow:`package_publish` sub-workflows. .. todo:: Not implemented: * ``enable_check_installability``, ``check_installability_suite`` (see :task:`CheckInstallability`) * ``enable_confirmation`` (see :task:`Confirm`) * ``publish_target``, ``publish_replace``