.. workflow:: qa Workflow ``qa`` =============== * ``task_data``: * ``prefix`` (string, optional): prefix this string to the item names provided in the internal collection * ``reference_prefix`` (string, optional unless ``enable_regression_tracking`` is set): prefix for the item names provided in the internal collection in the corresponding workflow run for reference tests * ``source_artifact`` (:ref:`lookup-single`, required): the :artifact:`debian:source-package` or :artifact:`debian:upload` artifact representing the source package to test * ``binary_artifacts`` (:ref:`lookup-multiple`, required): the :artifact:`debian:binary-package` or :artifact:`debian:upload` artifacts representing the binary packages to test * ``package_build_logs`` (:ref:`lookup-multiple`, optional): the :artifact:`debian:package-build-log` artifacts representing the build logs. Required if ``enable_blhc`` is True * ``qa_suite`` (:ref:`lookup-single`, optional unless ``enable_regression_tracking``, ``enable_reverse_dependencies_autopkgtest``, ``update_qa_results``, or ``enable_debdiff`` is True): the :collection:`debian:suite` collection that reference tests are being run against to detect regressions, to search for reverse-dependencies, and to debdiff against * ``reference_qa_results`` (:ref:`lookup-single`, optional unless ``enable_regression_tracking`` or ``update_qa_results`` is True): the :collection:`debian:qa-results` collection that contains the reference results of QA tasks to use to detect regressions * ``enable_regression_tracking`` (boolean, defaults to False): configure the workflow to detect and display regressions in QA results * ``update_qa_results`` (boolean, defaults to False): when set to True, the workflow runs QA tasks and updates the collection passed in ``reference_qa_results`` with the results. * ``vendor`` (string, required): the distribution vendor on which to run tests * ``codename`` (string, required): the distribution codename on which to run tests * ``extra_repositories`` (optional): see :task:`PackageBuild` * ``architectures`` (list of strings, optional): if set, only run on any of these architecture names * ``architectures_allowlist`` (list of strings, optional, either concrete architecture names or ``all``): if set, only run on any of these architecture names; while ``architectures`` is intended to be supplied by users or passed down from a higher-level workflow, this field is intended to be provided via :ref:`task-configuration` * ``architectures_denylist`` (list of strings, optional, either concrete architecture names or ``all``): if set, do not run on any of these architecture names; this field is intended to be provided via :ref:`task-configuration` * ``arch_all_build_architecture`` (string, defaults to ``amd64``): concrete architecture on which to run tasks for ``Architecture: all`` packages * ``enable_check_installability`` (boolean, defaults to True): whether to include installability-checking tasks * ``check_installability_suite`` (:ref:`lookup-single`, required if ``enable_check_installability`` is True): the :collection:`debian:suite` collection to check installability against; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so * ``enable_autopkgtest`` (boolean, defaults to True): whether to include autopkgtest tasks * ``autopkgtest_backend`` (string, optional): see :task:`Autopkgtest` * ``enable_reverse_dependencies_autopkgtest`` (boolean, defaults to False): whether to include autopkgtest tasks for reverse-dependencies * ``enable_lintian`` (boolean, defaults to True): whether to include lintian tasks * ``lintian_backend`` (string, optional): see :task:`Lintian` * ``lintian_fail_on_severity`` (string, optional): see :task:`Lintian` * ``enable_piuparts`` (boolean, defaults to True): whether to include piuparts tasks * ``piuparts_backend`` (string, optional): see :task:`Piuparts` * ``piuparts_environment`` (string, optional): the environment to run piuparts in * ``enable_debdiff`` (boolean, defaults to False): whether to include debdiff tasks for source and binary packages. Compares the supplied source package and the binary packages against the packages available in the distribution identified by ``qa_suite``. * ``enable_blhc`` (boolean, defaults to False): whether to include ``blhc`` tasks for the build logs * ``fail_on`` (string, optional): indicate the conditions to trigger a failure of the whole workflow. Allowed values are ``failure``, ``regression``, ``never``. With ``failure``, the workflow is marked as failed if one of the QA task fails. With ``regression``, the workflow fails only if one of the QA result is a regression compared to the former result. With ``never``, the workflow always succeeds. The default value is ``regression`` if ``enable_regression_tracking`` is True, otherwise it is ``failure``. The workflow computes dynamic metadata as: .. dynamic_data:: :method: debusine.server.workflows.qa::QAWorkflow.build_dynamic_data Any of the lookups in ``source_artifact`` or ``binary_artifacts`` may result in :bare-data:`promises `, and in that case the workflow adds corresponding dependencies. Binary promises must include an ``architecture`` field in their data. The effective set of architectures is ``{architectures}`` (defaulting to all architectures supported by this Debusine instance and the ``{vendor}:{codename}`` suite, plus ``all``), intersecting ``{architectures_allowlist}`` if set, and subtracting ``{architectures_denylist}`` if set. The workflow creates sub-workflows and tasks as follows, with substitutions based on its own task data: * if ``enable_check_installability`` is set, a single :task:`CheckInstallability`, with task data: * ``suite``: ``{check_installability_suite}`` * ``binary_artifacts``: the subset of the lookup in this workflow's ``binary_artifacts`` for each available architecture * if ``enable_autopkgtest`` is set, an :workflow:`autopkgtest` sub-workflow, with task data: * ``source_artifact``: ``{source_artifact}`` * ``binary_artifacts``: the subset of the lookup in this workflow's ``binary_artifacts`` for each of ``all`` and the concrete architecture in question that exist * ``vendor``: ``{vendor}`` * ``codename``: ``{codename}`` * ``backend``: ``{autopkgtest_backend}`` * ``architectures``: the effective set of architectures * ``arch_all_build_architecture``: ``{arch_all_build_architecture}`` * if ``enable_reverse_dependencies_autopkgtest`` is set, a :workflow:`reverse_dependencies_autopkgtest` sub-workflow, with task data: * ``source_artifact``: ``{source_artifact}`` * ``binary_artifacts``: the subset of the lookup in this workflow's ``binary_artifacts`` for each of ``all`` and the concrete architecture in question that exist * ``qa_suite``: ``{qa_suite}`` * ``vendor``: ``{vendor}`` * ``codename``: ``{codename}`` * ``backend``: ``{autopkgtest_backend}`` * ``architectures``: the effective set of architectures * ``arch_all_build_architecture``: ``{arch_all_build_architecture}`` * if ``enable_lintian`` is set, a :workflow:`lintian` sub-workflow, with task data: * ``source_artifact``: ``{source_artifact}`` * ``binary_artifacts``: the subset of the lookup in this workflow's ``binary_artifacts`` for each of ``all`` and the concrete architecture in question that exist * ``vendor``: ``{vendor}`` * ``codename``: ``{codename}`` * ``backend``: ``{lintian_backend}`` * ``architectures``: the effective set of architectures * ``arch_all_build_architecture``: ``{arch_all_build_architecture}`` * ``fail_on_severity``: ``{lintian_fail_on_severity}`` * if ``enable_piuparts`` is set, a :workflow:`piuparts` sub-workflow, with task data: * ``binary_artifacts``: the subset of the lookup in this workflow's ``binary_artifacts`` for each of ``all`` and the concrete architecture in question that exist * ``vendor``: ``{vendor}`` * ``codename``: ``{codename}`` * ``backend``: ``{piuparts_backend}`` * ``architectures``: the effective set of architectures * ``arch_all_build_architecture``: ``{arch_all_build_architecture}`` .. todo:: Not implemented: ``enable_check_installability`` and ``check_installability_suite``. Behavior with ``update_qa_results`` set to True ----------------------------------------------- When ``update_qa_results`` is set to True, the goal of the workflow is modified: its only purpose is to provide reference results to be stored in a :collection:`debian:qa-results` collection. Task failures are never fatal for the parent workflow or for dependent tasks. During orchestration, the workflow compares the data available in the :collection:`debian:qa-results` collection together with information about the submitted ``source_artifact`` and ``binary_artifacts``. When a missing or outdated QA result is detected, it schedules the appropriate QA task, and it creates a corresponding promise in the internal collection (the name of the promise is the prefix followed by the expected name of the collection entry). The QA task has the following event reactions: * ``on_assignment``: an action to :ref:`skip the work request ` if the latest relevant item in the :collection:`debian:qa-results` collection has changed since the work request has created; this avoids wasting resources if multiple parallel workflows trigger an update of the same QA results * ``on_success``: an action to add the result to the :collection:`debian:qa-results` collection * ``on_failure``: same as ``on_success`` Note that when ``enable_reverse_dependencies_autopkgtest`` is set to True, it must also update the autopkgtest results of the reverse dependencies and thus compute the same list of packages as the ``reverse_dependencies_autopkgtest`` workflow (using the same ``qa_suite`` collection). Behavior with ``enable_regression_tracking`` set to True -------------------------------------------------------- When ``enable_regression_tracking`` is set to True, the orchestrator of the ``qa`` workflow schedules :ref:`workflow callbacks ` that will perform the regression analysis. In order to wait for the availability of the QA result(s), those callbacks have dependencies against: * the promises associated to the QA result(s) that are required from the additional ``qa`` workflow building reference results * the promises associated to the QA result(s) that are required from the sub-workflows The ``workflow_data`` field for those workflow callbacks have: * ``visible`` set to False so that they do not show up in the workflow hierarchy (new feature to implement) * ``step`` set to ``regression-analysis`` As part of the callback, the analysis is performed and the result of the analysis is stored in the ``output_data`` field of the workflow. .. note:: We use simple workflow callbacks instead of full-fledged worker tasks or server tasks because we assume that regression analysis can be completed just by comparing the artifact metadata and/or the collection item. Workflow callbacks are already dealt through celery tasks so they are relatively cheap. Note however that the large number of callbacks requires use of careful locking to serialize the operations between concurrent runs trying to update the same workflow. Handling of ``fail_on`` ----------------------- With ``fail_on: never`` or ``fail_on: regression``, all the sub-workflows are run with ``workflow_data.allow_failure: true``. With ``fail_on: regression``, a final orchestrator callback is scheduled: * it depends on all the ``regression-analysis`` callbacks * ``workflow_data.visible`` is set to True * ``workflow_data.step`` is ``final-regression-analysis`` * ``workflow_data.display_name`` is ``Regression analysis`` The callback reviews the data in ``output_data.regression_analysis`` and sets its own result to FAILURE in case of regression, or SUCCESS otherwise.