qa

Workflow qa

• task_data:

  • prefix (string, optional): prefix this string to the item names provided in the internal collection

  • reference_prefix (string, optional unless enable_regression_tracking is set): prefix for the item names provided in the internal collection in the corresponding workflow run for reference tests

  • source_artifact (Single lookup, required): the debian:source-package or debian:upload artifact representing the source package to test

  • binary_artifacts (Multiple lookup, required): the debian:binary-package, debian:binary-packages, or debian:upload artifacts representing the binary packages to test

  • package_build_logs (Multiple lookup, optional): the debian:package-build-log artifacts representing the build logs. Required if enable_blhc is True

  • qa_suite (Single lookup, optional unless enable_regression_tracking, update_qa_results, or enable_debdiff is True): the debian:suite collection that reference tests are being run against to detect regressions or the collection to search for debdiff original packages

  • reference_qa_results (Single lookup, optional unless enable_regression_tracking or update_qa_results is True): the debian:qa-results collection that contains the reference results of QA tasks to use to detect regressions

  • enable_regression_tracking (boolean, defaults to False): configure the workflow to detect and display regressions in QA results

  • update_qa_results (boolean, defaults to False): when set to True, the workflow runs QA tasks and updates the collection passed in reference_qa_results with the results.

  • vendor (string, required): the distribution vendor on which to run tests

  • codename (string, required): the distribution codename on which to run tests

  • extra_repositories (optional): see PackageBuild

  • architectures (list of strings, optional): if set, only run on any of these architecture names

  • architectures_allowlist (list of strings, optional, either concrete architecture names or all): if set, only run on any of these architecture names; while architectures is intended to be supplied by users or passed down from a higher-level workflow, this field is intended to be provided via Task configuration

  • architectures_denylist (list of strings, optional, either concrete architecture names or all): if set, do not run on any of these architecture names; this field is intended to be provided via Task configuration

  • arch_all_host_architecture (string, defaults to amd64): concrete architecture on which to run tasks for Architecture: all packages

  • qa_suite (Single lookup, required if enable_reverse_dependencies_autopkgtest is True): the debian:suite collection to search for reverse-dependencies; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

  • enable_check_installability (boolean, defaults to True): whether to include installability-checking tasks

  • check_installability_suite (Single lookup, required if enable_check_installability is True): the debian:suite collection to check installability against; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

  • enable_autopkgtest (boolean, defaults to True): whether to include autopkgtest tasks

  • autopkgtest_backend (string, optional): see Autopkgtest

  • enable_reverse_dependencies_autopkgtest (boolean, defaults to False): whether to include autopkgtest tasks for reverse-dependencies

  • enable_lintian (boolean, defaults to True): whether to include lintian tasks

  • lintian_backend (string, optional): see Lintian

  • lintian_fail_on_severity (string, optional): see Lintian

  • enable_piuparts (boolean, defaults to True): whether to include piuparts tasks

  • piuparts_backend (string, optional): see Piuparts

  • piuparts_environment (string, optional): the environment to run piuparts in

  • enable_debdiff (boolean, defaults to False): whether to include debdiff tasks for source and binary packages. Compares the supplied source package and the binary packages against the packages available in the distribution identified by qa_suite.

  • enable_blhc (boolean, defaults to False): whether to include blhc tasks for the build logs

  • fail_on (string, optional): indicate the conditions to trigger a failure of the whole workflow. Allowed values are failure, regression, never. With failure, the workflow is marked as failed if one of the QA task fails. With regression, the workflow fails only if one of the QA result is a regression compared to the former result. With never, the workflow always succeeds. The default value is regression if enable_regression_tracking is True, otherwise it is failure.

The workflow computes dynamic metadata as:

• subject: package name of source_artifact

Any of the lookups in source_artifact or binary_artifacts may result in promises, and in that case the workflow adds corresponding dependencies. Binary promises must include an architecture field in their data.

The effective set of architectures is {architectures} (defaulting to all architectures supported by this Debusine instance and the {vendor}:{codename} suite, plus all), intersecting {architectures_allowlist} if set, and subtracting {architectures_denylist} if set.

The workflow creates sub-workflows and tasks as follows, with substitutions based on its own task data:

• if enable_check_installability is set, a single CheckInstallability, with task data:

  • suite: {check_installability_suite}

  • binary_artifacts: the subset of the lookup in this workflow’s binary_artifacts for each available architecture

• if enable_autopkgtest is set, an autopkgtest sub-workflow, with task data:

  • source_artifact: {source_artifact}

  • binary_artifacts: the subset of the lookup in this workflow’s binary_artifacts for each of all and the concrete architecture in question that exist

  • vendor: {vendor}

  • codename: {codename}

  • backend: {autopkgtest_backend}

  • architectures: the effective set of architectures

  • arch_all_host_architecture: {arch_all_host_architecture}

• if enable_reverse_dependencies_autopkgtest is set, a reverse_dependencies_autopkgtest sub-workflow, with task data:

  • source_artifact: {source_artifact}

  • binary_artifacts: the subset of the lookup in this workflow’s binary_artifacts for each of all and the concrete architecture in question that exist

  • qa_suite: {qa_suite}

  • vendor: {vendor}

  • codename: {codename}

  • backend: {autopkgtest_backend}

  • architectures: the effective set of architectures

  • arch_all_host_architecture: {arch_all_host_architecture}

• if enable_lintian is set, a lintian sub-workflow, with task data:

  • source_artifact: {source_artifact}

  • binary_artifacts: the subset of the lookup in this workflow’s binary_artifacts for each of all and the concrete architecture in question that exist

  • vendor: {vendor}

  • codename: {codename}

  • backend: {lintian_backend}

  • architectures: the effective set of architectures

  • arch_all_host_architecture: {arch_all_host_architecture}

  • fail_on_severity: {lintian_fail_on_severity}

• if enable_piuparts is set, a piuparts sub-workflow, with task data:

  • binary_artifacts: the subset of the lookup in this workflow’s binary_artifacts for each of all and the concrete architecture in question that exist

  • vendor: {vendor}

  • codename: {codename}

  • backend: {piuparts_backend}

  • architectures: the effective set of architectures

  • arch_all_host_architecture: {arch_all_host_architecture}

Todo

Not implemented: enable_check_installability and check_installability_suite.

Behavior with update_qa_results set to True

When update_qa_results is set to True, the goal of the workflow is modified: its only purpose is to provide reference results to be stored in a debian:qa-results collection. Task failures are never fatal for the parent workflow or for dependent tasks.

During orchestration, the workflow compares the data available in the debian:qa-results collection together with information about the submitted source_artifact and binary_artifacts.

When a missing or outdated QA result is detected, it schedules the appropriate QA task, and it creates a corresponding promise in the internal collection (the name of the promise is the prefix followed by the expected name of the collection entry). The QA task has the following event reactions:

• on_assignment: an action to skip the work request if the latest relevant item in the debian:qa-results collection has changed since the work request has created; this avoids wasting resources if multiple parallel workflows trigger an update of the same QA results

• on_success: an action to add the result to the debian:qa-results collection

• on_failure: same as on_success

Note that when enable_reverse_dependencies_autopkgtest is set to True, it must also update the autopkgtest results of the reverse dependencies and thus compute the same list of packages as the reverse_dependencies_autopkgtest workflow (using the same qa_suite collection).

Behavior with enable_regression_tracking set to True

When enable_regression_tracking is set to True, the orchestrator of the qa workflow schedules workflow callbacks that will perform the regression analysis. In order to wait for the availability of the QA result(s), those callbacks have dependencies against:

• the promises associated to the QA result(s) that are required from the additional qa workflow building reference results

• the promises associated to the QA result(s) that are required from the sub-workflows

The workflow_data field for those workflow callbacks have:

• visible set to False so that they do not show up in the workflow hierarchy (new feature to implement)

• step set to regression-analysis

As part of the callback, the analysis is performed and the result of the analysis is stored in the output_data field of the workflow.

Note

We use simple workflow callbacks instead of full-fledged worker tasks or server tasks because we assume that regression analysis can be completed just by comparing the artifact metadata and/or the collection item. Workflow callbacks are already dealt through celery tasks so they are relatively cheap. Note however that the large number of callbacks requires use of careful locking to serialize the operations between concurrent runs trying to update the same workflow.

Handling of fail_on

With fail_on: never or fail_on: regression, all the sub-workflows are run with workflow_data.allow_failure: true.

With fail_on: regression, a final orchestrator callback is scheduled:

• it depends on all the regression-analysis callbacks

• workflow_data.visible is set to True

• workflow_data.step is final-regression-analysis

• workflow_data.display_name is Regression analysis

The callback reviews the data in output_data.regression_analysis and sets its own result to FAILURE in case of regression, or SUCCESS otherwise.