Regression tracking

Regression tracking helps distinguish new QA failures introduced by a package from failures that already exist in the distribution.

QA workflows such as autopkgtest, lintian, piuparts, and blhc may report failures that are unrelated to the most recent change to the package under test. For example, reverse-dependency autopkgtests may already be failing, or a package may already emit a given set of lintian tags. Looking only at the latest QA results does not show whether those failures are new or whether they were already present before the package was tested.

To address this, Debusine maintains a collection of reference QA results for a distribution. When regression tracking is enabled, new QA runs are compared against those reference results to determine whether they represent a regression, an improvement, or no meaningful change.

Reference QA results

Regression tracking relies on a set of reference QA results that represent the current state of a distribution. They are stored in a debian:qa-results collection and provide the basis for comparison with subsequent QA runs.

Reference QA results are maintained independently of the package under test. Rather than only recording whether a QA task succeeded or failed, they provide the reference state needed to determine whether a new result represents a change in the distribution.

Each reference result corresponds to a particular QA workflow, package, and architecture. When regression tracking is enabled, Debusine locates the appropriate reference results and compares them with the results produced by the current QA run.

Reference tests

Reference QA results are produced by running the same QA workflows that are used to test packages, but against the reference suite rather than the package under test. This ensures that both reference and new results are generated under comparable conditions.

When regression tracking requires reference results that are missing or no longer current, Debusine schedules reference tests to populate or refresh the debian:qa-results collection. Existing reference results may be reused when they are still current, avoiding unnecessary work.

By using the same QA workflows for both reference and package-specific runs, regression tracking can compare equivalent results while avoiding separate implementations for reference testing.

Regression analysis

Once both reference and new QA results are available, Debusine compares them to determine whether the package under test has changed the outcome of the QA checks.

Each QA workflow performs a comparison appropriate for the type of results it produces. For example, workflows may compare test outcomes or the reported QA findings, identifying whether the new results represent a regression, an improvement, or no meaningful change.

Each comparison produces a regression analysis for the corresponding QA workflow. The qa workflow then combines those individual analyses into an overall assessment, providing a single view of the package’s QA status while preserving the individual results for each workflow.

Supported workflows

Regression tracking is integrated into Debusine’s QA workflows. Each supported workflow performs the comparison appropriate for its own QA results while sharing the same regression tracking mechanism.

The following QA workflows currently support regression tracking:

  • autopkgtest compares autopkgtest outcomes against the corresponding reference test results.

  • lintian compares the QA findings reported by lintian against the reference results.

  • piuparts compares package installation and upgrade test results with the reference state.

  • blhc compares build log hardening analysis against the reference results.

Higher-level workflows, such as qa and debian_pipeline, orchestrate these QA workflows and present the combined regression analysis as part of the overall QA results.

Configuration

Regression tracking is configured through the QA workflows that support it. Higher-level workflows typically manage the underlying regression tracking configuration automatically, while lower-level workflows expose additional options for controlling reference QA results when needed.

The qa_suite parameter specifies the distribution used for reference tests, while reference_qa_results identifies the debian:qa-results collection used to store and retrieve reference QA results.

The enable_regression_tracking and update_qa_results parameters control whether regression analysis is performed and whether reference QA results are created or refreshed. Their exact behaviour depends on the workflow being used.

For the available user-facing configuration options and their detailed behaviour, see the documentation for debian_pipeline.