.. _regression-tracking:

====================
Regression tracking
====================

Regression tracking helps distinguish new QA failures introduced by a package
from failures that already exist in the distribution.

QA workflows such as :workflow:`autopkgtest`, :workflow:`lintian`,
:workflow:`piuparts`, and :workflow:`blhc` may report failures that are
unrelated to the most recent change to the package under test. For example,
reverse-dependency autopkgtests may already be failing, or a package may
already emit a given set of ``lintian`` tags. Looking only at the latest QA
results does not show whether those failures are new or whether they were
already present before the package was tested.

To address this, Debusine maintains a collection of reference QA results for
a distribution. When regression tracking is enabled, new QA runs are compared
against those reference results to determine whether they represent a
regression, an improvement, or no meaningful change.

Reference QA results
====================

Regression tracking relies on a set of reference QA results that represent
the current state of a distribution. They are stored in a
:collection:`debian:qa-results` collection and provide the basis for
comparison with subsequent QA runs.

Reference QA results are maintained independently of the package under test.
Rather than only recording whether a QA task succeeded or failed, they
provide the reference state needed to determine whether a new result
represents a change in the distribution.

Each reference result corresponds to a particular QA workflow, package, and
architecture. When regression tracking is enabled, Debusine locates the
appropriate reference results and compares them with the results produced by
the current QA run.

Reference tests
===============

Reference QA results are produced by running the same QA workflows that are
used to test packages, but against the reference suite rather than the
package under test. This ensures that both reference and new results are
generated under comparable conditions.

When regression tracking requires reference results that are missing or no
longer current, Debusine schedules reference tests to populate or refresh the
:collection:`debian:qa-results` collection. Existing reference results may be
reused when they are still current, avoiding unnecessary work.

By using the same QA workflows for both reference and package-specific runs,
regression tracking can compare equivalent results while avoiding separate
implementations for reference testing.

Regression analysis
===================

Once both reference and new QA results are available, Debusine compares them
to determine whether the package under test has changed the outcome of the
QA checks.

Each QA workflow performs a comparison appropriate for the type of results it
produces. For example, workflows may compare test outcomes or the reported QA
findings, identifying whether the new results represent a regression, an
improvement, or no meaningful change.

Each comparison produces a regression analysis for the corresponding QA
workflow. The :workflow:`qa` workflow then combines those individual analyses
into an overall assessment, providing a single view of the package's QA status
while preserving the individual results for each workflow.

Supported workflows
===================

Regression tracking is integrated into Debusine's QA workflows. Each
supported workflow performs the comparison appropriate for its own QA results
while sharing the same regression tracking mechanism.

The following QA workflows currently support regression tracking:

* :workflow:`autopkgtest` compares autopkgtest outcomes against the
  corresponding reference test results.
* :workflow:`lintian` compares the QA findings reported by ``lintian`` against
  the reference results.
* :workflow:`piuparts` compares package installation and upgrade test results
  with the reference state.
* :workflow:`blhc` compares build log hardening analysis against the reference
  results.

Higher-level workflows, such as :workflow:`qa` and
:workflow:`debian_pipeline`, orchestrate these QA workflows and present the
combined regression analysis as part of the overall QA results.

Configuration
=============

Regression tracking is configured through the QA workflows that support it.
Higher-level workflows typically manage the underlying regression tracking
configuration automatically, while lower-level workflows expose additional
options for controlling reference QA results when needed.

The ``qa_suite`` parameter specifies the distribution used for reference
tests, while ``reference_qa_results`` identifies the
:collection:`debian:qa-results` collection used to store and retrieve
reference QA results.

The ``enable_regression_tracking`` and ``update_qa_results`` parameters
control whether regression analysis is performed and whether reference QA
results are created or refreshed. Their exact behaviour depends on the
workflow being used.

For the available user-facing configuration options and their detailed
behaviour, see the documentation for :workflow:`debian_pipeline`.