Build instructions

This page discusses various ways to control the package building process in debusine. The idea is to achieve at a minimum some level of parity with features proposed by buildd and wanna-build. But ideally we want to go further than that.

Many of the features rely on the management of a collection of Task configuration. We refer to this as the task-configuration mechanism in the text below.

Desired features

Control of architectures

We want to be able to control the set of architectures being built for a package.

This can be handled through architectures_allowlist and architectures_denylist options on the debian-pipeline workflow and the task-configuration mechanism.

Host architecture for arch-all builds

We want to be able to control the architecture of the worker used to build Architecture: all packages. Currently, amd64 is hardcoded.

At some point, we will want the packages to be able to provide the value directly. But for now, we want to be able to control that architecture through the task-configuration mechanism.

The sbuild task doesn’t require any change. We can already feed the architecture of our choice in host_architecture and set build_components to all.

Thus it’s at the level of the debian-pipeline and sbuild workflows that we need a new arch_all_host_architecture field.

Control the build backend and the associated build environment

We want to be able to control the build backend used by the worker, and possibly also use an alternative build environment.

The debian-pipeline and sbuild workflows respectively have the sbuild_backend and backend options already. The sbuild workflow also has an environment_variant option to select a specific variant of the build environment. All those can be controlled through the task-configuration mechanism.

We should add the same parameter at the debian-pipeline level for convenience.

Control DEB_BUILD_PROFILES and DEB_BUILD_OPTIONS

Those are very specific options that do not deserve to be controlled from the debian-pipeline workflow level. However they should be usable at the sbuild workflow and worker task level, so that the task-configuration mechanism can be used to control them.

The sbuild workflow supports build_profiles but not build_options. The sbuild task supports both parameters, but doesn’t implement the build_options one. build_profiles is defined as a list, but build_options as a string.

We should harmonize all this.

Control workers assigned for specific packages/tasks

Some packages have specific requirements and are known to fail on workers that do not match those requirements. Typically small workers are not able to build some heavy packages. Or some packages require a GPU for their tests.

The current plan to address this is to implement a tag-based approach.

To complement this, we need some way to define some specific requirements while submitting the worker tasks. The proposal is to add a new worker_requirements field in task_data. That way users that can run individual work requests are able to specify their requirements, and for tasks run through a workflow, it can be controlled through the task-configuration mechanism.

Note

This design offers no way to merge user submitted requirements with the requirements provided through task-configuration.

Note

This design offers no way to merge worker requirements coming from different “configuration contexts”. One can override lists from one level to the next, but not expand them.

Control packages tested during autopkgtest of reverse dependencies

We want to be able to restrict the set of autopkgtests which are run as part of the reverse_dependencies_autopkgtest workflow:

• Some packages have a very large dependency tree, and we might not want to execute all of them. Instead we might want to rely on a representative subset.

• On the opposite, some packages might have autopkgtest that are resource hungry and that we’d rather not run as part of such a workflow (the tests would only run when the package itself gets updated, not when reverse dependencies get updated).

Both of those could be achieved with the task-configuration mechanism if the workflow is extended with two new parameters — packages_allowlist and packages_denylist — listing respectively the set of source package names whose tests can be scheduled and the set of source package names whose tests should not be scheduled.

Proposed changes

Changes to the sbuild workflow

• Addition of the build_options option. Must be passed down to the sbuild task.

Changes to the sbuild task

• Change the schema of the build_options parameter to be a list like build_profiles instead of a string.

• Implement support of the build_options parameter by setting the DEB_BUILD_OPTIONS environment variable before calling sbuild. Note that when the build_profiles parameter is also provided and contains either nodoc or nocheck, those values might be automatically added to DEB_BUILD_OPTIONS on top of the user-provided value.

Changes related to the worker requirements feature

• Add a new optional worker_requirements field in BaseTaskData as a list of strings.

• At some point when #326 gets implemented:

  • add a new method compute_worker_requirements() that can generate additional requirements based on the analysis of the task parameters (and of the available history if any)

  • expand configure_on_server() to transform values coming from the worker_requirements task_data field and from the new compute_worker_requirements() method into real relations with the new WorkerTag model (while respecting the various restrictions).