Manage task configuration

See Task configuration for the basic concepts.

Introduction

More concretely when applying task configuration, Debusine will retrieve a sorted list of debusine:task-configuration entries matching the work request to configure and apply the instructions contained there.

The matching criteria are:

the task type (required)
the task name (required)
the “subject” of the task
the “configuration context” of the task
the scheduler tags

Note

Subject, configuration context and scheduler tags are data that are dynamically computed by each task. You can review the generated data in the “Internals” tab of each work request. While the scheduler tags have their own dedicated section, the subject and context are part of “Dynamic data” section and can be found under the “subject” and “configuration_context” field respectively (beware to not mix with the “runtime_context” which is used for managing task history).

The possible operations are:

setting a default value to a field
overriding the value of a field
deleting the value of a field
locking the value of a field (forbidding further overrides)
requiring additional scheduler tags

Editing through text files

Debusine allows maintaining the debusine:task-configuration collection in text files in a local directory, preferably a git repository.

To initialize a task configuration git repository, follow these steps. If you wish, you can follow the same process without using git, just creating an empty directory to work in, and skipping the git steps.

Create a new git repository:

$ git init task-config
$ cd task-config

Pull the current collection contents into the repository:

$ debusine --server NAME task-config pull --workspace WORKSPACE
0 added, 0 updated, 0 deleted, 0 unchanged

This will create a MANIFEST file, and YAML files for any existing configuration. The workspace name is saved in this file, so we won’t need to specify it again, when working with this repository.

Commit all of this to git:

$ git add .
$ git commit -m "Initialize workspace configuration repository"

Add configuration by making new YAML files, or editing existing files.
Commit the changes.

Push them back to the configuration collection:

$ debusine --server NAME task-config push
Pushing data to server...
0 added, 0 updated, 0 removed, 1 unchanged

Push your git repo to somewhere that’s shared with the other workspace admins.

The client will protect you from overwriting configuration if the current server configuration is not in your git repository’s history:

$ debusine --server NAME task-config push
server collection was pushed from commit 148a8269de622667b7635d63c505b15a9ab8381d which is not known to ~/task-config

If you need to override this (maybe because you rewrote git history), you can:

$ debusine --server NAME task-config push --force
server collection was pushed from commit 148a8269de622667b7635d63c505b15a9ab8381d which is not known to ~/task-config
Pushing data to server...
0 added, 0 updated, 0 removed, 1 unchanged

Naming of files

debusine:task-configuration entries pulled from the server are stored as YAML files. The file names are ignored when matching remote entries with local ones, so you can rename files and organize them into directories as you find it convenient.

Multiple rules can be contained in a single file, by making a YAML list of rules in one file.

When new entries are found on the server during a pull, they are stored inside a new/ directory from which they can be safely dispatched to a more meaningful location in the checkout.

When pulling changes on existing files, debusine is able to locate the local files according to their contents and update them in place.

Git and non-git

If the local checkout is in git:

Push and pull refuse to work if there are uncommitted changes to .yaml files.
If an entry has been deleted on the server, it will also be deleted locally.
Push will refuse to work if the server contents were pushed from git, and their commit hash is not an ancestor of the local git commit

If the local checkout is not in git:

If an entry has been deleted on the server, it will not be deleted locally to prevent accidental data loss, although it will be reported in a warning.

You can use --force to bypass the checks above.

Dry run

If using --dry-run, push and pull operations will go through all the motions, including download and upload of data and reporting of change statistics, but changes are not stored in the database.

Example task configuration files

YAML files in the checkout are the debusine:task-configuration serialized to YAML (or list of those).

Here are a few examples of task configuration items. You can find additional real-word examples in the debusine.debian.net configuration repository.

For simplicity, all the examples below have all the content in a single file, but you can split the various entries across multiple files as you see fit, in particular to better control the order in which they are processed (see Ordering matching entries) since when there’s overlap, the order is particularly important.

Example 1: overriding a value for a field

Here’s how you would override sbuild’s build_options to parallel=2 when it is building the openjdk-17 source package. Note that if the parent workflow starting the task is defining a value for the build_options field, that value would not be used, it is replaced by the value provided through this mechanism.

task_type: Worker
task_name: sbuild
tags_provided:
  - task:source-package:openjdk-17
override_values:
  build_options: parallel=2
comment: reduce parallelism to avoid crashing workers

Note that we rely on the scheduler tags here to filter the source package, but we could have use the subject field too (subject: openjdk-17) since many tasks use the source package as subject.

You can add additional tags in tags_provided to restrict even further the work requests where it would be applied. Debusine generate multiple tags to let you match work requests by scope, workspace, user, groups of the user, etc. See Tags provided by tasks.

Example 2: setting a default value for a field

With the entries below, lintian tasks will be provided a default value for the exclude_tags field except when it is run in the “sid” distribution (where delete_values effectively cancels the former entry).

But if the user starting the workflow provided a value of his own, or if the workflow template already provided a default value, then that value would prevail over the value provided through the task configuration mechanism.

- task_type: Worker
  task_name: lintian
  default_values:
    exclude_tags:
      - source-nmu-has-incorrect-version-number
      - older-source-format
  comment: By default for stable releases, ignore some tags.

- task_type: Worker
  task_name: lintian
  context: sid
  delete_values:
    - exclude_tags
  comment: For sid/unstable, we don't want to ignore tags.

Example 3: locking a value when overlap is expected

Imagine that openjdk-17 needs a custom build_profiles value and you want to ensure that further task configuration (for example one applying to all packages on a given architecture) will not overwrite that value. You can do that by using lock_values in addition the default_values or override_values.

- task_type: Worker
  task_name: sbuild
  tags_provided:
    - task:source-package:openjdk-17
  override_values:
    build_profiles:
      - nodoc
  lock_values:
    - build_profiles
  comment: we really need nodoc profile or it fails to build see bug 312

- task_type: Worker
  task_name: sbuild
  tags_required:
    - worker:build-arch:riscv64
  override_values:
    build_profiles:
      - nocheck
  comment: Skip checks while we're bootstrapping the architecture

Note that those two task configuration entries would likely be stored in two different files in practice. The net result is that any build of openjdk-17 will always use the “nodoc” build profile, and any riscv64 build (except for openjdk-17) will use the “nocheck” build profile.

Without lock_values, a risc64 build of openjdk-17 would also use the “nocheck” build profile because the latter entry would override the change made by the first entry.

Example 5: require an additional tag from the worker

The example below introduces the require_tags field which lets you specify additional scheduler tags that the worker must have. Let’s build a list of source packages where sbuild and autopkgtest tasks will run on workers that have been marked by the administrator as being “large”.

- template: large-worker
  require_tags:
    - worker:class:large

- task_type: Worker
  task_name: sbuild
  subject: firefox
  use_templates:
    - large-worker

- task_type: Worker
  task_name: autopkgtest
  subject: firefox
  use_templates:
    - large-worker

[ ... repeat for other source packages ... ]