More flexible workflow templates

Possible future work

Nested structures

Counterintuitively, only top-level keys are considered while merging workflow template and user data: lists and dictionaries are not merged, and if a list or a dictionary exists in both the workflow template and the user data, the version in the user data is ignored. Most of our current workflows have been designed to take this into account, but that requires a flat structure for workflow parameters that can be a little overwhelming.

However, changing this would have other implications. dput-ng profiles have similar nesting limitations, which caused us to unnest Debusine-specific entries there in !1769, and adding nesting within debusine_workflow_data would reintroduce that difficulty.

Also, the current flat structure suggests the possibility of a web UI to start a workflow that offers each of the available parameters as options and introspects the corresponding Pydantic models to pick the appropriate widgets; it becomes less obvious how to do this if we increase nesting.

Additional variable templating

Some workflow task data entries are naturally constructed from other entries: for example, collection names can be constructed based on vendor/codename. A possible direction would be to add a template language that can build template data based on some kind of variable expansion pass, in order that only a minimal set of variables needs to be presented as options to the user.

However, adding templates to generate templates carries intrinsic risk of confusion, and many of the same benefits can be gained by more thoughtful workflow design: workflow orchestrators are free to do their own computations when building values for child work requests, and they can do so in ways that would be difficult to express in a template language, such as looking up state in the database. For now, it seems best to defer this until (if ever) we can design a domain-specific language powerful enough to express everything we do in workflow orchestrators, at which point something like this might be a good fit for it.

autopkgtest parameter choice restrictions

A workspace administrator might want to allow setting autopkgtest_backend to unshare or incus-lxc but not to incus-vm or qemu, for resource consumption reasons.

However, this particular case cannot be handled solely at the workflow template level. autopkgtest_backend: auto is currently synonymous with autopkgtest_backend: unshare, but for #803 it would be useful to have it be sensitive to whether the autopkgtest in question declares the isolation-container or isolation-machine restriction and use the simplest possible backend that will work.

To make that work, the task itself would need to be responsible for deciding what auto means. Merging template and user task data happens before the task runs, so neither new WorkflowTemplate fields nor the current task configuration mechanism can easily express this restriction.

A new allow_backends parameter seems tempting, but there are some problems. The configuration context for the Autopkgtest task is currently just the codename, so we can’t use that for architecture-specific configuration and would need to complicate the task with explicit by-architecture parameters. This may be a reason to add multiple configuration contexts.

We also need to handle this sort of backend selection at the workflow level rather than the task level, since the decision of which worker to dispatch the task to may depend on which backend is selected. That introduces the difficulty that the workflow doesn’t currently have access to the Restrictions declared by a given source package’s tests, and that information isn’t in the .dsc; it would have to unpack the source package to find those, which would have to be done on a worker since running dpkg-source -x on untrusted source packages isn’t safe, and then cache the result somewhere.