Advanced workflows / sub-workflows

Advanced workflows can be created by combining multiple limited-purpose workflows.

Sub-workflows are integrated in the general graph of their parent workflow as WorkRequests of type workflow.

From a user interface perspective, sub-workflows are typically hidden as a single step in the visual representation of the parent’s workflow.

Cooperation between workflows is defined at the level of workflows. Individual work requests should not concern themselves with this; they are designed to take inputs using lookups and produce output artifacts that are linked to the work request.

Sub-workflow coordination takes place through the workflow’s internal collection (which is shared among all sub-workflows of the same root workflow), providing a mechanism for some work requests to declare that they will provide certain kinds of artifacts which may then be required by work requests in other sub-workflows.

On the providing side, workflows use the update-collection-with-artifacts event reaction to add relevant output artifacts from work requests to the internal collection, and create promises to indicate to other workflows that they have done so. Providing workflows choose item names in the internal collection; it is the responsibility of workflow designers to ensure that they do not clash, and workflows that provide output artifacts have a optional prefix field in their task data to allow multiple instances of the same workflow to cooperate under the same root workflow.

On the requiring side, workflows look up the names of artifacts they require in the internal collection; each of those lookups may return nothing, or a promise including a work request ID, or an artifact that already exists, and they may use that to determine which child work requests they create. They use lookups in their child work requests to refer to items in the internal collection (e.g. internal@collections/name:build-amd64), and add corresponding dependencies on work requests that promise to provide those items.

Sub-workflows may depend on other steps within the root workflow while still being fully populated in advance of being able to run. A workflow that needs more information before being able to populate child work requests should use workflow callbacks to run the workflow orchestrator again when it is ready. (For example, a workflow that creates a source package and then builds it may not know which work requests it needs to create until it has created the source package and can look at its Architecture field.)