Debusine concepts¶
Artifacts¶
Artifacts are at the heart of Debusine. Artifacts are both inputs (submitted by users) and outputs (generated by tasks). An artifact combines:
an arbitrary set of files
arbitrary key-value data (stored as a JSON-encoded dictionary)
a type
The type is just a string identifier used to recognize artifacts sharing the same structure. You can create and use types as you see fit but we have defined a basic ontology suited for the case of a Debian-based distribution.
Artifacts can have relations with other artifacts:
built-using: indicates that the build of the artifact used the target artifact (ex: “binary-packages” artefacts are built using “source-package” artefacts)
extends: indicates that the artifact is extending the target artifact in some way (ex: a “source-upload” artifact extends a “source-package” artifact with target distribution information)
relates-to: indicates that the artifact relates to another one in some way (ex: a “binary-upload” artifact relates-to a “binary-package”, or a “package-build-log” artifact relates to a “binary-package”).
Artifacts are not deleted:
as long as they are referenced by another artifact (through one of the above relationships)
as long as their expiration date is not over
as long as they are not manually deleted (if they don’t have any expiration date)
Artifacts can have additional properties:
immutable: when set to True, nothing can be changed in the artifact through the API
creation timestamp: timestamp indicating when the artifact has been created
last updated timestamp: timestamp indicating when the artifact has been last modified/updated
The following operations are possible on artifacts:
create a new artifact
upload content of one of its file
set key-value data
attach/remove a file
add/remove a relationship
delete an artifact
Workspaces¶
A Workspace is a concept tying together a set of Artifacts and a set of Users. Since Artifacts have to be stored somewhere, Workspaces also tie together the set of FileStore where files can be stored.
Workspaces have the following important properties:
public: a boolean which indicates whether the Artifacts are publicly accessible or if they are restricted to the users belonging to the workspace
default_expiration_delay: the minimal time (in seconds) that a new artifact is kept in the workspace before being expired. This value is only used when the Artifact is initially created. It can be manually modified afterwards. If this value is None, then Artifacts are never expired until they are manually removed.
default_file_store: the default FileStore where newly uploaded files are stored.
Workspaces are also the entities upon which access control rules are built. Each workspace has a a set of users that can have 3 different levels of access:
read-only access: can access all objects within the workspace but not make any change
upload-only access: same as read-only but can create new artifacts and can modify their own artifacts.
read-write access: can access all objects within the workspace and modify them, even those created by others.
admin access: same as read-write and can also add/remove users to the workspace, and change generic properties of the workspace itself
Tasks¶
Tasks are time-consuming operations that are typically offloaded to dedicated workers. They consume artifacts as input and generate artifacts as output. The generated artifacts automatically have built-using relationships linking to the artifacts used as input.
Tasks can require specific features from the workers on which it will run. This will be used to ensure things like:
architecture selection (when managing builders on different architectures)
required memory amount
required free disk space amount
availability of specific build chroot
Tasks are required to use the public API to interact with artifacts. They are passed a dedicated token that has the proper permissions to retrieve the required artifacts and to upload the generated artifacts.
Workflows¶
Workflows are advanced server-side logic entirely driven by code. They can trigger tasks, analyze their results, and use the API to create/modify artifacts. They often have associated web interface for users to inspect the results and/or to provide inputs in some of the steps.
Workflows can be started by users or external events, through the web interface or through the API.
Workflows can only be started if they have been registered by an admin in a workspace. This process:
grants a unique name to the workflow so that it can be easily identified and started by users
defines all the input parameters that will not change between runs of the registered workflow
The input parameters that are not set during registration are called run-time parameters and they have to be provided by the user that starts the workflow. Those parameters are stored in a WorkflowInstance model that will be used during the whole duration of the process controlled by the workflow.
Workflow instances have the following properties:
status: running / aborted / completed
result: success / failure / error / neutral
parameters: dictionary of run-time parameters
Once completed, the remaining lifetime of the workflow instances is controlled by their expiration date and/or they can be tied to the existence of some associated artifact.