Uploading packages
MakeSourcePackageUpload task
This worker task makes a debian:upload
artifact from a
debian:source-package
artifact. This involves unpacking the source
package and running dpkg-genchanges
on it.
The task_data
for this task may contain the following keys:
input
(required): a dictionary describing the input data:source_artifact
(Single lookup, required): adebian:source-package
artifact
since_version
(string, optional): include changelog information from all versions strictly later than this version in the.changes
file; the default is to include only the topmost changelog entrytarget_distribution
(string, optional): override the targetDistribution
field in the.changes
file to this value; the default is to use the distribution from the topmost changelog entryenvironment
(Single lookup with default categorydebian:environments
, required):debian:system-tarball
ordebian:system-image
artifact (as appropriate for the selecteddebian:environments
, required):debian:system-tarball
artifact that will be used to rundpkg-source
anddpkg-genchanges
using theunshare
backend.
The output is a debian:upload
artifact with extends
and
relates-to
relationships to the input source package artifact (to match
the behaviour of debusine import-debian-artifact
).
MergeUploads task
This worker task combines multiple debian:upload
artifacts into a single
one, in preparation for uploading them together. This involves running
mergechanges
on them, or the equivalent.
The task_data
for this task may contain the following keys:
input
(required): a dictionary describing the input data:uploads
(Multiple lookup, required): a list ofdebian:upload
artifacts
environment
(Single lookup with default categorydebian:environments
, required):debian:system-tarball
ordebian:system-image
artifact (as appropriate for the selecteddebian:environments
, required):debian:system-tarball
artifact that will be used to runmergechanges
using theunshare
backend.
The output is a debian:upload
artifact with extends
relationships to
each of the input upload artifacts.
Debsign task
This is a signing task that signs a debian:upload artifact on a signing worker. It is separate from the
Sign task because signing uploads is a customized operation involving
signing multiple files and possibly updating checksums in the .changes
file to match the signed versions of other files.
The task_data
for this task may contain the following keys:
unsigned
(Single lookup, required): thedebian:upload
artifact whose contents should be signedkey
(Single lookup, required): thedebusine:signing-key
artifact to sign the upload with, which must have purposeopenpgp
The output will be provided as a debian:upload
artifact, with relates-to
relations to the unsigned and key artifacts.
ExternalDebsign task
This wait task blocks until a user provides a signature for an upload.
The task_data
for this task may contain the following keys:
unsigned
(Single lookup, required): thedebian:upload
artifact whose contents should be signed
Running this task does not do anything, but the task data informs the client
that it needs to use the
/api/1.0/work-request/<int:work_request_id>/external-debsign/
view below
to record a debian:upload
artifact as an output of the task. The
containing workflow should normally then use an event reaction to add that
output to a suitable collection (usually the workflow’s internal
collection).
The task does not verify the signature, since it doesn’t necessarily have the public key available. It remains the responsibility of whatever would normally verify the signature (e.g. an external upload queue) to do so.
PackageUpload task
This worker task uploads Debian packages to an upload queue.
It is the equivalent of running dput
, but since other parts of debusine
ensure that the upload is well-formed, there’s no need for most of the
complexity of dput
and we can avoid needing an environment for it.
The task_data
for this task may contain the following keys:
input
(required): a dictionary describing the input data:upload
(Single lookup, required): adebian:upload
artifact
target
(required): the upload queue, as anftp://
orsftp://
URL
The implementation should take care to use a suitable connection timeout.
An SSH private key should be provided in the ~/.ssh/
directory of the
user running debusine.
Workflow package_upload
This workflow signs and uploads source and/or binary packages to an upload queue. It is normally expected to be used as a sub-workflow.
task_data
:source_artifact
(Single lookup, optional): adebian:source-package
ordebian:upload
artifact representing the source package (the former is used when the workflow is started based on a.dsc
rather than a.changes
)binary_artifacts
(Multiple lookup, optional): a list ofdebian:upload
artifacts representing the binary packagesmerge_uploads
(boolean, defaults to False): if True, merge the uploads and create a singlePackageUpload
task to upload them all together; if False, create a separatePackageUpload
task for each uploadsince_version
(string, optional): passed to MakeSourcePackageUpload task ifsource_artifact
is adebian:source-package
target_distribution
(string, optional): passed to MakeSourcePackageUpload task ifsource_artifact
is adebian:source-package
key
(Single lookup, optional): thedebusine:signing-key
artifact to sign the upload with, which must have purposeopenpgp
require_signature
(boolean, defaults to True): whether the upload must be signedtarget
(required): the upload queue, as anftp://
orsftp://
URL
At least one of source_artifact
and binary_artifacts
must be set.
The workflow creates the following tasks, each of which has a dependency on the previous one in sequence, using event reactions to store output in the workflow’s internal collection for use by later tasks:
if
source_artifact
is adebian:source-package
artifact: a MakeSourcePackageUpload task (withsince_version
andtarget_distribution
) to build a corresponding.changes
fileif
merge_uploads
is True and there is more than one source and/or binary artifact: a MergeUploads task to combine them into a single uploadfor each upload (or for the single merged upload, if merging):
if
key
is provided: a Debsign task to have debusine sign the upload with the given keyif
key
is not provided andrequire_signature
is True: an ExternalDebsign task to wait until a user provides a signature, which debusine will then include with the uploada PackageUpload task, to upload the result to the given upload queue
API changes
A new /api/1.0/work-request/<int:work_request_id>/external-debsign/
view
allows handling the new ExternalDebsign
wait task.
On GET
, the view returns a response including the unsigned
debian:upload
artifact ID. The client downloads the files from this
artifact that need to be signed (.changes
, plus whichever of .dsc
and .buildinfo
exist), runs debsign
on them, creates a new
debian:upload
artifact, and uploads the files to it. (This relies on
some changes to avoid needing to reupload unchanged files.)
It then sends a POST
request back to the view, including the signed
artifact ID.
On POST
, the view works as follows, in a transaction:
check that the work request is
WAIT/ExternalDebsign
and is running, and otherwise return HTTP 400check that the authenticated user is the same as the user who created the work request, and otherwise return HTTP 403
inspect the work request’s task data to find the unsigned artifact, and look up the signed artifact from the request data, returning HTTP 400 if either of these fails
check that the signed artifact is a
debian:upload
artifact, and otherwise return HTTP 400check that the signed artifact does not add any new files when compared to the unsigned artifact, that it replaces the
.changes
file, and that it only replaces files with permitted suffixes (.changes
,.dsc
, and.buildinfo
); otherwise return HTTP 400add a
relates-to
relation from the signed artifact to the unsigned artifactlink the signed artifact to the work request
mark the work request completed
return HTTP 200
UI changes
Wait tasks will generally require some form of UI change. In particular,
when showing a blocked ExternalDebsign
work request, the web UI advises
the user to run a new client command (e.g. debusine provide-signature
<work request ID>
, which downloads the unsigned upload, signs it, and
uploads it back to the external-debsign
API view above.
Pipeline considerations
The package_upload workflow will typically be used as a sub-workflow of a smart “pipeline” workflow. There are three main use cases:
Upload a source package to debusine, have it tested, and then have debusine pass on that upload to an external upload queue.
In this case, the
package_upload
workflow’s task data should setsource_artifact
to the source package and leavebinary_artifacts
empty.Upload a source package to debusine, have it tested, and then have debusine upload both the source and all built binaries to an external upload queue. (For example, this is useful when uploading a package to Debian that will land in the NEW queue, since Debian currently requires binaries for NEW uploads.)
In this case, the
package_upload
workflow’s task data should setsource_artifact
to the source package, setbinary_artifacts
to a list of single lookups matching each of the binary uploads from the super-workflow’s internal collection (e.g.[internal@collections/name:build-all, internal@collections/name:build-amd64]
, and setmerge_uploads
to True.debusine acts as a build daemon, building a source package for a number of architectures and uploading each of them as soon as the builds finish.
In this case, the
package_upload
workflow’s task data should leavesource_artifact
unset, setbinary_artifacts
to a list of single lookups matching each of the binary uploads from the super-workflow’s internal collection, and setmerge_uploads
to False.
Generic code for creating child work requests using artifacts from the workflow’s internal collection adds appropriate dependencies on the work requests that are expected to provide those artifacts.
If the parent workflow needs some kind of manual validation step to complete
before starting the upload (typically in the case of manual uploads but not
when acting as a build daemon), it should add a dependency from the
package_upload
sub-workflow to the validation workflow step. The
package_upload
sub-workflow will be populated before validation is
complete (since the root workflow handles population of all its
sub-workflows), but it will not start running until validation is complete.