File store policies

File stores have some policies that apply to the store as a whole, which can be set using debusine-admin file_store create:

instance_wide (boolean, defaults to True): If True, this store can be used by any scope on this debusine instance. If False, it may only be used by a single scope.
soft_max_size (integer, optional): A soft limit in bytes for the total capacity of the store. This may be exceeded temporarily during uploads; debusine-admin vacuum_storage will move files from least-recently-used artifacts to another file store to get back below the limit.
max_size (integer, optional): A hard limit in bytes for the total capacity of the store. This may not be exceeded even temporarily during uploads.

There are also policies that apply to an individual scope’s use of a store, which can be set using debusine-manage scope add_file_store and debusine-manage scope edit_file_store:

upload_priority (integer, optional): The priority of this store for the purpose of storing new files. When adding a new file, debusine tries stores whose policies allow adding new files in descending order of upload priority, counting null as the lowest.
download_priority (integer, optional): The priority of this store for the purpose of serving files to clients. When downloading a file, debusine tries stores in descending order of download priority, counting null as the lowest; it breaks ties in descending order of upload priority, again counting null as the lowest. If there is still a tie, it picks one of the possibilities arbitrarily.
populate (boolean, defaults to False): If True, debusine-admin vacuum_storage ensures that this store has a copy of all files in the scope.
drain (boolean, defaults to False): If True, debusine-admin vacuum_storage moves all files in this scope to some other store in the same scope, following the same rules for finding a target store as for uploads of new files. It does not move into a store if that would take its total size over soft_max_size (either for the scope or the file store), and it logs an error if it cannot find any eligible target store.
drain_to (string, optional): If this field is set, then constrain drain to use the store with the given name in this scope.
read_only (boolean, defaults to False): If True, debusine will not add new files to this store. Use this in combination with drain to prepare for removing the file store.
write_only (boolean, defaults to False): If True, debusine will not read files from this store. This is suitable for provider storage classes that are designed for long-term archival rather than routine retrieval, such as S3 Glacier Deep Archive. debusine will only upload files to this store when applying the populate policy.
soft_max_size (integer, optional): An integer specifying the number of bytes that the file store can hold for this scope (accounting files that are in multiple scopes to all of the scopes in question). This limit may be exceeded temporarily during uploads; debusine-admin vacuum_storage will move files from least-recently-created artifacts to another file store to get back below the limit.

Storage policy recommendations

The shared local storage should normally have the highest upload_priority of any store, in order not to block uploads of new files on slow data transfers. Its store-level soft_max_size field should be set somewhat below the available file system size, with clearance for at least a week’s worth of uploads if possible. That will give the daily debusine-admin vacuum_storage job time to move files from least-recently-created artifacts to other file stores.

To guard against data loss, files may be in multiple stores: for example, a backup store might use the populate policy to ensure that it has a copy of all files, and perhaps write_only to ensure that debusine does not try to serve files directly from it. Alternatively, an administrator might use lower-level tools such as rclone to handle backups.