Koschei SOP

Koschei deployment is managed by two Ansible playbooks:

Koschei is deployed on two separate machines - koschei-backend and koschei-web

Frontend (koschei-web) is a Flask WSGi application running with httpd. It displays information to users and allows editing package groups and changing priorities.

Backend (koschei-backend) consists of multiple services:

Koschei configuration is in /etc/koschei/config-backend.cfg and /etc/koschei/config-frontend.cfg, and is merged with the default configuration in /usr/share/koschei/config.cfg (the ones in /etc overrides the defaults in /usr). Note the merge is recursive. The configuration contains all configurable items for all Koschei services and the frontend. The alterations to configuration that aren’t temporary should be done through ansible playbook. Configuration changes have no effect on already running services — they need to be restarted, which happens automatically when using the playbook.

Koschei doesn’t keep on disk anything that couldn’t be recreated easily - all important data is stored in PostgreSQL database, configuration is managed by Ansible, code installed by RPM and so on.

To speed up operation and reduce load on external servers, Koschei caches some data obtained from services it integrates with. Most notably, YUM repositories downloaded from Koji are kept in /var/cache/koschei/repodata. Each repository takes about 100 MB of disk space. Maximal number of repositories kept at time is controlled by cache_l2_capacity parameter in config-backend.cfg (config-backend.cfg.j2 in Ansible). If repodata cache starts to consume too much disk space, that value can be decreased - after restart, koschei-*-resolver will remove least recently used cache entries to respect configured cache capacity.

Koschei needs to connect to a PostgreSQL database, other database systems are not supported. Database connection is specified in the configuration under the database_config key that can contain the following keys: username, password, host, port, database.

After an update of koschei, the database needs to be migrated to new schema. This happens automatically when using the upgrade playbook. Alternatively, it can be executed manulally using:

The backend services need to be stopped during the migration.

Koschei services are systemd units managed through systemctl. They can be started and stopped independently in any order. The frontend is run using httpd.

For stopping builds from being scheduled, stopping the koschei-scheduler service is enough. For planned Koji outages, it’s recommended to stop koschei-scheduler. It is not necessary, as koschei can recover from Koji errors and network errors automatically, but when Koji builders are stopped, it may cause unexpected build failures that would be reported to users. Other services can be left running as they automatically restart themselves on Koji and network errors.

Koschei is by default limited to 30 concurrently running builds. This limit can be changed in the configuration under koji_config.max_builds key. There’s also Koji load monitoring, that prevents builds from being scheduled when Koji load is higher that certain threshold. That should prevent scheduling builds during mass rebuilds, so it’s not necessary to stop scheduling during those.

Koschei optionally supports sending fedmsg notifications for package state changes. The fedmsg dispatch can be turned on and off in the configuration (key fedmsg-publisher.enabled). Koschei doesn’t supply configuration for fedmsg, it lets the library to load it’s own (in /etc/fedmsg.d/).

Koschei can display announcement in web UI. This is mostly useful to inform users about outages or other problems.

To set announcement, run as koschei user:

or:

To clear announcement, run as koschei user:

Packages can be added to one or more group.

To add new group named "mynewgroup", run as koschei user:

To add new group named "mynewgroup" and populate it with some packages, run as koschei user:

Some packages are more or less important and can have higher or lower priority. Any user can change manual priority, which is reset after package is rebuilt. Admins can additionally set static priority, which is not affected by package rebuilds.

To set static priority of package "foo" to value "100", run as koschei user:

After branching occurs and Koji build targets have been created, Koschei should be updated to reflect the new state. There is a special admin command for this purpose, which takes care of copying the configuration and also last builds from the history.

To branch the collection from Fedora 27 to Fedora 28, use the following:

Then you can optionally verify that the collection configuration is correct by visiting https://apps.fedoraproject.org/koschei/collections and examining the configuration of the newly branched collection.