Anitya Infrastructure SOP

Anitya is used by Fedora to track upstream project releases and maps them to downstream distribution packages, including (but not limited to) Fedora.

Anitya staging instance: https://stg.release-monitoring.org

Anitya production instance: https://release-monitoring.org

Anitya project page: https://github.com/fedora-infra/anitya

Contact Information

Owner

Fedora Infrastructure Team

Contact

#fedora-admin, #fedora-apps

Persons

zlopez

Location

iad2.fedoraproject.org

Servers

Production +

  • os-master01.iad2.fedoraproject.org + Staging +

  • os-master01.stg.iad2.fedoraproject.org

Purpose

Map upstream releases to Fedora packages.

Hosts

The current deployment is made up of release-monitoring OpenShift namespace.

release-monitoring

This OpenShift namespace runs following pods:

  • The apache/mod_wsgi application for release-monitoring.org

  • A libraries.io SSE client

  • A service checking for new releases

This OpenShift project relies on:

  • A postgres db server running in OpenShift

  • Lots of external third-party services. The anitya webapp can scrape pypi, rubygems.org, sourceforge and many others on command.

  • Lots of external third-party services. The check service makes all kinds of requests out to the Internet that can fail in various ways.

  • Fedora messaging RabbitMQ hub for publishing messages

Things that rely on this host:

  • hotness-sop is a fedora messaging consumer running in Fedora Infra in OpenShift. It listens for Anitya messages from here and performs actions on koji and bugzilla.

Releasing

The release process is described in Anitya documentation.

Deploying

Staging deployment of Anitya is deployed in OpenShift on os-master01.stg.iad2.fedoraproject.org.

To deploy staging instance of Anitya you need to push changes to staging branch on Anitya GitHub. GitHub webhook will then automatically deploy a new version of Anitya on staging.

Production deployment of Anitya is deployed in OpenShift on os-master01.iad2.fedoraproject.org.

To deploy production instance of Anitya you need to push changes to production branch on Anitya GitHub. GitHub webhook will then automatically deploy a new version of Anitya on production.

Configuration

To deploy the new configuration, you need ssh access to batcave01.iad2.fedoraproject.org and permissions to run the Ansible playbook.

All the following commands should be run from batcave01.

First, ensure there are no configuration changes required for the new update. If there are, update the Ansible anitya role(s) and optionally run the playbook:

$ sudo rbac-playbook openshift-apps/release-monitoring.yml

The configuration changes could be limited to staging only using:

$ sudo rbac-playbook openshift-apps/release-monitoring.yml -l staging

This is recommended for testing new configuration changes.

Upgrading

Staging

To deploy new version of Anitya you need to push changes to staging branch on Anitya GitHub. GitHub webhook will then automatically deploy a new version of Anitya on staging.

Production

To deploy new version of Anitya you need to push changes to production branch on Anitya GitHub. GitHub webhook will then automatically deploy a new version of Anitya on production.

Congratulations! The new version should now be deployed.

Administrating release-monitoring.org

Anitya web application offers some functionality to administer itself.

User admin status is tracked in Anitya database. Admin users can grant or revoke admin priviledges to users in the users tab.

Admin users have additional functionality available in web interface. In particular, admins can view flagged projects, remove projects and remove package mappings etc.

For more information see Admin user guide in Anitya documentation.

Flags

Anitya lets users flag projects for administrator attention. This is accessible to administrators in the flags tab.

Monitoring

To monitor the activity of Anitya you can connect to Fedora infra OpenShift and look at the state of pods.

For staging look at the release-monitoring namespace in staging OpenShift instance.

For production look at the release-monitoring namespace in production OpenShift instance.

Troubleshooting

This section contains various issues encountered during deployment or configuration changes and possible solutions.

Fedmsg messages aren’t sent

Issue: Fedmsg messages aren’t sent.

Solution: Set USER environment variable in pod.

Explanation: Fedmsg is using USER env variable as a username inside messages. Without USER env set it just crashes and didn’t send anything.

Cronjob is crashing

Issue: Cronjob pod is crashing on start, even after configuration change that should fix the behavior.

Solution: Restart the cronjob. This could be done by OPS.

Explanation: Every time the cronjob is executed after crash it is trying to actually reuse the pod with bad configuration instead of creating a new one with new configuration.

Database migration is taking too long

Issue: Database migration is taking few hours to complete.

Solution: Stop every pod and cronjob before migration.

Explanation: When creating new index or doing some other complex operation on database, the migration script needs exclusive access to the database.

Old version is deployed instead the new one

Issue: The pod is deployed with old version of Anitya, but it says that it was triggered by correct commit.

Solution: Set dockerStrategy in buildconfig.yml to noCache.

Explanation: The OpenShift is by default caching the layers of docker containers, so if there is no change in Dockerfile it will just use the cached version and don’t run the commands again.