OpenShift SOP

OpenShift is used in Fedora Infrastructure to host a number of applications. This SOP is applicable to the OpenShift cluster and not the application running on it.

Production instance: https://os.fedoraproject.org/ Staging instance: https://os.stg.fedoraproject.org/

Contents

Contact information

Owner

Fedora Infrastrucutre Team

Contact

#fedora-admin

Persons

.oncall

Location

Phoenix

Servers

  • os-master01.phx2.fedoraproject.org

  • os-master02.phx2.fedoraproject.org

  • os-master03.phx2.fedoraproject.org

  • os-node01.phx2.fedoraproject.org

  • os-node02.phx2.fedoraproject.org

  • os-node03.phx2.fedoraproject.org

  • os-node04.phx2.fedoraproject.org

  • os-node05.phx2.fedoraproject.org

Purpose

Run Fedora Infrastructure applications

Things that could go wrong

Application build is stuck

If an application build seems stuck, it generally helps to restart the docker service on the node used for the build.

First check which builds are currently running on the cluster. :

[os-master01] # oc get builds --all-namespaces | grep -i running

if the builds seems stuck (ie running for more than 20 mins), Check on which nodes they are scheduled.

Let’s take a bodhi build for example :

[os-master01] # oc -n bodhi get builds

[os-master01] # oc -n bodhi describe build bodhi-base-49 | grep os-node

Once you have identified which node the build is running on, you can restart the docker service on this node. :

[os-node02] # systemctl restart docker

You can start a new build.:

[os-master01] # oc -n bodhi start-build bodhi-base

Finally you can check if there are any more build stuck. If that’s the case just repeat these steps. :

[os-master01] # oc get builds --all-namespaces