Log Infrastructure SOP

Logs are centrally referred to our loghost and managed from there by rsyslog to create several log outputs.

Epylog provides twice-daily log reports of activities on our systems. It runs on our central loghost and generates reports on all systems centrally logging.

Contact Information

Owner: Fedora Infrastructure Team
Contact: #fedora-admin, sysadmin-main
Location: Phoenix
Servers: log01.phx2.fedoraproject.org
Purpose: Provides our central logs and reporting

Essential data/locations:

Logs compiled using rsyslog on log01 into a single set of logs for all systems:
```
/var/log/merged/
```
These logs are rotated every day and kept for only 2 days. This set of logs is only used for immediate analysis and more trivial 'tailing' of the log file to watch for events.
Logs for each system separately in /var/log/hosts

These logs are maintained forever, practically, or for as long as we possibly can. They are broken out into a $hostname/$YEAR/$MON/$DAY directory structure so we can locate a specific day’s log immediately.
Log reports generated by epylog: Log reports generated by epylog are outputted to /srv/web/epylog/merged

The reports are accessible via a web browser from https://admin.fedoraproject.org/epylog/merged/

This path requires a username and a password to access. To add your username and password you must first join the sysadmin-logs group then login to log01.phx2.fedoraproject.org and run this command:
```
htpasswd -m /srv/web/epylog/.htpasswd $your_username
```
when prompted for a password please input a password which is NOT YOUR FEDORA ACCOUNT SYSTEM PASSWORD.

Important

Let’s say that again to be sure you got it:

DO _NOT HAVE THIS BE THE SAME AS YOUR FAS PASSWORD

Configs:

Epylog configs are controlled by ansible - please see the ansible epylog module for more details. Specifically the files in roles/epylog/files/merged/

Generating a one-off epylog report:

If you wish to generate a specific log report you will need to run the following command on log01:

sudo /usr/sbin/epylog -c /etc/epylog/merged/epylog.conf --last 5h

You can replace '5h' with other time measurements to control the amount of time you want to view from the merged logs. This will mail a report notification to all the people in the sysadmin-logs group.

Audit logs, centrally:

We’ve taken the audit logs and enabled our rsyslogd on the hosts to relay the audit log contents to our central log server.

Here’s how we did that:

modify the selinux policy so that rsyslogd can read the file(s) in /var/log/audit/audit.log

BEGIN Selinux policy module:

module audit_via_syslog 1.0;

require {
    type syslogd_t;
    type auditd_log_t;
    class dir { search };
    class file { getattr read open };

}

#============= syslogd_t ==============
allow syslogd_t auditd_log_t:dir search;
allow syslogd_t auditd_log_t:file { getattr read open };

END selinux policy module

add config to rsyslog on the clients to repeatedly send all changes to their audit.log file to the central syslog server as local6:

# monitor auditd log and send out over local6 to central loghost
$ModLoad imfile.so

# auditd audit.log
$InputFileName /var/log/audit/audit.log
$InputFileTag tag_audit_log:
$InputFileStateFile audit_log
$InputFileSeverity info
$InputFileFacility local6
$InputRunFileMonitor

then modify your emitter to the syslog server to send local6.* there

on the syslog server - setup log destinations for:
- merged audit logs of all hosts explicitly drop any non-AVC audit message here) magic exclude line is:
  :msg, !contains, "type=AVC" ~
  that line must be directly above the log entry you want to filter and it has a cascade effect on everything below it unless you disable the filter
  per-host audit logs - this is everything from audit.log
On the syslog server - we can run audit2allow/audit2why on the audit logs sent there by doing this:
```
grep 'hostname' /var/log/merged/audit.log | sed 's/^.*tag_audit_log: //' | audit2allow
```
the sed is to remove the log prefix garbage from syslog transferring the msg

Future:

additional log reports for errors from http processes or servers
SEC Simple Event Coordinator to report, immediately, on events from a log stream - available in fedora/epel.
New report modules within epylog