ForemanAbrt

This plugin allows your Foreman instance to receive problem reports generated on your hosts by ABRT (Automatic Bug Reporting Tool). These reports can be inspected and eventually sent to the ABRT server for analysis.

Overview

Whenever a bug is caught by ABRT on the managed host, it is sent to the Smart proxy instead of being sent directly to the ABRT server.
The Smart proxy receives the report and stores it to the disk. Stored reports are then sent to Foreman every 30 minutes (by means of cron job). The proxy may optionally:
- Forward the report to an ABRT server immediately after being received. Server's response is discarded.
- Aggregate stored reports prior to sending them to the Foreman. Only one instance of set of similar reports from a host is sent, together with number of the reports in the set.
Foreman receives the aggregated report and stores it to the database. The reports can be inspected and forwarded ("sent for analysis") to the ABRT server. If the server responds with additional information about the report, such as links to bug trackers or suggested solutions, it is displayed alongside the report.

foreman abrt workflow

Network ports

Reports from managed hosts are sent to smart-proxy's https port which is 8443 for standalone Foreman installations and 9090 for Katello.

The proxy sends the reports using Foreman's API, which normally runs on standard https (port 443).

Installation

To be able to see ABRT problem reports in your Foreman instance, you need to install the plugin itself, install ABRT plugin for your smart proxies and configure your hosts to send the problem reports to their smart proxy.

Please note that the configuration is slightly different when using Foreman stand-alone and when it is part of Katello. Any configuration specific to Katello will be explicitly indicated.

Both plugins are available as RPMs in Foreman YUM repositories.

Prerequisites

The plugins require both Foreman and smart-proxy to be version 1.7 or later.

The plugins have been tested on Fedora 19, RHEL 6 and RHEL 7. Versions of the plugins compatible with Katello are foreman_abrt-0.0.4 and smart_proxy_abrt-0.0.6.

To have hosts automatically send ureports to Foreman, you need to have ABRT 2.1.11 or higher installed on them. RHEL 7 and Fedora 19 and higher satisfy this. ABRT in RHEL6 does not have ureport support, however you can use an unofficial build if you wish to test this feature.

Installing the Foreman plugin

To install the Foreman plugin, follow the plugin installation instructions.

You need to install the rubygem-foreman_abrt package (or ruby193-rubygem-foreman_abrt on RHEL/CentOS).

After installing the plugin, restart Foreman with touch /usr/share/foreman/tmp/restart.txt.

Setting up smart proxies

Follow the smart-proxy plugin installation instructions. You need to install the rubygem-smart_proxy_abrt package.

The plugin needs some configuration in order to work correctly.

Ensure that /etc/foreman-proxy/settings.d/abrt.yml contains the following line:
```
:enabled: true
```
After installing and configuring the plugin, smart-proxy has to be restarted.
```
service foreman-proxy restart
```
Cron is used to transfer the captured problem reports to Foreman in batches. Ensure that the smart-proxy-abrt-send command is run periodically. The provided RPM package contains a cron snippet that runs the command every 30 minutes.

Configuring hosts to send problem reports to Foreman

This setup needs to be performed on every host that you wish to report its crashes to Foreman. The host has to be managed by Puppet, i.e. host record must exist in Foreman web interface.

Make sure that ABRT is installed and running.

yum install abrt-cli
systemctl start abrtd abrt-ccpp
systemctl enable abrtd abrt-ccpp

Enable auto-reporting by running the following command:

  abrt-auto-reporting enabled

Configure ABRT reporting destination - /etc/libreport/plugins/ureport.conf should contain following:

  # URL of your foreman-proxy, with /abrt path.
  URL = https://smartproxy.tld:8443/abrt
  # Verify the server certificate.
  SSLVerify = yes
  # This asks puppet config for the path to the ceritificates. you can
  # explicitly provide path by using /path/to/cert:/path/to/key on the
  # right hand side.
  SSLClientAuth = puppet

Katello: when using Foreman with Katello, the host needs to be registered (using subscription-manager) to your Katello instance. You can find the registration instructions on https://foreman.tld/content_hosts/register. The /etc/libreport/plugins/ureport.conf file should instead contain the following:

  # URL of your foreman-proxy, with /abrt path.
  URL = https://smartproxy.tld:9090/abrt
  # Verify the server certificate.
  SSLVerify = yes
  # Use the subscription management certificates to authenticate to the proxy.
  SSLClientAuth = /etc/pki/consumer/cert.pem:/etc/pki/consumer/key.pem

Add the Puppet CA to the list of trusted certificate authorities. This is needed for verifying the validity of smart-proxy's certificate:

  cp /var/lib/puppet/ssl/certs/ca.pem /etc/pki/ca-trust/source/anchors/
  update-ca-trust

Katello: when using Foreman with Katello, the subscription management certificate should be used instead:

  cp /etc/rhsm/ca/katello-server-ca.pem /etc/pki/ca-trust/source/anchors/
  update-ca-trust

Verifying that the setup works

You can verify your setup by crashing something on your managed host. We have a set of utilities in the Fedora/EPEL repository especially for this purpose:

yum -y install will-crash
will_segfault
Will segfault.
Segmentation fault (core dumped)

After a couple of seconds, a new file should appear in /var/spool/foreman-proxy-abrt on the smart-proxy host. The reports from the smart-proxy are sent to the Foreman in batches every half an hour (by default). This means that within half an hour you should be able to see the bug report in the Foreman web interface. You can send the reports to Foreman manually by running the smart-proxy-abrt-send command.

Please note that you need to enable the EPEL repository if you are using RHEL in order to install will-crash. If the will-crash package is not available, you can try the following. Please make sure not to actually report this to sleep maintainers, though:

sleep 1d &
kill -SEGV $!

Testing aggregation

If you crash the same program twice (on one host) within the same period that smart-proxy waits between forwarding the reports to foreman, then only one report with count = 2 should appear in the web interface.

However, be careful about ABRT's rate limiting - if you crash a program and then crash it again sooner that 20 seconds then the second crash is simply ignored.

Usage

The list of received problem reports can be accessed by clicking on Problem reports link in the Monitor menu. To see detailed information for a report, click on its reported date.

List of problem reports coming from a particular host is also displayed on the page with the details about the host in the Problem reports tab on the left.

Forwarding the report to the ABRT server

On the problem report details page you can forward the problem report to an actual ABRT server by clicking the Send for analysis button. The ABRT server may respond with some information it knows about the bug, such as the list of URLs related to the bug (e.g. Bugzilla link) and list of possible solutions to the problem that caused the bug to occur.

The forwarding functionality may have to be configured in Abrt tab of the configuration screen (Administer->Settings).

TODO

Use puppet to configure managed hosts to send ureports to Foreman.

Copyright

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.