Introduction

gitlab-monitor is a Prometheus Web exporter that does the following:

  1. Collects GitLab production metrics via custom probes defined in a YAML configuration file.
  2. Custom probes gather measurements in the form of key/value pairs.
  3. For each probe, gitlab-monitor creates an HTTP endpoint /<probe_name> (by default on port 9168) that delivers these metrics to a Prometheus scraper.

A central Prometheus process is configured to poll exporters at a specified frequency.

Supported Probes

Below is a list of probes added by this exporter, and their corresponding metrics.

  1. Database
    • Per-table tuple stats -- gitlab_database_stat_table_*
    • Row count queries -- gitlab_database_rows
    • CI builds -- ci_pending_builds, ci_created_builds, ci_stale_builds, ci_running_builds
    • Bloat -- gitlab_database_bloat_$type_$key with type btree (index bloat) or table (table bloat) and keys bloat_ratio bloat_size extra_size real_size (see below)
    • Remote mirrors -- project_remote_mirror_last_successful_update_time_seconds, project_remote_mirror_last_update_time_seconds
  2. Git
    • git pull/push timings -- git_pull_time_milliseconds, git_push_time_milliseconds
    • git processes stats (see Process below)
  3. Process
    • CPU time -- process_cpu_seconds_total
    • Start time -- process_start_time_seconds
    • Count -- process_count
    • Memory usage
      • Data from /proc//cmdline:
      • process_resident_memory_bytes
      • process_virtual_memory_bytes
      • Data from /proc//smaps -- probe_smaps (off by default):
      • process_smaps_size_bytes
      • process_smaps_rss_bytes
      • process_smaps_shared_clean_bytes
      • process_smaps_shared_dirty_bytes
      • process_smaps_private_clean_bytes
      • process_smaps_private_dirty_bytes
      • process_smaps_swap_bytes
      • process_smaps_pss_bytes
  4. Sidekiq
    • Stats
      • sidekiq_jobs_processed_total
      • sidekiq_jobs_failed_total
      • sidekiq_jobs_enqueued_size
      • sidekiq_jobs_scheduled_size
      • sidekiq_jobs_retry_size
      • sidekiq_jobs_dead_size
      • sidekiq_default_queue_latency_seconds
      • sidekiq_processes_size
      • sidekiq_workers_size
    • Queues
      • sidekiq_queue_size
      • sidekiq_queue_paused
      • sidekiq_queue_latency_seconds
    • Jobs
      • sidekiq_enqueued_jobs
      • sidekiq_running_jobs
      • sidekiq_to_be_retried_jobs

Setup with GitLab Development Kit

gitlab-monitor can be setup with the GitLab Development Kit for development. When using the gitlab-monitor CLI, you'll need to set the --db-conn flag to connect to the PostgreSQL instance in your GDK folder. For example:

bin/gitlab-mon row-counts --db-conn="dbname=gitlabhq_development host=/Users/<user>/gitlab-development-kit/postgresql"

Running gitlab-monitor as a Web exporter

When serving the pages on localhost, you'll need to edit the YAML configuration file. An example can be found under config/gitlab-monitor.yml.example. For each probe that has to connect to the database, set the connection_string to dbname=gitlabhq_development host=/Users/<user>/gitlab-development-kit/postgresql

Once you have this configured, you can then run:

bin/gitlab-mon web -c config/gitlab-monitor.yml

Once running, you can point your browser or curl to the following URLs:

Database Bloat Metrics

Database bloat is measured for indexes (btree) and/or tables (table). Returned metrics contain:

  • bloat_ratio: estimated ratio of the real size used by bloat_size.
  • bloat_size: estimated size of the bloat without the extra space kept for the fillfactor.
  • extra_size: estimated extra size not used/needed by the index. This extra size is composed by the fillfactor, bloat and alignment padding spaces.
  • real_size: real size of the index

Also see the original documentation.

Note that all metrics returned are estimates without an upper bound for the error.

Contributing

gitlab-monitor is an open source project and we are very happy to accept community contributions. Please refer to CONTRIBUTING.md for details.