GitLab (Prometheus) Eating Disk Space

Do you own a GitLab server and is it eating more and more disk space? This article will help you to find the problem. In case it is related to Prometheus we will also provide some solutions.

Diagnose

To figure out what folder consumes the disk space run du -sh /path/*. This will show the files and folders in the given folder with the size.

Prometheus eating space

If the storage issue is caused by /var/opt/gitlab/prometheus/data, it means that Prometheus consumes a lot of data. You can solve this by either disabling Prometheus or limiting the service.

Disable Prometheus

To disable Prometheus open /etc/gitlab/gitlab.rb and add/edit the following line:

prometheus_monitoring['enable'] = false

Now reconfigure GitLab using gitlab-ctl reconfigure, this will automagically clean the Prometheus folder as well.

Limit Prometheus

In case you want to keep the Prometheus service in GitLab you can also limit the service. To do this open /etc/gitlab/gitlab.rb and add/edit the following line:

prometheus['flags'] = {
  'storage.tsdb.path' => "/var/opt/gitlab/prometheus/data",
  'storage.tsdb.retention.time' => "7d",
  'storage.tsdb.retention.size' => "2GB",
  'config.file' => "/var/opt/gitlab/prometheus/prometheus.yml"
}

Now reconfigure GitLab using gitlab-ctl reconfigure.

Sources

• How to Check Disk Space Usage in Linux (hostinger.com)
• /var/opt/gitlab/prometheus consumes too much disk space (docs.gitlab.com)