Monitor a Ceph cluster with Ceph Dash

Ceph Storage can be completely managed and monitored via its command line tools. But, wouldn’t it be better to have a nice interface to see it running, checking its status, and have some performance statistics? Well, yes, and here’s a way to have it, thanks to Ceph Dash.

I’ve started this blog post many months ago, right after my series about building Ceph clusters. I planned to use Ceph native graphical interface, called Calamari. But, long story short, I’ve been hit multiple times by bugs, issues, problems that others found and posted on Ceph support site and stayed open for months… I was excited at first when Inktank was acquired by RedHat, as Calamari was previously part of the Enterprise Edition of Ceph, so in order to run it you would have to subscribe to their licenses. After the acquisition, RedHat decided to make Calamari completely free, but my results were really bad.

It may be me, or probably it is me, but the connections of Calamari with so many other components (both Ceph and other needed tools you need to install and configure, like Saltstack to list one…) made me feel in those months I was up to an impossible task, at least for my skills.

Then, I discovered Ceph Dash.

Ceph Dash

As described by its author, Christian Eichelmannm, on his blog post, “Ceph-Dash does not depend on the ceph_rest_api. It is a Flask based WSGI application, that can (and should) be deployed into Apache or Nginx. Calling it via Web-Browser will show a (hopefully) useful and clean dashboard containing all relevant health information about your precious Ceph cluster. All you need, is a system which has a ceph.conf, a ceph keyring (user only needs read privleges) and python-ceph installed. For testing purposes you can just fire it up and Flask will bring up a simple HTTP server on port 5000, where you can check if everything wroks. If you call this url with Content-Type: application/json, you will get all the information as a json formatted output, which you can use for further (automated) processing.”

This sounded really interesting to me, especially compared with the super complex setup required by Calamari. So, I setup my lab to try it, and eventually integrate Ceph Dash into my clyster.

As I already have a management node in my Ceph cluster, I’ve used this machine also to run Ceph Dash. In the machine, I first create a new directory for it:

mkdir /cephdash
cd /cephdash

Then, I cloned locally the github repository:

git clone https://github.com/Crapworks/ceph-dash.git

There are a couple of packages that are required to run Ceph Dash:

yum install -y python-pip
easy_install Jinja2

We are ready to test Ceph Dash. Just run /cephdash/ceph-dash/ceph-dash.py, and we can immediately appreciate the nice dashboard:

Run Ceph Dash in a webserver

Once we have verified that Ceph Dash is working properly, it’s time to have it run automatically as a service, inside a web server. For this reason, we will first install apache. In my CentOS 7 machine, these are the required steps:

yum -y install httpd mod_wsgi mod_ssl
systemctl start httpd
systemctl enable httpd

I prefer to have Ceph Dash under the regular apache folder, so I’ve removed the previous git clone and created a new one:

cd /var/www/html
git clone https://github.com/Crapworks/ceph-dash.git

Inside the github repository, Ceph Dash has already a pre-defined virtual host to be created inside Apache. So, we go and configure it:

vi /etc/httpd/conf.d/cephdash.conf

<VirtualHost *:80>
ServerName ceph.cloudconnect.local

RewriteEngine On
RewriteCond %{REQUEST_URI} !^/server-status
RewriteRule ^/?(.*) https://%{HTTP_HOST}/$1 [R,L]
</VirtualHost>

<VirtualHost *:443>
ServerName ceph.cloudconnect.local

WSGIDaemonProcess cephdash user=apache group=apache processes=1 threads=5
WSGIScriptAlias / /var/www/html/ceph-dash/contrib/wsgi/cephdash.wsgi
WSGIPassAuthorization On

SSLEngine on
SSLCertificateFile /etc/httpd/ssl/apache.crt
SSLCertificateKeyFile /etc/httpd/ssl/apache.key

<Directory /var/www/html/ceph-dash>
WSGIProcessGroup cephdash
WSGIApplicationGroup %{GLOBAL}
Order allow,deny
Allow from all
</Directory>
</VirtualHost>

Now, let’s quickly create the self-signed certificate for Apache:

mkdir /etc/httpd/ssl/
 openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/httpd/ssl/apache.key -out /etc/httpd/ssl/apache.crt

We can now restart Apache to load the new configuration:

systemctl restart httpd.service

And we can connect to Ceph Dash via the webserver:

By the way, the error in Monitor node 2 popped up after a reboot of that node, and exactly thanks to Ceph Dash i was able to spot the error state, and go fix the issue. You will still need to go into the logs to dig deeper into the status of a Ceph cluster, but for real time monitoring this is a great solution!