Installing and configuring Graphite with Collectd on FreeBSD

Using Graphite for a smidgen blog like mine is definitely an overkill and Munin fits much better but I wasn’t able to confront my curiosity. So that’s why this post.
Before I start, I’d like to mention that even if I’ll be speaking about my endeavor with Graphite on a system running FreeBSD 10.0-RC5, the same configuration would certainly work on Linux. But since different distros install Graphite’s components into different directories (certainly different from what FreeBSD does) you should update your configuration files smartly.

• First things first… Install the packages:

# pkg install py27-graphite-web
Updating repository catalogue
The following 13 packages will be installed:

	Installing sqlite3: 3.8.2
	Installing py27-cairo: 1.10.0_1
	Installing py27-zope.interface: 3.8.0_1
	Installing py27-thrift: 0.9.1,1
	Installing py27-setuptools: 2.0.1
	Installing py27-whisper: 0.9.12
	Installing py27-sqlite3: 2.7.6_3
	Installing py27-twistedCore: 13.2.0
	Installing py27-django: 1.6.1
	Installing py27-txamqp: 0.3_2
	Installing py27-django-tagging: 0.3.1
	Installing py27-carbon: 0.9.12
	Installing py27-graphite-web: 0.9.12

The installation will require 62 MB more space

8 MB to be downloaded

Proceed with installing packages [y/N]: y

• That was easy. Out next step is to setup Carbon. You could read more about the Carbon daemons here:

# cd /usr/local/etc/carbon
# cp storage-schemas.conf.example storage-schemas.conf
# cp carbon.conf.example carbon.conf

• Open your favorite editor and set directory paths:

  GRAPHITE_ROOT        = /usr/local/graphite
  GRAPHITE_CONF_DIR    = /usr/local/etc/carbon
  GRAPHITE_STORAGE_DIR = /usr/local/graphite/storage/
  STORAGE_DIR          = /usr/local/graphite/storage/
  LOCAL_DATA_DIR       = /usr/local/graphite/storage/whisper/
  CONF_DIR             = /usr/local/etc/carbon
  LOG_DIR              = /usr/local/graphite/storage/log/
  PID_DIR              = /var/run
  

• That’s basically it. I had to change some other parameters in the file to make the carbon process listen on the localhost interface only since by default it listens on 0.0.0.0:

LINE_RECEIVER_INTERFACE = 127.0.0.1
PICKLE_RECEIVER_INTERFACE = 127.0.0.1
CACHE_QUERY_INTERFACE = 127.0.0.1

[relay]
LINE_RECEIVER_INTERFACE = 127.0.0.1
LINE_RECEIVER_PORT = 2013
PICKLE_RECEIVER_INTERFACE = 127.0.0.1
PICKLE_RECEIVER_PORT = 2014

[aggregator]
LINE_RECEIVER_INTERFACE = 127.0.0.1
PICKLE_RECEIVER_PORT = 2024

• Everything is ready and carbon could be started.

# /usr/local/etc/rc.d/carbon start
Starting carbon.
Traceback (most recent call last):
  File "/usr/local/bin/carbon-cache.py", line 28, in <module>
    from carbon.util import run_twistd_plugin
  File "/usr/local/lib/python2.7/site-packages/carbon/util.py", line 21, in <module>
    from twisted.scripts._twistd_unix import daemonize
ImportError: cannot import name daemonize
</pre>
<li>What's that?!. Thankfully, this could be fix in a second by installing daemonize and editing util.py</li>
<pre>
# easy_install pip
# pip install daemonize
# vi /usr/local/lib/python2.7/site-packages/carbon/util.py

Original version:

from twisted.scripts._twistd_unix import daemonize

After editing:

#from twisted.scripts._twistd_unix import daemonize
import daemonize

• Starting carbon once again. And don’t forget to add carbon_enable=”YES” into your rc.conf file.

# /usr/local/etc/rc.d/carbon start
Starting carbon.
Starting carbon-cache (instance a)


Nginx, Django and UWSGI

That was the most difficult and stumbling part for me since I knew close to nothing anything about Django framework, the way it should be configured, etc., etc., etc. And I still have a misty understanding of how it works.

• First of all a database (I will be using sqlite but other DB are also supported) must be initialized. To do that local_settings.py file must be slightly updated to have some critical bits in it:

# cd /usr/local/lib/python2.7/site-packages/graphite
# cp local_settings.py.example local_settings.py
# vi local_settings.py

GRAPHITE_ROOT = '/usr/local/graphite'
CONF_DIR = '/usr/local/etc/graphite'
STORAGE_DIR = '/usr/local/storage'
CONTENT_DIR = '/usr/local/graphite/webapp/content'
DASHBOARD_CONF = '/usr/local/etc/graphite/dashboard.conf'
GRAPHTEMPLATES_CONF = '/usr/local/etc/graphite/graphTemplates.conf'
LOG_DIR = '/usr/local/graphite/storage/log/webapp'
INDEX_FILE = '/usr/local/graphite/storage/index' 

DATABASES = {
    'default': {
        'NAME': '/usr/local/graphite/storage/graphite.db',
        'ENGINE': 'django.db.backends.sqlite3',
        'USER': '',
        'PASSWORD': '',
        'HOST': '',
        'PORT': ''
    }
}

• Now we are ready to create our database:

# python manage.py syncdb
Creating tables ...
Creating table account_profile
Creating table account_variable
Creating table account_view
Creating table account_window
Creating table account_mygraph
Creating table dashboard_dashboard_owners
Creating table dashboard_dashboard
Creating table events_event
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_session
Creating table django_admin_log
Creating table django_content_type
Creating table tagging_tag
Creating table tagging_taggeditem

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes

• Copying example configuration files.

# cd /usr/local/etc/graphite/
# cp graphite.wsgi.example graphite.wsgi
# cp dashboard.conf.example dashboard.conf
# cp graphTemplates.conf.example graphTemplates.conf

• Install UWSGI either from ports or using pkg tool.
• Configure UWSGI. That was the moment I thought I would go crazy. The main reason is of course the lack of experience with Django framework. But even now I still don’t understand why uwsgi would throw an error if I call a module a “graphite”:

# cd /usr/local/etc/graphite
# cat graphite.py 
import os, sys
sys.path.append('/usr/local/graphite/webapp')
os.environ['DJANGO_SETTINGS_MODULE'] = 'graphite.settings'

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

# READ THIS
# Initializing the search index can be very expensive, please include
# the WSGIImportScript directive pointing to this script in your vhost
# config to ensure the index is preloaded before any requests are handed
# to the process.
from graphite.logger import log
log.info("graphite.wsgi - pid %d - reloading search index" % os.getpid())
import graphite.metrics.search

# cat /tmp/uwsgi.init 
[uwsgi]
master = true
processes = 3
socket = /tmp/uwsgi.sock
chmod-socket = 666
gid = 80
uid = 80
python-path = /usr/local/lib/python2.7/site-packages/
chdir = /usr/local/etc/graphite/
module = graphite

# tail /var/log/uwsgi.log 
mapped 290976 bytes (284 KB) for 3 cores
*** Operational MODE: preforking ***
added /usr/local/lib/python2.7/site-packages/ to pythonpath.
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
--- no python application found, check your startup logs for errors ---

• But once the module is changed to something else, mygraphite for example, everything becomes nice and dandy. Below is my ultimate uwsgi configuration options from rc.conf:

uwsgi_flags="-L -M -p 3 --socket /tmp/uwsgi.sock --gid 80 --uid 80 --python-path /usr/local/lib/python2.7/site-packages/ --chdir /usr/local/etc/graphite/ -w mygraphite"

• Next and finally Nginx:

cat graphite.flagword.net.conf
upstream django {
server unix:/tmp/uwsgi.sock;
}

server {
listen 80;
server_name graphite.flagword.net;
return 301 https://$host$request_uri;
}

server {
listen 443;
server_name graphite.flagword.net;
root /usr/local/graphite/webapp;

access_log /var/log/nginx/graphite.flagword.access.log main;
error_log /var/log/nginx/graphite.flagword.error.log;

ssl on;
ssl_certificate #path to SSL certificate;
ssl_certificate_key #path to SSL Private Key;
ssl_session_timeout 5m;
ssl_protocols SSLv2 SSLv3 TLSv1;
ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
ssl_prefer_server_ciphers on;

location /content/ {
alias /usr/local/graphite/webapp/content/;
}

location / {
auth_basic “Graphite”;
auth_basic_user_file #Path to auth file;
uwsgi_pass django;
include uwsgi_params;
}

location = /favicon.ico {
log_not_found off;
access_log off;
}
}

• If graphite still doesn’t work for you and you see the following error in uwsgi’s log file then check Stackoverflow for the explanation.

ImportError: No module named defaults

• I had to use a simple one-liner to find and replace all occurrences of the string “from django.conf.urls.defaults” with “from django.conf.urls”:

find /usr/local/lib/python2.7/site-packages/graphite -type f -exec sed -ine 's/from django.conf.urls.defaults/from django.conf.urls/g' '{}' ';'

• Hopefully I’ve struggled throw all obstacles and hurdles successfully and now it’s the right time to feed some data it. I used collected for that. I’m not going to dwell upon it very much since it’s absolutely trivial to setup. Take a look at its configuration file, it’s self explanatory. Just don’t forget to uncomment write_graphite plugin to push the data right into Graphite:

/usr/local/etc/collectd.conf:

LoadPlugin write_graphite
<Plugin write_graphite>
  <Node "rnode.flagowrd.net">
    Host "localhost"
    Port "2003"
    Protocol "tcp"
    LogSendErrors true
    Prefix "collectd"
    Postfix "collectd"
    StoreRates true
    AlwaysAppendDS false
    EscapeCharacter "_"
  </Node>
</Plugin>

• Give it some time to run and you should start seeing a new data under “/usr/local/graphite/storage/whisper/”

  $ ls -l /usr/local/graphite/storage/whisper/
  total 17
  drwxr-xr-x   3 root  wheel   3 Jan 10 19:57 carbon
  drwxr-xr-x  10 root  wheel  10 Jan 11 15:44 collectdrnode_flagword_netcollectd
  
  ls -l /usr/local/graphite/storage/whisper/collectdrnode_flagword_netcollectd/
  total 84
  drwxr-xr-x  2 root  wheel   7 Jan 11 10:14 cpu-0
  drwxr-xr-x  5 root  wheel   5 Jan 11 10:34 interface-xn0
  drwxr-xr-x  3 root  wheel   3 Jan 11 10:14 load
  drwxr-xr-x  2 root  wheel   7 Jan 11 10:14 memory
  drwxr-xr-x  2 root  wheel   9 Jan 11 15:45 nginx
  drwxr-xr-x  2 root  wheel  33 Jan 24 00:31 ntpd
  drwxr-xr-x  2 root  wheel   4 Jan 11 10:14 swap
  drwxr-xr-x  3 root  wheel  22 Jan 11 10:28 zfs_arc
  

Hopefully now you could log into Graphite and start building new graphs.