Installing and configuring Graphite with Collectd on FreeBSD
Using Graphite for a smidgen blog like mine is definitely an overkill and Munin fits much better but I wasn’t able to confront my curiosity. So that’s why this post.
Before I start, I’d like to mention that even if I’ll be speaking about my endeavor with Graphite on a system running FreeBSD 10.0-RC5, the same configuration would certainly work on Linux. But since different distros install Graphite’s components into different directories (certainly different from what FreeBSD does) you should update your configuration files smartly.
- First things first… Install the packages:
# pkg install py27-graphite-web Updating repository catalogue The following 13 packages will be installed: Installing sqlite3: 3.8.2 Installing py27-cairo: 1.10.0_1 Installing py27-zope.interface: 3.8.0_1 Installing py27-thrift: 0.9.1,1 Installing py27-setuptools: 2.0.1 Installing py27-whisper: 0.9.12 Installing py27-sqlite3: 2.7.6_3 Installing py27-twistedCore: 13.2.0 Installing py27-django: 1.6.1 Installing py27-txamqp: 0.3_2 Installing py27-django-tagging: 0.3.1 Installing py27-carbon: 0.9.12 Installing py27-graphite-web: 0.9.12 The installation will require 62 MB more space 8 MB to be downloaded Proceed with installing packages [y/N]: y
# cd /usr/local/etc/carbon # cp storage-schemas.conf.example storage-schemas.conf # cp carbon.conf.example carbon.conf
GRAPHITE_ROOT = /usr/local/graphite GRAPHITE_CONF_DIR = /usr/local/etc/carbon GRAPHITE_STORAGE_DIR = /usr/local/graphite/storage/ STORAGE_DIR = /usr/local/graphite/storage/ LOCAL_DATA_DIR = /usr/local/graphite/storage/whisper/ CONF_DIR = /usr/local/etc/carbon LOG_DIR = /usr/local/graphite/storage/log/ PID_DIR = /var/run
LINE_RECEIVER_INTERFACE = 127.0.0.1 PICKLE_RECEIVER_INTERFACE = 127.0.0.1 CACHE_QUERY_INTERFACE = 127.0.0.1 [relay] LINE_RECEIVER_INTERFACE = 127.0.0.1 LINE_RECEIVER_PORT = 2013 PICKLE_RECEIVER_INTERFACE = 127.0.0.1 PICKLE_RECEIVER_PORT = 2014 [aggregator] LINE_RECEIVER_INTERFACE = 127.0.0.1 PICKLE_RECEIVER_PORT = 2024
# /usr/local/etc/rc.d/carbon start Starting carbon. Traceback (most recent call last): File "/usr/local/bin/carbon-cache.py", line 28, in <module> from carbon.util import run_twistd_plugin File "/usr/local/lib/python2.7/site-packages/carbon/util.py", line 21, in <module> from twisted.scripts._twistd_unix import daemonize ImportError: cannot import name daemonize </pre> <li>What's that?!. Thankfully, this could be fix in a second by installing daemonize and editing util.py</li> <pre> # easy_install pip # pip install daemonize # vi /usr/local/lib/python2.7/site-packages/carbon/util.py
Original version:
from twisted.scripts._twistd_unix import daemonize
After editing:
#from twisted.scripts._twistd_unix import daemonize
import daemonize
# /usr/local/etc/rc.d/carbon start Starting carbon. Starting carbon-cache (instance a)
Another web server powered by FreeBSD
During the last couple of days I’ve been migrating data and various configurations from my old Ubuntu server to a new one which is running FreeBSD 10.0-RC5 now.
Upgrading FreeBSD 9.2 to 10.0-RC4 on Rackpsace’s instance
In reality it turned out to be a fairly straightforward procedure:
- This is crucial. You must save XENHVM config before the upgrade otherwise you won’t be able to re-build the kernel which is required to bring the network up.
# cp /usr/src/sys/amd64/conf/XENHVM /root/amd64_XENHVM # cp /usr/src/sys/i386/conf/XENHVM /root/i386_XENHVM
# freebsd-update -r 10.0-RC4 upgrade Looking up update.FreeBSD.org mirrors... 5 mirrors found. Fetching metadata signature for 9.2-RELEASE from update5.freebsd.org... done. Fetching metadata index... done. Fetching 1 metadata patches. done. Applying metadata patches... done. Fetching 1 metadata files... done. Inspecting system... done. WARNING: This system is running a "xenhvm" kernel, which is not a kernel configuration distributed as part of FreeBSD 9.2-RELEASE. This kernel will not be updated: you MUST update the kernel manually before running "/usr/sbin/freebsd-update install". ... Skipped for brevity ... To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".
# cd /usr/src # cp /root/amd64_XENHVM /usr/src/sys/amd64/conf/XENHVM # cp /root/i386_XENHVM /usr/src/sys/i386/conf/XENHVM # make buildkernel KERNCONF=XENHVM # make installkernel KERNCONF=XENHVM # shutdown -r now
# freebsd-version 10.0-RC4
Grow root ZFS pool online under FreeBSD 9.2 using Rackspace’s instance
If you opted for a FreeBSD instance on Rackspace and chose standard configuration then you’ll end up with 512MB of RAM and 20GB of space. That’s ok for the starter but eventually you’d need more one day and will upgrade RAM to 1GB for example. With this upgrade your disk will grow upto 40GB but OS would not notice that (Rackspace doesn’t support automatic disk partitioning on FreeBSD). But you could fix that easily by yourself with a few simple CLI commands.
So lets plunge into the command line and see what we have and what we could do about that:
- Before the upgrade disk’s layout looks similarly to what is shown below:
# gpart show -p => 1 1048575 ada2 MBR (512M) 1 1048575 ada2s1 linux-data (512M) => 34 41942973 ada0 GPT (20G) 34 94 ada0p1 freebsd-boot (47k) 128 41942879 ada0p2 freebsd-zfs (20G) # zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT zroot 19.9G 3.67G 16.2G 18% 1.00x ONLINE -
# gpart show => 1 2097151 ada2 MBR (1.0G) 1 2097151 1 linux-data (1G) => 34 41942973 ada0 GPT (40G) [CORRUPT] 34 94 1 freebsd-boot (47k) 128 41942879 2 freebsd-zfs (20G) # zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT zroot 19.9G 3.67G 16.2G 18% 1.00x ONLINE -
GEOM: ada0: the secondary GPT header is not in the last LBA.
…The GPT primary metadata is stored at the beginning of the device. For
redundancy, a secondary (backup) copy of the metadata is stored at the
end of the device…
If the size of the device has changed (e.g., volume expansion) the sec-
ondary GPT header will no longer be located in the last sector. This is
not a metadata corruption, but it is dangerous because any corruption of
the primary GPT will lead to loss of the partition table. This problem
is reported by the kernel with the message:GEOM: provider: the secondary GPT header is not in the last LBA.
# gpart recover ada0 # gpart show => 1 2097151 ada2 MBR (1.0G) 1 2097151 1 linux-data (1G) => 34 83886013 ada0 GPT (40G) 34 94 1 freebsd-boot (47k) 128 41942879 2 freebsd-zfs (20G) 41943007 41943040 - free - (20G)
# gpart resize -i 2 ada0 gpart: Device busy
# sysctl kern.geom.debugflags=16 kern.geom.debugflags: 0 -> 16 # gpart resize -i 2 ada0 # gpart show => 1 2097151 ada2 MBR (1.0G) 1 2097151 1 linux-data (1G) => 34 83886013 ada0 GPT (40G) 34 94 1 freebsd-boot (47k) 128 83885919 2 freebsd-zfs (40G) # zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT zroot 19.9G 3.67G 16.2G 18% 1.00x ONLINE -
# zpool set autoexpand=on zroot # zpool online -e zroot ada0p2 ada0p2 # zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT zroot 39.9G 3.67G 36.2G 9% 1.00x ONLINE -
Bye!
Never make plans but I’ll try
From New Year’s Resolutions for SysAdmins presented by USENIX I have picked just three for myself:
- “Gather more metrics and become a data driven IT team.”
- “I will not ever again work for a cheapskate business owner who cannot, or will not, pay me a fair rate.”
- “Learn to be a better coder.”
The latter one one is the most important one.
How to say “yes” to pkgrm
If you got bored pressing “y” every time you delete a package and pkgrm asks its “Do you want to remove this package? [y,n,?,q]” question, there is a very easy solution:
# yes | pkgrm package1 package2 ... package3
Has saved me a lot of time.
ruBSD 2013 Conference
Just noticed that the registration for ruBSD 2013 Conference (translated) has been closed. So happy I applied earlier and will have a chance to listen to Theo de Raadt, Konstantin Belousov, Henning Brauer and other BSD starts in person.
Why doesn’t Solaris SMF aware service start on boot?
A quick hint on how to find a possible root cause why a newly imported service doesn’t auto start on system reboot. So… You’ve just imported service’s manifest, rebooted the system and noticed the service is still in the offline state:
svccfg import service_manifest.xml init 6 svcs service_name STATE STIME FMRI offline 15:55:52 svc:service_name:default
A good start is to use “svcs -l service_name” to check that all service instances upon your service depends (used ssh as an example):
$ svcs -l ssh fmri svc:/network/ssh:default name SSH server enabled true state online next_state none state_time Wed Nov 06 15:55:01 2013 logfile /var/svc/log/network-ssh:default.log restarter svc:/system/svc/restarter:default contract_id 60 dependency require_all/none svc:/system/filesystem/local (online) dependency optional_all/none svc:/system/filesystem/autofs (disabled) dependency require_all/none svc:/network/loopback (online) dependency require_all/none svc:/network/physical (online) dependency require_all/none svc:/system/cryptosvc (online) dependency require_all/none svc:/system/utmp (online) dependency require_all/restart file://localhost/etc/ssh/sshd_config (online)
So you’ve checked the logs and all the dependencies and still don’t fully understand why the new service did start. I feel your pain as I was in exactly the same situation. Take a look at the services that depend on your service (again I will use ssh as an example):
svcs -D ssh STATE STIME FMRI online 15:55:51 svc:/milestone/multi-user-server:default
Or instead, the services it depends upon:
svcs -d svc:/application/management/sma:default STATE STIME FMRI online 15:54:38 svc:/milestone/name-services:default online 15:54:49 svc:/system/cryptosvc:default online 15:54:50 svc:/milestone/network:default online 15:54:51 svc:/system/filesystem/local:default online 15:55:00 svc:/milestone/sysconfig:default online 15:55:01 svc:/system/system-log:default online 15:55:06 svc:/network/rpc/rstat:default
If you don’t see “milestone” being mentioned in the output then that is definitely the problem and explains why the service didn’t start when you rebooted the system. Basically it’s the same as if you hadn’t run “chkconfig service_name on” on a RedHat-like or “update-rc.d serice_name defaults” on a Debian-like Linux system. Refer to the table below to get an idea which run level corresponds to what milestone:
Run Level | SMF Milestones |
---|---|
S | milestone/single-user:default |
2 | milestone/multi-user:default |
3 | milestone/multi-user-server:default |
For the sake of completeness here is a full list of all milestones available (at least on the system I have near me):
svcs "milestone*" STATE STIME FMRI disabled 15:54:42 svc:/milestone/patching:default online 15:54:38 svc:/milestone/name-services:default online 15:54:49 svc:/milestone/devices:default online 15:54:50 svc:/milestone/network:default online 15:54:50 svc:/milestone/single-user:default online 15:55:00 svc:/milestone/sysconfig:default online 15:55:34 svc:/milestone/multi-user:default online 15:55:51 svc:/milestone/multi-user-server:default
Time to open the service’s manifest file and edit it. And here you have two options:
- Either a milestone will depend upon your service (the first example);
- Or make your service dependent on a milestone (the latter one);
<dependent name='ssh_multi-user-server' grouping='optional_all' restart_on='none'> <service_fmri value='svc:/milestone/multi-user-server' /> </dependent> <dependency name='network' grouping='require_all' restart_on='error' type='service'> <service_fmri value='svc:/milestone/network:default'/> </dependency>
Now disable, delete, import and enable the service once again:
svcadmin disable service_name svccfg delete service_name svccfg import service_name.xml svcadmin enable service_name
At this point you could safely reboot the system.
Back to school with Coursera
There are two courses that have been recently relaunched at coursera.org:
- Machine Learning
- Game Theory
Both are very interesting and highly recommended to attend. So don’t think twice and join the class whilst they are in their first week only.
Hopefully I will have enough time to complete both of them.
Matching Oracle ASM disks names with the physical devices
Quite often it’s required to find a physical device which backs up ASM disk. The easiest way a Linux administrator could accomplish that is by using oracleasm command (of course, if ASMlib was used to create them):
# oracleasm listdisks ASMARCHIVE1 ASMARCHIVE2 ASMARCHIVE3 ASMDATA1 ASMDATA2 ASMDATA3 OCR1 OCR2 VOTEDISK1 VOTEDISK2 VOTEDISK3
# oracleasm querydisk -p OCR1 Disk "OCR1" is a valid ASM disk /dev/mapper/mpath6: LABEL="OCR1" TYPE="oracleasm" /dev/sdx: LABEL="OCR1" TYPE="oracleasm" /dev/sdh: LABEL="OCR1" TYPE="oracleasm" /dev/sdan: LABEL="OCR1" TYPE="oracleasm" /dev/sdbd: LABEL="OCR1" TYPE="oracleasm"