Openfire. Migrating from HSQLDB to MySQL.
The other day, I had to migtate Openfire from HSQLDB to MySQL using MySQL Migration tool and below just a couple tips that could save a bit of your time if you up to the same task:
- I used Windowds XP.
- MySQL Migration tool has been EOLed but it is still available from mysql.com.
- Java 1.5 is required to run MySQL Migration tool.
- Set -Xmx to 512m or bigger, as shown below, if your openfire.script is big. Mine was 135MB and that was essential.
- Not doing so you will get the following error:
- Using MySQL Migration tool is trivial but you should provide a proper connection string. If you don’t none of your tables will be migrated and what you’ll see in the end is a report similar to this one:
- I stopped Openfire copied the content (there are actually just two files inside – openfire.log and openfire.script) of /opt/openfire/embedded-db to c:\temp\embedded-db on my Windows PC
- Copied hsqldb.jar from the server to lib/ directory of MySQL Migration tool where it keeps various jars.
- used the following connection string and the class name respectively (also shown on the screenshot):
cd "c:\Program Files\MySQL\MySQL Tools for 5.0\ .\MySQLMigrationTool.exe -Xmx 512m
Connecting to source database and retrieve schemata names.
Initializing JDBC driver …
Driver class Generic Jdbc
Opening connection …
Connection jdbc:hsqldb:c:\temp\embedded-db\openfire
The list of schema names could not be retrieved (error: 0).
ReverseEngineeringGeneric.getSchemata :Out of Memory
Details:
1. Schema Migration
——————-Number of migrated schemata: 1
Schema Name: PUBLIC
– Tables: 0
– Views: 0
– Routines: 0
– Routine Groups: 0
– Synonyms: 0
– Structured Types: 0
– Sequences: 0
jdbc:hsqldb:file:c:\temp\embedded-db\openfire org.hsqldb.jdbc.Driver
The rest is just a series of clicks on the “Next” button.
Please note that if you choose to migrate the data directly into your MySQL DB all the tables will be created with their names in UPPER case. If it’s not what you prefer instead of checking “Create Objects Online” and “Trabfer Data Online” simply select “Create Script File for Create Statements” and “Create Script File for Insert Statements” and the tool will create to files Creates.sql and Inserts.sql which you could later update to meet your preferences.
To solve that issue I came up with a dumb and bold script that fixes that which I put on my GitHub repository
VM cold migration. Invalid configuration for device.
Last weekend I was doing a cold migration of one of our VMs and when the migration was one step before it would be done the following error popped up:
Invalid configuration for device 15.
Must admit, that wasn’t one of the helpful messages I expected to see. Anyway, quick googling led me to the following article at kb.vmware.com with more information and a workaround that finally helped me to finish the task.
Hope that would save someone’s time in the future.
How to setup Solaris 10 ldap client and glue it with ssh
Recently I wrote a post about configuring OpenLDAP server with TLS support using RHEL available here. There I also mentioned how to setup Linux to authenticate against a LDAP server. But I didn’t said a word about Solaris. That’s unfair and I’m going to fix that by providing a quick guide on how to setup LDAP client in Solaris 10.
- First of all add LDAP server’s certificate into your locale certificate database. Otherwise, you won’t be able to setup a TLS session:
- Just verify that everything was done right:
- Setup Solaris ldap client:
- All the rest is just almost like in the Linux world:
- Just take another look at your configuration:
- Use some very basic tools,i.e. id or getent, to make sure your could query and receive correct response from LDAP server.
- Finally, try to ssh into your server with a LDAP aware account.
/usr/sfw/bin/certutil -N -d /var/ldap/ /usr/sfw/bin/certutil -A -n "LDAP server certificate" -i /path_to_where_you_copied_ldap_certificate_file -a -t CT -d /var/ldap
/usr/sfw/bin/certutil -L -d /var/ldap/
ldapclient manual \ -a credentialLevel=proxy \ -a authenticationMethod=tls:simple \ -a domainName=example.com \ -a defaultSearchBase=DC=example,DC=com \ -a proxyDN="cn=svc_ldp_proxy,dc=example,dc=com" \ -a proxyPAssword=PASSWORD \ -a serviceSearchDescriptor="passwd:ou=people,?sub" \ -a serviceSearchDescriptor="group:ou=group,?sub?gidnumber" \ -a serviceSearchDescriptor="netgroup:ou=netgroup,?sub" \ -a serviceSearchDescriptor="shadow:ou=people,?sub?uid=*" \ -a followReferrals=false LDAP_SERVER_IP:LDAP_SERVER_PORT
Please note that your serviceSearchDescriptor attribute might be different and that depends on your LDAP structure. This attribute just instruct ldap client how it should build its query to search, in my particular case, for passwd, group and net group records.
nssswitch.conf
passwd: compat passwd_compat: ldap group: files ldap hosts: files dns ipnodes: files dns networks: files protocols: files rpc: files ethers: files netmasks: files bootparams: files publickey: files netgroup: ldap automount: files aliases: files services: files printers: user files auth_attr: files prof_attr: files project: files tnrhtp: files tnrhdb: files
cat /etc/pam.conf | grep sshd-kbdint
sshd-kbdint auth requisite pam_authtok_get.so.1 debug sshd-kbdint auth required pam_unix_cred.so.1 debug sshd-kbdint auth binding pam_unix_auth.so.1 server_policy debug sshd-kbdint auth required pam_ldap.so.1 debug
ldapclient list
If anything goes wrong your could do the following:
- Use ldapsearch -v to make you sure you could setup a TLS session with your LDAP server successfully.
- Enable PAM debugging and check the logs. To do that just run “touch /etc/pam_debug”, edit /etc/syslog.conf and add a new line (if it doesn’t already there of course):
*.debug /path_to_where_you_want_to_store_debug_log
And restart syslog with “svcadm restart svc:/system/system-log:default”.
- Analyze the logs on your LDAP server.
- Switch off TLS and try to sniff the traffic with snoop to make sure your ldap client sends reasonable queries.
-
Have fun and happy tinkering!
TAO: Facebook’s Distributed Data Store for the Social Graph
This post will be very succinct. Just a single link to a publication of TAO’s details at USENIX and a quote from the abstract part of the paper to catch your eye:
We introduce a simple data model and API tailored for serving the social graph, and TAO, an implementation of this model. TAO is a geographically distributed data store that provides efficient and timely access to the social graph for Facebook’s demanding workload using a fixed set of queries. It is deployed at Facebook, replacing memcache for many data types that fit its model. The system runs on thousands of machines, is widely distributed, and provides access to many petabytes of data. TAO can process a billion reads and millions of writes each second.
Definitely worth reading.
Apple WWDC 2013
In no way being another Apple deaf and blind fanboy, though I prefer to use Apple products, I must admit, after watching the keynotes, that it was an astonishing and fantastic show. And I truly concur with the audience when it stood up to welcome new iOS7 and to share their emotions. Hats off to all engineers, designers and everyone who has been involved to make all those new products to shine. Thank you!
When I met Anthony Kiedis
Back in 1999 met Anthony Kiedis on one the most famous Moscow street – Old Arbat Street. And the result of this short meet up you could see below:
Simple Python scp wrapper
Inspired by the following article I decided to write a simply python scp wrapper which is based on the ideas taken from the perl script also mentioned on the same page.
The whole point of this wrapper is dead simple: to allow certain users scp into their specific directories with out granting full ssh access.
The configuration file is straightforward:
/opt/etc/scponly.conf
scponly=/tmp:/home/scptest
Below is the wrapper in all its amature (must admit I’m not a developer at all) glory ;-)
/opt/bin/scp-wrapper.py:
#!/usr/bin/python
import sys
import os
import pwd
from subprocess import call, Popen, PIPE
import re
def fail(msg):
"""(str) -> str Prints error message to STDOUT
>>> fail('Error')
Error
>>> fail('Wrong argument')
Wrong argument
"""
print msg
sys.exit(1)
def access_verify(user, dirto):
"""(str) -> Boolean Returns TRUE iff user is allowed to scp
"""
if user in users:
for d in users[user]:
print user, dirto[-1:], d.rstrip()[-1:], dirto[:-1] == d.rstrip(), dirto == d[:-1].rstrip()
if dirto == d.rstrip():
return True
elif (dirto[-1:] == "/" or d.rstrip()[-1:] == "/") and (dirto[:-1] == d.rstrip() or dirto == d.rstrip()[:-1]):
return True
else:
return False
return False
if __name__ == '__main__':
users = {}
conf = "/usr/local/etc/scponly.conf"
# Reading configuration file
fp = open(conf, "r")
for line in fp.readlines():
record = line.split("=")
users[record[0]] = record[1].split(":")
fp.close()
command = sys.argv[2]
scp_args = command.split()
if scp_args[0] != "scp":
msg = "Only scp is allowed"
fail(msg)
if scp_args[1] != "-t" and not "-f" and not "-v":
msg = "Restricted; only server mode is allowed"
fail(msg)
destdir = scp_args[-1]
if not os.path.isfile(destdir) or os.path.isfile(destdir):
destdirv = os.path.dirname(destdir)
else:
destdirv = destdir
uname = pwd.getpwuid(os.getuid())[0]
if not access_verify(uname, destdirv):
msg = "User " + uname + " is not authorized to scp to this host."
fail(msg)
else:
scp_args.pop(0)
if len(scp_args) == 2:
call(["/usr/bin/scp", scp_args[0], destdir])
elif len(scp_args) == 3:
call(["/usr/bin/scp", scp_args[0], scp_args[1], destdir])
Just create a new user and set its shell (-s option in useradd) to /opt/bin/scp-wrapper.py. Hope it helps.
Enjoy and have fun!
OpenLDAP do_syncrep retrying attempts
Do you observe the error messages on your Linux OpenLDAP replica or master server similar to the ones listed below:
May 16 12:05:21 ldapserver1 slapd[5420]: do_syncrep2: rid=005 (-1) Can’t contact LDAP server
May 16 12:05:21 ldapserver1 slapd[5420]: do_syncrepl: rid=005 rc -1 retrying (4 retries left)
May 16 12:05:21 ldapserver1 slapd[5420]: do_syncrep2: rid=002 (-1) Can’t contact LDAP server
May 16 12:05:21 ldapserver1 slapd[5420]: do_syncrepl: rid=002 rc -1 retrying (4 retries left)
May 16 14:05:27 ldapserver1 slapd[5420]: do_syncrep2: rid=005 (-1) Can’t contact LDAP server
May 16 14:05:27 ldapserver1 slapd[5420]: do_syncrepl: rid=005 rc -1 retrying (4 retries left)
May 16 14:05:27 ldapserver1 slapd[5420]: do_syncrep2: rid=002 (-1) Can’t contact LDAP server
May 16 14:05:27 ldapserver1 slapd[5420]: do_syncrepl: rid=002 rc -1 retrying (4 retries left)
May 16 16:05:32 ldapserver1 slapd[5420]: do_syncrep2: rid=005 (-1) Can’t contact LDAP server
May 16 16:05:32 ldapserver1 slapd[5420]: do_syncrepl: rid=005 rc -1 retrying (4 retries left)
May 16 16:05:32 ldapserver1 slapd[5420]: do_syncrep2: rid=002 (-1) Can’t contact LDAP server
May 16 16:05:32 ldapserver1 slapd[5420]: do_syncrepl: rid=002 rc -1 retrying (4 retries left)
If yes and these messages seem to pop up every two hours then you might consider updating the following sysctl parameters:
net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes
Where:
- net.ipv4.tcp_keepalive_time – How often TCP sends out keepalive messages when keepalive is enabled. Default: 2hours.
- net.ipv4.tcp_keepalive_intvl – How frequently the probes are send out. Multiplied by
tcp_keepalive_probes it is time to kill not responding connection, after probes started. Default value: 75sec i.e. connection will be aborted after ~11 minutes of retries. - net.ipv4.tcp_keepalive_probes – How many keepalive probes TCP sends out, until it decides that the connection is broken. Default value: 9.
Hopefully that would make your OpenLDAP replication more reliable.
How the SCP protocol works
That’s not my title but the one I stole from a fantastic Jan Pechanec’s blog post that describes how the SCP protocol works – https://blogs.oracle.com/janp/entry/how_the_scp_protocol_works
I really encourage everyone to read it.
In: FreeBSD, HP-UX, Linux, Solaris
How to create new thin devices using symcli and add them to a masking view and a storage group
Below are the steps that I used recently to create and present a few thin devices. More precisely:
- Boot disk – 50GB
- Two TDEVs 150GB each. One for data and the other for logs.
- Create a nested initiator group:
- Show initiator group’s details:
- Create storage group:
- Create thin devices and bind them to a pool:
- Add new devices to our storage group:
- Create a new masking view and assign LUN IDs to the devices:
- Add third device to the storage group:
- Associate the storage group with a FAST VP policy:
- View the details of newly created masking view:
symaccess -sid VMAX-SID create -name IG_SERVERNAME_HBA0 -type initiator -consistent_lun symaccess -sid VMAX-SID -name IG_SERVERNAME_HBA0 -type initiator -wwn 50060b0000c30e38 add symaccess -sid VMAX-SID create -name IG_SERVERNAME_HBA1 -type initiator -consistent_lun symaccess -sid VMAX-SID -name IG_SERVERNAME_HBA1 -type initiator -wwn 50060b0000c30e3a add symaccess -sid VMAX-SID create -name IG_SERVERNAME -type initiator -consistent_lun symaccess -sid VMAX-SID -name IG_SERVERNAME -type initiator -ig IG_SERVERNAME_HBA0 add symaccess -sid VMAX-SID -name IG_SERVERNAME -type initiator -ig IG_SERVERNAME_HBA1 add
symaccess -sid VMAX-SID show IG_SERVERNAME -type initiator -detail
symaccess -sid VMAX-SID create -name SG_SERVERNAME -type storage
symconfigure -sid VMAX-SID -cmd "create dev count=1, size=50GB, emulation=fba, config=TDEV, binding to pool=Pool1;" prepare symconfigure -sid VMAX-SID -cmd "create dev count=1, size=50GB, emulation=fba, config=TDEV, binding to pool=Pool1;" commit symconfigure -sid VMAX-SID -cmd "create dev count=2, size=150GB, emulation=fba, config=TDEV, binding to pool=Pool1;" prepare symconfigure -sid VMAX-SID -cmd "create dev count=2, size=150GB, emulation=fba, config=TDEV, binding to pool=Pool1;" commit
Don’t forget to record the device IDs. In my case they were the following: 016B (50GB),016C (150GB) and 005F (150GB) respectively.
Adding disks to the storage group and further masking require additional explanation. As all we know, boot device must be lun 0. But you can’t assign a specific LUN ID to every new device in one go during the masking step since there is only a single option, i.e. -lun, which tells the starting LUN ID. BUT, and that’s important, this option, -lun, would assign the starting LUN ID to a device with the smallest ID. And in my case that would be TDEV 005F and that wasn’t what my task was all about. To overcome that hurdle, I firstly added devices 016B (boot disk) and 016C (data disk) and masked them to a storage group with -lun 0 options. By doing that, I could guarantee that my boot device would be assigned LUN 0 (obviously 016B (hex) is smaller than 016C (hex)). After that all I had to do is to add the third data device to the storage group and it would be automatically assigned the next available LUN ID.
symaccess -sid VMAX-SID -name SG_SERVERNAME -type storage add devs 016B,016C
symaccess -sid VMAX-SID create view -name MV_SERVERNAME -sg SG_SERVERNAME -pg PG_NAME -ig IG_SERVERNAME -lun 0
symaccess -sid VMAX-SID -name SG_SERVERNAME -type storage add devs 005F
symfast -sid VMAX-SID -fp_name FAST_VP_NAME associate -sg SG_SERVERNAME
symaccess -sid VMAX-SID show view MV_SERVERNAME
Job done!