Using OpenSolaris SMF for managing MySQL Cluster

One of the things I really like about Solaris (and OpenSolaris) is the Services Management Framework (SMF). And one of the things I like about my job is the opportunity to learn and work with new stuff. And, of course, the best way to learn something "non-lethal" is to try it yourself and do something with it.

So, how about using SMF for managing a MySQL Cluster setup? Something simple like two data nodes, one mgmt node, and one MySQL server node running on one OS instance? Sounds like a good exercise and, from a quick search, I couldn't find where it had been done before. Loaded up the latest release of OpenSolaris (2009.06) into a Virtual Box VM, downloaded the latest tar ball of MySQL Cluster (7.0.5) for Solaris x86 32-bit, kicked off Pandora (gotta have music), and got started.

OK, OS installed and running, time to install Cluster. For simplicity, I put all MySQL related files and data in /usr/local/mysql/. This included creating the Cluster data directory at /usr/local/mysql/data_cluster. The only exceptions were the my.cnf and config.ini files, which were put in /etc/mysql/. All the files I reference are provided in the ndb_smf_files.tar.gz compressed tar file. (If you are already familiar with SMF and are just looking for the files, just grab the archive file above and you can go on your merry way.)

Rather than include the full text of the my.cnf file, I will just list the relevant changes:

[mysqld]

...

ndbcluster

ndb-connectstring = localhost

skip-innodb # not needed for my test system

...

[ndbd]

connect-string = localhost

[ndb_mgm]

connect-string = localhost

[ndb_mgmd]

config-file = /etc/mysql/config.ini

For the Cluster configuration file, I just copied the

$MYSQL_HOME/support-files/ndb-config-2-node.ini file to config.ini and changed the data directory parameters:

[ndbd default]

...

DataDir= /usr/local/mysql/data_cluster

...

[ndb_mgmd default]

DataDir= /usr/local/mysql/data_cluster

...

Now, onto the SMF setup. There are a pair of files for each service we want to manage: a service configuration file and a "scripts" file. For example, the Cluster management service (the ndb_mgmd process) configuration is described in the mysql_ndb_mgmd.xml file and the methods for starting and stopping the service are provided in the mysql_ndb_mgmd script. So, let's download the tar file, unpack, and move the configuration and script files to their respective directories. (For the steps below I prefer to open a root shell instead of continually invoking sudo but do whatever works for you.)

1. Grab the ndb_smf_files.tar.gz and extract the files:
   

   root# cd /tmproot# wget https://dl.dropbox.com/u/1880934/ndb_smf_files.tar.gz
   root# tar xzf ndb_smf_files.tar.gz

2. Create the /var/svc/manifest/application/database directory if it does not exist.
   

   root# mkdir /var/svc/manifest/application/database

3. Copy the three service configuration files to /var/svc/manifest/application/database:
   

   root# cd /tmp/ndb_smf_files/svcmanifests
   root# cp mysql*.xml /var/svc/manifest/application/database

4. A "request for action" script is similar to an "rc" script and is used for starting/stopping/restarting a service. Copy the three scripts to the /lib/svc/method directory and make sure they are executable:
   

   root# cp /tmp/ndb_smf_files/svcmethods/mysql* /lib/svc/method
   root# chmod +x /lib/svc/method/mysql*

5. Now time to get the manifests into the SMF repository. For this step, we use the svccfg utility to import the XML files::
   

   root# cd /var/svc/manifest/application/database
   root# svccfg import mysql.xml
   root# svccfg import mysql_ndb_mgmd.xml
   root# svccfg import mysql_ndbd.xml

6. Create the Cluster data directory.
   

   root# mkdir /usr/local/mysql/data_cluster
   root# chown mysql:mysql /usr/local/mysql/data_cluster
   root# chmod 750 /usr/local/mysql/data_cluster

7. If you are not familiar with SMF, this is a good time to take a look at the xml files. A few things to note are the dependencies defined, the methods that will be invoked to start and stop the service (which shows the relationship to the files we moved to /lib/svc/method), the method_credential (which user to use to start the service), and the properties defined (which can be referenced from the methods script).

For Cluster, there is a defined order of the processes that need to be started. Since the NDB Mgmt server controls the configuration of the Cluster, it must be started before the NDBD (data node) processes. And if you are using the MySQL server to access the Cluster, the data nodes need to be up and running before starting the mysqld process(es). So, for the data nodes (mysql_ndbd.xml file), we see:

which means that before the data nodes can start, the mgmt server must be up and running. In the MySQL server service configuration file, the dependencies are the network, file system, and the Cluster data nodes. From looking at the data node dependencies, just make sure that at least one of the two nodes is running:

Ok, everything should be ready to go so let's start things up. Start the Cluster mgmt server and verify it is up and running:

root# svcadm enable mysql_ndb_mgmd

root# svcs -l mysql_ndb_mgmd

In the svcs listing, you should see "enabled true" and "state online". If not, run "svcs -x mysql_ndb_mgmd" and check the log listed for the reason why (logs are in the /var/svc/log directory).

If everything looks good, start the data nodes and the MySQL server:

root# svcadm enable mysql_ndbd:node1

root# svcadm enable mysql_ndbd:node2

root# svcs -l mysql_ndbd

root# svcadm enable mysql

root# svcs -l mysql

That's it! Hopefully everything worked (and if not, that's what man pages and cursing are for). To stop a service, just "disable" instead of "enable".

Next thing to try is dynamically adding nodes to Cluster and seeing how SMF handles services on other servers.

-- Craig