24 hours to move all our web applications and service onto Ubuntu 10.04 TLS server from Centos 5.4. The move was stupidly fast and what’s more, Ubuntu’s got more cutting edge applications built in as standard – no more faffing with extra repositories just to get php5.3 installed…
Our core servers still run on Centos 5 – mainly because that’s what the DB and Application server are supported on (well, RedHat but there’s not much difference).
Our multiple web, application, database and monitoring servers are all now hosted on Amazon’s Elastic Cloud (EC2). EC2 gives us greater capacity and control than standard hosting, it’s cheaper, faster and more scalable. We no longer worry about hardware failures, disk space, memory, back-ups, electricity bills or bandwidth.
Centos 5.4 has done us proud for our secondary systems too (websites, hosting, mysql databases, monitoring) but I’ve been spending more and more time trying to get the latest versions of everyday programs installed, like php 5.3. The Centos 5 repositories don’t always include the latest versions for security / stability reasons so we were frequently compiling from source. Not ideal when you need to upgrade.
The switch to Ubnutu 10.04 TLS took only a few minutes. It took 5 minutes to fire up an instance in the cloud, update it and get the databases and files over from the previous server. Here are the steps I followed:
Launch Ubuntu Instance on EC2
There’s an Amazon AMI already setup and ready to go, pretty much out of the box. Using the command line tools, I started an instance, added a security group and added security credentials.
Once started, I quickly updated the software using aptitude, installed mysql server 5, php5.3, a few libraries and dependancies and rebooted.
Move MYSQL Databases and Websites
A simple mysqldump and import took care of the databases. I scp’d the dump and a tar of our sites over, untarred in the right place and went on to modify the http configuration files. Apache2 on Ubuntu’s slightly different to Centos but simpler to manage and faster (I think). I needed to strip out my virtual hosts from a single configuration file and create separate files in the ‘sites-available’ folder.
The a2ensite command makes enabling and disabling sites a blast – much easier than commenting out in the httpd.conf file.
Testing the sites, I found Apache’s mod_rewrite wasn’t enabled by default so ran:
a2enmod rewrite
I’d also forgotten to flush the priviledges on the MySQL db after importing so none of the sites could connect. Again, another simple fix.
Nearly 100 databases and websites (mostly WordPress these days) moved in 20 minutes.
Move IP address to new server
Amazon use Elastic IP addresses so I didn’t have to change all our dns entries. I simply disassociated the existing server and associated with the new one. A 10 second migration.
Checking through the log files for the first few minutes, I could see the following error appearing
Fatal error: Call to undefined function imagetypes()
A quick search revealed the GD library was required for WordPress and was missing. All I needed to do was run:
apt-get install php5-gd
Moving Webalizer History
Installed webalizer using apt-get again, moved history over from previous server and then merged the log files using:
cat newlog oldlog > log
I found our stats were out until I deleted the history files after merging the log files and then ran the command again.
Installing Request Tracker
Our support team use Request Tracker (RT) which is often a pain to install. The process was significantly faster on Ubuntu but I still had to wait for about an hour while it went off and found all required dependancies.
It didn’t take long to get it configured once installed and within two hours of firing the server up, we were in business.
Nagios and Cacti
Nagios took moments to install and all I needed to do was scp the /etc directory over to the new server. Although I could see all the servers and services we monitored, there were some strange errors appearing because of the plugins.
I wasn’t worried about moving the history files over because we use it as realtime monitoring rather than to provide historical data.
Getting Cacti installed was just as fast as Nagios – getting the latest version to run on Centos was a pain in the neck.
Then, moved the database over from another Ubuntu server (office based) as well as the /var/lib/rra folder.
Quickly realised that there were significant problems and we were losing data. Looking through the logs, I could see a couple of errors – mostly with depreciated functions because of the php version (5.3) we were running. There’s a fix coming in Cacti 0.88 so I supressed the errors in my php.ini file for the time being:
error_reporting = E_ALL & ~E_DEPRECATED
The issue with graphs not appearing was due to the x86 architecuture on the latest machine – we were migrating for a i386 non-server Ubuntu release.
In order to get this error sorted, I needed to export and import using the rrdtool. There’s a great command on this site for processing multiple files.
Graphs were in but strangely not collecting new data… Will get sorted.
Amanda Backup
As well as our webserver, this machine also deals with our nightly backups to s3 using Amanda. I thought this would be the hardest part of the migration but it was also quite straightforward. I downloaded the amanda server dpkg here, copied the configuration files over, ran amcheck, sorted firewall rules and was sorted in 15 minutes.
Daily Snapshots
One of the best parts of EC2 is their snapshot facility which allows you to create an instant snapshot of a server and then store out on s3. If there’s ever an issue with a machine, you can rollback in seconds. The only problem is that there’s no automation tools for creating and deleting these.
Using the ec2 command line tools, I wrote a simple bash script to create a snapshot of all our volumes twice a day. I wasn’t able to create one to automatically delete them after X days however. Digging around, I found one written in php that did the job and built this into my script. Thanks to Oren Solomianik’s for this one although it needed modification to support the eu-west region. In the end, I plumped for a modified version of the code found here.
That’s it…
Still need to sort Cacti out but it’s sucking in data at least.
