Today, people rely on IT services around the clock, but how do you keep them running despite hardware faults and software upgrades? The answer lies, surprisingly, in a line from the movie Contact: "why build one when you can have two for twice the price?". By adding a redundant server, you can boost your uptime dramatically, and with some neat Linux features, you don't even need to double your costs.
Whether you're running a web farm with a thousand servers or a home file server on a couple of old desktop boxes, the idea is the same: running your applications on redundant servers helps you ensure service to your users despite server crashes or scheduled maintenance.
The main drawback to redundancy is cost: the extra server hardware costs money, and the added complexity increases maintenance requirements. It's not all bad, though - compared to a single high-end server and its expensive rapid-response support contract, a pair of cheaper redundant servers might not cost much more overall.
The biggest potential cost is in storage hardware. There are some tasks that don't need to store data: serving static web pages, for instance, is fine as long as each server has its own up-to-date copy of the pages.
Most other applications need some kind of storage though, and if you don't want your redundant servers to go to waste, you need an equally reliable shared storage system. Traditional solutions can cost tens of thousands of dollars, but Linux has a trick up its sleeve to bring that cost right down.
Sharing the love
The 'enterprise' solution for storage is a Storage Area Network device, or SAN, which puts a bunch of hard drives in a highly redundant enclosure, with high-speed Fibre Channel links to your servers. Even a low-end SAN can cost around $10,000 though, and cheap SANs often have lousy performance.
The low-cost Linux alternative is the Distributed Replicated Block Device, or DRBD (www.drbd.org). When set up on a pair of servers, it sits between the filesystem and hard drives on one of them (the 'primary' server), automatically replicating any disk changes over a network link to the other (the 'secondary' server). Only the primary server can access the data on the drives, but if it fails, the secondary can take over the primary role and gain access in its place.
The beauty of DRBD is that it doesn't require any expensive hardware - the only thing you really want is a direct, dedicated network link between the two servers to run the DRBD replication. Gigabit ethernet adapters with a cross-over cable are a perfect cheap solution: most machines have gigabit ethernet as standard these days, and if not, a pair of adapters can cost less than $100.
Set up DRBD
Once the hardware is in place, follow these steps to set up DRBD:
1. Configure the underlying storage on each system. My systems run LVM, so I just created a new LVM volume called "drbd" on each system, but a partition or dedicated hard drive would work just as well. They'll need to be the same size, though, for obvious reasons.
2. Install the DRBD software, which consists of a kernel module and the user-space configuration tools, on each system. Ubuntu 8.04 ships with DRBD as part of the standard kernel, so I just had to install the tools:
sudo aptitude install drbd8-utils
3. Create the "/etc/drbd.conf" config file and configure your DRBD device. This file must be identical on both systems:
common {
protocol C;
}
resource mydrbddrive {
syncer {
rate 50M;
}
on server1 {
device /dev/drbd1;
disk /dev/ubuntu/drbdtest;
address 10.197.10.1:7789;
meta-disk internal;
}
on server2 {
device /dev/drbd1;
disk /dev/ubuntu/drbdtest;
address 10.197.10.2:7789;
meta-disk internal;
}
}
You'll have to change the server names, disk devices, and IP addresses to match your own configuration. If you're running a dedicated network for DRBD, make sure you use those IP addresses here.