We test the big three OSes on level ground - Boot Camp.
A lot can happen in a month, especially with something as big as Apple’s x86 transition. We've looked at Boot Camp, the set of software and firmware updates from Apple that let Intel Macs run Linux and Windows with ease, even if installing them both alongside Mac OS X on the one machine is a little tricky. Since then, Apple has released its relatively low-cost MacBook, which gives you a 1.83GHz Core Duo CPU (just like the one in my MacBook Pro) and a 13-in widescreen LCD for just $1749. That base configuration comes with only 512MB of RAM, which simply isn’t enough for OS X, but even after an upgrade to 1GB of RAM you’ll still get change from your two grand.
Another nice launch last month was for a nifty boot loader, which can alleviate some of those triple-boot issues. Called rEFIt, it replaces the standard Apple boot selector with a more colourful boot menu that lists each installed OS with appropriate icons. As long as your Linux install has LILO installed in the MBR, rEFIt will display it at boot, avoiding the need to chain-load Linux through the Windows boot loader. It also contains a tool that can re-sync your MBR and GPT partition tables, so if you do accidentally mess things up while installing Linux, you can (hopefully) set things straight without resorting to a complete rebuild of all three OSes.
On top of all that, Ubuntu 6.06 (Dapper) is now out, and it works a treat on the MacBook Pro, and apparently on the MacBook as well. Dapper has a clever new installer system which is great for testing out Linux on an Intel Mac. The install CD is actually a full live CD, with a new GUI-based installer available from an icon on the desktop. With a triple-booting Intel Mac setup now easier and cheaper than ever, it’s sure to become a lot more common.
Mactel performance
Having the three major operating systems all installed on the one machine opens interesting possibilities for cross-platform performance testing. Comparing Linux and Windows is not unusual, given that both can be tested on x86 hardware, but now with Boot Camp, we have an equal platform to test Mac OS X too.
Now we can test all three systems on the one piece of hardware, which is something I’ve been wanting to do for a while. Many long-time Linux/UNIX users have argued that Mac OS X is quite shoddy as a UNIX platform, with Apple’s implementation of a microkernel architecture lacking in performance compared to Linux, or even FreeBSD, from which Apple borrowed quite a bit of code.
What aspects of performance can we test? Low-level tests of CPU performance and memory bandwidth are good, but a quality OS shouldn’t interfere with such tests. We need to find applications that stress parts of the OS itself, calling in to libraries and interacting with the kernel. The problem lies in finding examples that work equally well across the different platforms, especially when comparing UNIX-like systems to Windows.
We came up with one key application: Apache 2.0. Apache is an old school UNIX app for sure, but with 2.0 it’s built upon a platform-neutral foundation called the Apache Portable Runtime (APR), which helps ensure good performance on all platforms. The other important feature is its Multi-Processing Module (MPM) infrastructure, which lets you use different threading models, including models customised to specific operating systems.
Apache testing should put some strain on the CPU and memory bandwidth in the system, but above all it will stress-test the multitasking abilities of each operating system, which is particularly important when you consider the growing availability of dual-core systems, Intel Mac and others.
I installed Apache 2.0.58 and PHP 5.1.4 on all three OSes, building from source on Linux and Mac OS X and using the official binaries on Windows, and performed two tests: one using a small static text file, and another using a PHP page with a simple processing loop in it. The static file test should really spend more time in the kernel than executing actual Apache code, so it really pushes the OS, while the PHP test uses much more CPU power, showing how the OS stands up under significant load.
For good measure, I also ran a bzip2 test for simple single-threaded CPU performance, and a simple memory bandwidth test of my own concoction, though I didn’t expect to see any significant differences between the three contenders on these tests.
The bzip2 and memory bandwidth numbers bear out my initial assumptions about memory and CPU performance: the numbers are pretty close across the three systems. In both cases, we have two OSes returning very close results, with the third about 10 percent better. That’s not significant enough to read too much into, however.
The Apache results tell a different story though, with Linux about 40 percent ahead of OS X and more than doubling the results from Windows. The PHP numbers, with their greater CPU load, are much closer, though the podium positions remain unchanged. I’d love to call this a big win for Linux over Windows, but Apache on Windows is still relatively young, so I’m sure there’s plenty of room for improvement there.
However, that excuse doesn’t fly for Mac OS X, and given its BSD heritage I’d have hoped for better results in the static test. It did do quite well in the PHP test though, falling behind Linux by only nine percent. Hopefully future improvements in the networking stack and multithreading code can help bridge that gap.
In terms of desktop interactivity though, especially when dealing with command-line tools like compilers, OS X did subjectively feel quite a bit slower than Linux, and in general desktop operation it seemed to lag both Windows and Linux, though given the ambitious nature of the OS X GUI this is understandable.
I think we’ll leave the testing there for now, but don’t be surprised if I revisit this topic in future - performance testing has always held a certain fascination to me. It’s very hard to test things like interactivity and desktop performance in a repeatable, quantifiable manner, and even harder to determine how these and other factors impact on that big, intangible idea of productivity, but if I come up with something, you’ll be sure to see it here.