The 5 day Linux Upgrade
There comes a time in every mans life when he has to bite the bullet and tackle that problem which has been lingering for too long.
A long history of Linux
So it was with my Linux upgrade. I''ve been using various versions of Linux as the O/S on my main computer for years. Many, many years. I started using Linux at some point in the early nineties, when a work colleague passed me a clutch of 3½inch floppy disks with the words "Slackware 0.96 boot from disk #1" on them. Being a long-time Unix hacker since the mid-80s I gave it a try and liked what I saw - which, considering the only alternative was Windows 3.0 - is hardly surprising.
So, here I am nearly 20 years later, still running Linux (although I do have a couple of Windows PCs for all the professional needs). For some years I had been running a 64bit version of Ubuntu 8.04. I find I''ve been using it less and less, preferring the polish and seamlessness of Windows for tasks where the end result is more important than the journey. I also had a feeling that the 1TB disk that I''ve been running 24*7 had clocked up enough rotations (something like 10 billion) and could do with a preemptive replacement. However the desire to replace the set of applications with newer, (hopefully) more developed and better integrated, and faster versions that had been developed in the preceding 3 years was tempered by the gnawing knowledge that things rarely go as expected and that this upgrade, like all the ones before it, was bound to be much more complicated than it should be.
So it was, with a new 2TB disk in the hand of a courier standing at my front door, the battle commenced.
The installation
As usual, I burned an installation CD - this time of the latest Ubuntu: version 10.10 (for some reason, people who want to make it as hard as possible for professionals to recommend and for industry to accept, have given this version a cutesy name. I can''t remember what it is, but since it''s also longer to type than 10.10 I''m sticking to that.) With that, and my new disk installed in place of the old one - to remove any possibility of accidentally overwriting it - started the installation. This went smoothly, as did the network update of all the fixes that have been made available since the official CD image was created in 2010. After about an hour, I had a bootable, basic Ubuntu installation on my brand new disk.
64 or 32 bit
When I bought this machine I got an AMD64 (64bit) processor. At the time it seemed logical therefore to install a 64bit version of Ubuntu - boy, was that a mistake. Right from the start I felt the machine was a little unresponsive. Firefox took up an enormous amount of memory, which increased with usage (a typical memory leak) to the point where it was consuming over 700MB of RAM, as reported by the RES column in top. Since it was 64 bit, I discovered that flash content did not play and some drivers: for example to my webcam, did not work reliably. One of the main reasons I wanted to do this upgrade was to get rid of the 64bit Ubuntu - since it gave me no benefits with my 3GB of memory and seemed to have several insurmountable problems. This decision has since been shown to be good. I''m very pleased with the speed of Ubuntu 10.10 and with its small memory usage. Firefox never seems to go above 170MB and my swap area has yet to see a single paging event.
In order to create the new installation, I had decided not to allocate the entire disk at install-time. Consequently, I chose to manually set up the disk partitions and created a 400GB root partition and left the rest of the disk untouched, for later. I also did not create a swap partition, since I had 3GB of memory and would create a swap FILE on the root filesystem later on.
Once the basic system had been installed, I set about downloading and installing all the applications that I regularly use. By opting for a newer version of Ubuntu, I would also be able to upgrade the apps. One point about the "long term support" versions of Ubuntu (I was upgrading from 8.04LTS) is that the support only extends to the stuff you install off the installation media and Ubuntu''s own software repository. It doesn''t cover other applications. So if you want a newer version of an application you downloaded from elsewhere, there''s no guarantee that it will work with even a "long term supported" older version of Ubuntu. In fact, backwards compatibility seems to be the exception rather than the rule in the Linux world. So, a couple of hours later, I had new versions of all my old favourites installed. All, that is, except the large number of Perl modules that I use to create scripts and personal utilities.
Upgrading Perl
So far as Perl goes, there is an extensive library of user-written modules available. A lot of them are very high quality and well-integrated into the repository, known as CPAN. However, there is no easy upgrade path. While there are some techniques for auditing what modules you have installed locally, I found that it only discovered a small proportion of the modules I needed to upgrade. Further, there is a clash of cultures between the Perl way of installing modules (via CPAN) and the Ubuntu way, which uses the Package Manager. After getting into a huge mess trying to reinstall the gd graphics and TrueType modules from CPAN, and spending about 4 hours downloading and rebuilding, then rebuilding by hand, then manually having to find and download dependencies, then compile install and integrate them with the Perl build process, I gave up and used Ubuntu''s Package Manager to pick through the hundred s of available modules to select the ones I thought I needed and having it install them for me, from precompiled packages. After that, it was a rather hit-and-miss affair of running each of my Perl scripts and see which ones broke from requiring modules that I hadn''t installed, and using CPAN to install them by hand.
Video card drivers
It then turned out that my NVidia FX5200 graphics card did not come with the 3D acceleration features, necessary to get graphical applications working at anything like a usable speed. For example, there is an astronomical application called Stellarium which is extremely nice to use, but is very heavy on graphics. Without the proprietary extensions it ran as a frame rate so slow as to be unusable - literally 10s of seconds between screen refreshes. Apparently there is some anti-commercial principle that the creators of Ubuntu have a bee in their collective bonnets about, which deems the drivers provided by NVidia as "impure" and they therefore make every one who has an NVidia graphics card download and install these udates themselves. Sadly, there is a great deal of misinformation and incorrect default behaviour surrounding which particular version of the NVidia drivers work with which graphics card (The "173" drivers grudgingly supplied for Ubuntu didn''t work with my card). After another fruitless hour chasing entirely the wrong version: installing it, rebooting it, testing and debugging. Then checking that I had followed the instructions - and trying all the other equally uninformed suggestions, I stumbled upon a post in a support forum which led me to the correct version of the drivers - 96.43.19 - purely by chance. After a second hour spent getting my graphics performance up to scratch, I was able to run Stellarium at a usable, if not exactly screamingly fast rate of 20 frames per second. Hundreds of times faster than the speed I got with the Ubuntu supplied code.
By this time, I had most of my familiar applications up and working. I had spent about 8 hours (i.e all the free time after work, for 2 days running) getting back to what I had pre-upgrade. There were still some applications that I had not been able to transfer my old setting into (such as Lifera, the RSS feed viewer I use) as they used proprietary data formats. However, it was "good enough" - for now. It was not all stress and frustration. The jump in versions meant I was able to dump the bloated, quirky and (for me at least) randomly-crashing "xine/kaffeine" video and TV application for something a whole lot tighter and with more features and better usability: hello MeTV! [Update. The engagement is over! While MeTV has some nice features (such as the EPG). It is also annoying in that it chooses to record programmes in an incompatible format that Ubuntu''s Movie Player doesn''t understand. As a consequence playback continually nags to download a plugin, which it turns out doesn''t exist. This is just pure sloppiness, which although very common with free software, you''d have hoped somoen would have spotted when compiling the Ubuntu package. Oh well, anyone know a TV player that does "just work"?)
Disk weirdness
This is where things got a littler strange(r). It came time to transfer the bulk of all my data: media, Virtualbox instances, email archive, links and documents onto the new disk. The plan was to reinstall the old disk, create a second partition on the new disk and transfer all this stuff, including the hundred-plus user accounts I''ve created over the years. To this end, I fired up the Disk Utility from the System->Administration menu and proceeded to create a new, empty 800GB partition to supplement the 400GB one I had used for the new Ubuntu installation. [I prefer not to allocate all the disk space in one go, as that makes me lazy insofar as housekeeping goes and I end up with multiple versions of almost identical files, all over the disk and can never recall which ones were modified for which purposes.] Creating the new partition was quite easy, as was copying across the hundreds of GB of stuff I have accumulated to date. Strangeness started when I powered down the machine in order to remove the old disk as the first step to keeping it somewhere safe - for backup purposes.
After I took the old disk out and rebooted, the box hung. The BIOS recognised the 2TB disk, yet the boot process got as far as displaying the message: Verifying DMI Pool Data and went no further. No error messages, no disk activity - nada.
Swapping SATA channels made no difference. A look on the web threw up a load of (often conflicting) advice and information, some going back 10 years, about the possible cause. Most people considered the problem was corrupted data in the BIOS. My investigation of this possible cause took another hour or two of swapping ports, removing all other devices (CD drives,DVDs etc.), resetting BIOS settings and so on. Eventually I came to the conclusion that none of the solutions I had seen were addressing my problem, so I went back to basics. Booting my Ubuntu 10.10 install CD still worked, as did booting the old disk. So I started to focus my attention on the new disk - some sort of data or hardware fault, maybe? Running the install CD with the new disk plugged in got me to the point where I was asked if I wanted to "try" Linux or install it. The plan was to get into the "try" system and see if I could find any hardware or filesystem problems on the disk. However, selecting "try" caused the CD drive to spin for several more seconds, then halt - silently. It was doing exactly what my BIOS was doing, getting as far as accessing the new disk and then stopping.
Plan B, or was it C, or R or whatever, was to take the new disk and test it in another Ubuntu box. I added it as a second disk to an Ubuntu 9.10 installation I jut happened to have available. This machine booted it''s own O/S and I could see the new disk and the two partitions I had created. Running the Disk Utiity''s SMART disgnostics showed that the disk was healthy and had not had any issues in its 4.8 days of powered on life. Likewise, running "fsck" on it''s two partitions showed they were clean, too. Plan
At this point I had eliminated the old hardware: it booted the old disk and my 10.10 installation CD. The problem moved with the new disk to another box - although the disk itself had no errors and the filesystems were fine. What''s left? The only thing that came to mind was the booting process. All the stuff about "DMI Data Pools" now looked to be a red herring. I believe the BIOS was getting past that, but never got to the point of loading the GRUB bootloader. Yet more time on the web ended me up on Ubuntuforums.org and a post entitled:
The Grub Rescue Mode Megathread
and from there to a second piece of information ominously called
HOWTO: Purge and Reinstall Grub 2 from the Live CD
which in turn led to diagnostic script at Sourceforge to check out the bootloader installation.
Now we''re getting somewhere! Running this little gem of a script on the Ubuntu 9.10 machine with my new disk as a second drive as:
# bash boot_info_script055.sh
note: you must be root and you must use "bash"
informed me of the following:
Boot Info Script 0.55 dated February 15th, 2010
=============== Boot Info Summary: ====================
=> Grub 2 is installed in the MBR of /dev/sda and looks on the same drive in
partition #1 for /boot/grub.
=> No boot loader is installed in the MBR of /dev/sdb
Someone (err, me?) or thing had knobbled my new disk''s boot block but if you were to ask the obvious question, I could look you straight in the eye and say with all sincerity that "I hadn''t the faintest idea how that happened." I''ve done my time in Unix support and so I won''t bore you with the standard response of: "all I did was ... " because I never believed it when I heard it, and I have no expectation that you would, either.
Leaving aside the post-mortem, my focus is now on how to fix the problem. The referring site suggests reinstalling the GRUB bootloader with the script:dpkg-reconfigure grub-pc
This runs in a console window (i.e produces text outputs) and running it against the new disk in the Ubuntu 9.10 box - i.e. disk /dev/sdb gives rise to the report:grub-setup: warn: This GPT partition label has no BIOS Boot Partition; embedding won''t be possible!
grub-setup: warn: Embedding is not possible. GRUB can only be installed in this setup by using blocklists. However, blocklists are UNRELIABLE and its use is discouraged.Installation finished. No error reported.This is the contents of the device map /boot/grub/device.map.
Check if this is correct or not. If any of the lines is incorrect,
fix it and re-run the script `grub-install''.
(hd0) /dev/sda
(hd1) /dev/sdb
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.31-19-generic-pae
Found initrd image: /boot/initrd.img-2.6.31-19-generic-pae
Found memtest86+ image: /boot/memtest86+.bin
Found Ubuntu 10.10 (10.10) on /dev/sdb1
done
and a quick "dd" of the start of the disk indicates that the first few blocks have, indeed, been altered. So, off with the power, reconnect the new disk to it''s "home" machine and action
Yay!, progress. The box starts up, gets past the "Verifying DMI Pool Data" message and starts loading Grub. In fact, it gets as far as putting the letters G R U and B followed by a space on the screen and then locks up - solid. I have to hit the reset button, Ctrl-Alt-Del does nothing. [N.B. This has also had an effect on the Ubuntu 9.10 machine - it now waits at the Grub menu, for me to select which entry to boot from.]
With the new disk back in the Ubuntu 9.10 box and running the next suggestion: update-grub generates a grub
.cfg file. After a quick shutdown, swap-out and boot-up, this step made no difference at all.
Rerunning the info script from earlier, tells me why this failed:
=============== Boot Info Summary: ====================
=> Grub 2 is installed in the MBR of /dev/sda and looks on the same drive in
partition #1 for /boot/grub.
=> Grub 2 is installed in the MBR of /dev/sdb and looks at sector 618599 of
the same hard drive for core.img, core.img is at this location on /dev/sda
and looks for (UUID=e2641371-4c17-4c30-bf63-f24e48049ead)/boot/grub.
From this information it transpires that the Grub I installed has taken it''s config from the Ubuntu 9.10 machine. It''s looking for the root partition with that box''s UUID ( ... 49ead ) and its core.img file''s block number.
This episode did reach a satisfactory conclusion - whcih may or may not be due to a year-old, known bug in the Ubuntu 10.10 installation process. If you really have got hooked on this part of the story, the remaining gory details can be found on the Ubuntu support forum where I got some invaluable advice from some very helpful people.
Almost done
After that little drama - which took 2 days to resolve -I went back to beating my new installation into shape. Installing more new software packages, discovering that you can''t just copy Firefox''s config directory across and get it to preserve all your bookmarks, settings and add-ons (you need to use another add-on called FEBE) and updating VirtualBox to the latest version.
In conclusion
So here I am, 5 days after I started this upgrade. Most things are back up and working again (apart from Lifera, the RSS feed viewer I use - which I can''t import my old feeds into the new version). However that has taken a lot of skill, deduction, help and luck to get to this point. Without a lot of deep Linux knowledge and a second, compatible, machine to test things on I would have been stuck and my rubbish bin would have lots of bits of PC in it. Even so, this upgrade has taken all the spare time from 5 working days: some 30 hours, to sort out. At commercial rates (even excluding all the time taken to get my Perl modules sorted out) that puts the cost of this free software upgrade at many times the price of a Windows 7 license.
I''ll probably stick with Linux for the long term. However, even before I started this upgrade I knew it would not go smoothly - I just didn''t know how it would go wrong. That was the main reason I had put it off for so long and is the reason that I would absolutely NOT recommend anyone else to attempt Linux upgrades - unless they have a great deal of experience and a lot of time to spend on sorting it out. If ever there is a time to heed the advice If it ain''t broke, don''t fix it it''s with Linux upgrades