Linux without (?) Windows

"Microsoft seems to have gotten a lot of mileage out of the C2 rating for NT with no network connection. I wonder if a B3 rating for Linux with no power cord might be of value."    — Michael Driscoll on the kernel mailing list.
On this page:

Ubuntu install

I've been using Linux professionally for quite a while: first SGI IRIX machines in 2000, then custom-built RAID PCs with Linux, then Mosix clusters, then driver development which is much easier on the Linux kernel than on modern versions of Windows, and finally full fledged embedded Linux systems where every bit of the bootloader and operating system is under my control and not under the control of Bill Gates. But there remains the issue of my home computer with its 10 year history of upgrades and cruft.

Why would I want to change OS on a system that works fine with XP ? Well, really I don't. Except that Bill gates in his infinite wisdom decided that if I change too much hardware he will deny the license. I wanted to change the motherboard/processor/memory combo but was afraid to do so. Also the main system disk was always on its knees on its olden days 80Gb, never mind the other 2Tb of disk inside the unit on which I cannot just 'move' the operating system.

So when the system drive had a major brain fart one day in early 2009, I decided to skip a ski-mountaineering trips to jump ship and install kubuntu on a new hard drive, import my data, and then try to figure out how to replace some of my faithful applications and Windows-only hardware. The installation of Linux was basically painless, with only two major glitches: the lack of a proper graphics mode in X for my monitor and the failure of the WiFi.

So I had to give up on my Belkin DWL-G122 wireless which I replaced with faster and more reliable Buy at Amazon.comPLC (power Line Carrier) units. No driver necessary. Anyway, the wifi did work fine 2 years ago but now with the ubiquity of wireless ADSL modems, there are more than 15 competing networks in range, leading to crap performance. A note of importance to all those with PLC devices: don't plug them into surge protected power strips. Those will filter out the high frequencies used to transmit leading to abysmal performances.

Among the first things I noticed is that the keyboard is too fast, I often type dooublee characters. I briefly saw a kcontrol tab to take care of it, but it disappeared forever after the first sytem update (while clearly visible on other identical systems). Now all I have is this flimsy "xset r rate 200 30"

Knode for usenet, Kmail for email and Kontact instead of wab take some getting used to after using outlook express for 15 years. Thunderbird had a bug where links would not be clickable.




[LinuxKillBill.jpg]

Imaging software

The main reason I held onto Windows for so long was my photo processing software: Paint Shop Pro, PTgui, SilkyPix, HS-V3, monitor calibration, etc... It takes years to get the most out of those programs.

So 2 years after the thorough review of raw processing software I did, I tried out the offerings for Linux. In short they are sub-par and I want my Buy at Amazon.comSilkyPix back. I installed it both in Wine (a Windows API implementation under Linux) and in VirtualBox (a virtual machine that runs a WinXP install simply). Both have pros and cons: Wine is faster but has some user interface hiccups. I did the same for PTgui and installed Paint Shop Pro in VirtualBox.

For scanning, the linux version of VueScan is flawless with my Nikon Coolscan 4000 slide scanner. But not so on the Epson Perfection V500. After reading quite a bit, I downloaded the DEB 32bit package [libltdl7] files from http://www.avasys.jp/ and installed them with sudo dpkg --force-architecture --install iscan*386.deb, never mind that I'm on a 64-bit system.

But for monitor and printer calibration, the expensive and shitty Buy at Amazon.comDataColor equipment is far from working. The software installs but won't run under Wine or VirtualBox. Linux won't recognize the Spyder probes by default, here's how to get it to work.

I had kept my previous monitor profile which can be used simply in Linux with the command "xcalib Profile.icm", but it took me a while to find a way to generate new ones (if I change graphic card or monitor for instance). Also the printer profiles generated work fine as long as I print under the virtualized WindowsXP, but I have found absolutely no way to use them in Linux.

About VirtualBox, this system is awesome as it can pass an arbitrary USB device to the virtual OS, even if the Linux host doesn't recognizes it ! For instance I can use my Buy at Amazon.comEpson R1800 under Linux, or pass it to Windows when I need to make a profiled print. I also (try to) use the spyder probes through VirtualBox. Make sure to set the number of processors before you install XP, as it doesn't like a change in the number of CPUs afterwards.

Still about VirtualBox, how do you mount a .vdi file on Linux ? So you can for instance run a virus scan on it (all this must be prefaced with sudo): aptitude install qemu-utils; modprobe nbd; qemu-nbd -c /dev/nbd0 Machine.vdi; fdisk -l /dev/nbd0; mount /dev/nbd0p1 /mnt; clamscan -ir /mnt; umount /mnt; qemu-nbd -d /dev/nbd0

Finally about graphics, never mind what you can find on some forums, I could not get my recent Hercules webcam to work. But a new Buy at Amazon.comLogitech QuickCam Pro 9000 worked out of the box. If going shopping, pick from this list.


Disk encryption

Disk encryption methods

There are several ways to do disk encryption on Linux:

Hardware level disk encryption
This should be the preferred method as it's OS agnostic. All you need is an encryption-capable hard drive and a BIOS that recognizes it. Most DELLs now come with that option, but apparently the largest disks don't have it and it requires UEFI boots which mess with Linux.
LVM partition encryption
Here the /boot partition asks for your passphrase before mounting your swap and root partitions. I will detail the installation method below.
Home directory encryption
This is available from the Ubuntu alternate install disk but I haven't tried it. Could be useful for servers with different users.
External disk or partition encryption
That's the method to use with TrueCrypt if you want large encrypted stores of data.
Partition inside an encryted file container
That's the standard way to work with TrueCrypt for small storage (for instance on a USB key).

Installing a new encrypted system

Where I work we are now required by law to encrypt all laptops. Since hardware-level disk encryption wasn't available to me, I used an LVM container. Very easy to install, with some caveats.

Moving a TrueCrypt container from Windows to Ubuntu

On Windows I had an NTFS disk encrypted by TrueCrypt at the device level. I failed to mount it on Linux directly. Ubuntu would mount new drives via TrueCrypt, but not this one, saying it was corrupted with the very unhelpful message: "$MFTMirr does not match $MFT (record 36)". Mounting it temporarily on Windows, doing "chkdsk /F" on it then allowed me to mount it on Linux and copy its content to another encrypted disk, this time using Linux native ext3 filesystem.

Doing a backup and then data extraction of my failing former Windows disk was easy: dd if=/dev/sda1 of=WinSysDisk.img bs=1M conv=noerror. Then I can just mount that file as an ntfs partition and copy its content over.




[SuckWindows.png]

Servers

One of the reasons I wanted to move to Linux was the ease of installation. Yes, times have changed ! On Windows you install the OS and reboot. Then install the security updates and reboot. Then the service packs and reboot. Then the various system update and reboot. Then the secondary updates and reboot. Then at this stage you still you have a unusably bare system and you need to google for all the current versions of your usual software, download them, install, reboot, launch, customize. Damn, that license doesn't work anymore with the new version, etc... Usually it takes me a good 3 days. I've wasted months of my life installing windows and associated software according to this method.

Linux makes it all so much simpler: you install the OS and reboot. The installer pops up with updates; apply and reboot. So far, not much difference. Then you launch the application installer and select all those you want. Click OK, that's it, no reboot necessary. And each time there's a new version of either the OS or the various applications, the updates are automated and all together. Magic.

For years I had a Buy at Amazon.comLAMP stack running on Windows: Apache web server, Php, MySQL server, PhpMyAdmin, gallery2, svn, etc... All very delicate to install and keep up to date. On Linux you just click them in the installer and then all you need to do is fine tune their config files according to your needs.

My former svn repositories required a bit of massaging to move to Linux. I had to run the following on each of them: svnadmin recover /path/to/repos and just ignore the errors. I miss TortoiseSvn though, although KdeSvn is not bad and learning the command line for basic stuff is no biggie.

By the way, it is a good idea to pass the /etc directory under version control. All your critical system configuration files are there and you never know if a silly upgrade will break things. For this you first create an etc repository in your svn server directory: svnadmin create /var/www/SVN/etc (as www-data, not as root). Then you create an empty etc directory somewhere (not in '/') which you check out as root: cd /tmp; sudo svn co http://localhost/svn/etc. You then move the .svn to the real /etc and delete the temporary one: sudo mv /tmp/etc/.svn /etc/; sudo rmdir etc; cd /etc; sudo svn add ...; sudo svn commit when you can add whatever /etc files you fancy under version control. Always remember to sudo before svn'ing anything in /etc

Installing the ssh server is a no brainer, just remember to NAT port 22 on the router.

I got VNC to work in order to remotely access my Ubuntu box the following 4 steps:

  1. From the local system, log into the remote ubuntu box with a ssh tunnel on the VNC port (here I also have a X tunnel in case I want to use both VNC and X): local $ export DISPLAY=localhost:0.0; ssh -XCL 5900:localhost:5900 user@remoteubuntubox.dnsalias.com This works also from Windows/cygwin.
  2. If it's not already installed on the remote system, install x11vnc: remote $ sudo aptitude install x11vnc
  3. Run the VNC server telling it to accept connections coming from localhost (remember, it's being tunneled through ssh, so you don't need to mess up your security by opening a NAT port in your router): remote $ x11vnc -safer -localhost -nopw -once -display :0
  4. The server is all set, now just start a VNC client on your local machine (give 'localhost' as the IP/hostname of the machine to connect to, port 5900), TightVNC client works great on Windows. If on another Ubuntu box, you can type local $ vncviewer -encodings "tight" localhost:0

nVidia drivers

Here's just the summary of what needs to be done when you do a sudo aptitude full-upgrade, receive a new kernel and then end up with a text terminal because you broke the nVidia driver kernel integration. First you must have downloaded a recent NVIDIA driver, which may be hard to do in text only.

$ sudo service kdm stop            # or "gdm" for Ubuntu)
$ sudo apt-get update
$ sudo apt-get install linux-headers-$(uname -r) build-essential
$ sudo sh NV{Tab-to-complete}      # Wherever the nVidia installation file is
$ sudo service kdm start

Formatting a large drive (≥3Tb)

If you use fdisk, the default is still to use DOS (!) partition tables, limited to 2Tb. So use GParted, then go into [Device][Create Partition Table] and select [gpt] from the list. Then proceed as usual.


Office stuff

By now everybody knows that OpenOffice is a perfect replacement for MS Office: it's free and it doesn't have the horribly messy ribbon menus of Office2007. And it can open tab separated .csv files !



Changing motherboard

So shortly after installing, I tested two of the reasons why Linux can beat Windows to a pulp. First one is a complete motherboard+processor+memory change. I had a Tyan mobo with Athlon 64 X2 with the AMD64 version of Ubuntu, so I had to stay in that line. I got a different brand of mobo (Buy at Amazon.comAsRock), Buy at Amazon.commore cores and 4 times the amount of Buy at Amazon.commemory. Assemble, put the old drive in... and it boots without even a tweak of the BIOS or GRUB. And it doesn't bring up a message telling me that either I copied Windows illegally or that I changed too much hardware, either way pay up, thank you bitch.

The only hitch is the ethernet driver that no longer works. I have to compile a slightly different version, which took all of 5 minutes, but then the next kernel upgrade fixed the issue... So now it's a non issue.

The second test was upgrading the hard drive. This time I used a Knoppix live CD to boot, used dd to copy one partition to the next drive and then enlarged the partition to fill the whole drive. Rather painless.



[PenguinMachineGun.jpg]

Various annoyances

What is the command line to get the content of the clipboard ? Or to pipe something in it ? In cygwin I used it very often, for instance to sort the content of the clipboard getclip | sort | putclip I'm aware that there are two different clipboards in Ubuntu, the Ctrl-C/Ctrl-V KDE facility and the select/middle-click ancient X-clipboard, so if they are two different commands, that's OK... but I couldn't find them !

Couldn't get KDevelop to compile even a simple hello world. I'll stay with makefiles for now.

The sound works fine, including the multimedia keys of my old Microsoft Multimedia Keyboard... somehow. The volume up and down keys would raise the PCM volume used by most apps, but Amarok uses the separate Master channel. The solution is to go in [System settings][Keyboard & Mouse][Global][Kmix] and to change the key assignment from [Decrease Volume] to [Decrease Volume - Master ...]. There may be more than one choice, try them out. Do the same thing for the [Mute] key.

The default sort order of konqueror/dolphin file manager makes no sense to me: it sorts A, _B and C in that order, instead of the correct _B, A, C. I can find no information on what characters are ignored or treated peculiarly. Fortunately they recently added the ability to avoid this 'natural sorting of items' in [Settings][Configure Dolphin][General][General]

I write daily in 3 languages but I prefer a US keyboard for programming. Unlike Windows there's no magical set of keys that will type any char: on any Windows box since at least version 3 I can do Alt+135 for a 'ç' and it will work on any keyboard setup and any installed language. On Linux I can define custom actions to bring a specific char, but it needs to be configured on every PC... I can add foreign keyboards and try to remember the key positions, sell my firstborn for an Optimus keyboard, or use the more useful 'US'intl' keyboard for accents. I don't like the Intl keyboard much because you need to use the mouse to activate the keyboard, and also some keys are missing, like the 'ç'. Besides using intl keyboard, I found that the best solution is to use the Caps Lock key as a Compose key. You go in [System Settings][Regional & Language][Keyboard layout][Advanced][Compose key position] and select Caps Lock. Now when you press [Caps lock] at the same time than ` and then e, you'll get a è, same thing with [Caps lock][,] followed by C which'll give a Ç. It works with various combinations of ' " , . ` ^ ~ as first character and just about anything as a second character. You get the picture. See below for more keyboard tricks.

Finding hardware devices can be daunting. Here's a list of commands that do it in different ways. Some of those need you to be root and some reside in /sbin/ and some are not installed by default:

lshw
List hardware
lshal
List Hardware Abstraction Layer devices (currently being phased out)
lspci
List PCI cards
lsusb
List USB devices
lsscsi
List SCSI devices
lsblk
Show block devices in tree form
systool
Part of sysfsutils
hwinfo
For instance use sudo hwinfo --framebuffer to get the vga codes of your video card to pass to the kernel as vga=xxx in /boot/grub/menu.lst
fdisk -l
List hard drives
dmidecode
Get info from bootloader
dmesg
Boot messages, check this if you can't find a device that you expect
lstopo
Topography of processor(s), caches and memory (better than lscpu)
cat /proc/cpuinfo
Processor information
cat /proc/meminfo
Current memory status
lpstat
Printer information

Other experimentations

As long as I'm doing big changes, I might as well keep going. After all, KDE is a window manager not unlike Windows ! There are other window managers that operate on completely different principles. I tried 'Awesome' which is an interesting concept... except that with the utter lack of a tutorial, I couldn't even figure out how to close an application, or hide it, or fullscreen it, or anything except drop to a console and kill it.

Some very useful software:

gdmap
File space viewer.
gpsprune
GPS trace cleaner, very simple and nice but doesn't allow for moving trackpoints (should be in future versions).
viking
GPS trace editor, very buggy.
qtpfsgui
HDR software
rawtherapee or darktable
Raw image processing
soundconverter
Obvious from the name.
picard
ID3 tagger for music files
petit
log file analyzer, for instance petit --hash --fingerprint /var/log/auth.log
pitivi
Video editor
disper
Commutes easily between monitors: disper -S to use an external screen (for a presentation), disper -s to use your main screen, disper -e to use multiple monitors.
pdftk
Merges various pdf files: pdftk A=file1.pdf B=file2.pdf cat A1-3 B10-11 output merge.pdf
photorec
Use it to recover images from deleted/formated memory card. Use is very easy: plug in your card but don't mount it, use dmesg to see which partition it uses, like sdh1, launch sudo photorec -d /tmp/pr /dev/sdh1 and follow the menus.
unetbootin
Use it to create a bootable USB key from a CD .iso file. Install UnetBootin, download the Ultimate Boot CD ISO, use UNetBootIn to write it to a USB key, also copy your firmware flash executable on it, boot with it, select FreeDOS, type C:, run the flash utility.
shotwell
Photo manager and downloader from camera.
SIR
Simple Image Resizer
Smartshine
Photo luminosity adjustment
SmartDeblur
Image sharpening tool
Urban Lightscape
Enhances local luminosity.
Tintii
Color separation
Fotowall
Image disposition
Luminance-HDR
Creates high dynamic range images
PhotoPrint
Print multiple images

Shortcut keys in bash

Ctrl-L
Clears the screen, same as clear command
Ctrl-U
Clears the line before the cursor position
Ctrl-K
Clears the line after the cursor position
Ctrl-W
Delete the word before the cursor
Ctrl-T
Swap the last 2 character before the cursor
Esc+T
Swap the last 2 words before the cursor
Tab
Auto-completes command and files names. See complete command
Ctrl-R
Search through previous commands
Ctrl-C
Interrupts whatever is running
Ctrl-Z
Pauses whatever is running and puts it in the background. Usually followed by the bg command to let it continue running in background. Also see jobs and fg.
Ctrl-D
Exit the current shell, usually used as cat >>somefile[Enter]Type something[Ctrl-D]

Some useful scripts / one liners / bash functions

Put those in your .bash_aliases file.

Instant plot of piped numerical data
function PlotX { cat >/tmp/PlotX; gnuplot -e 'plot "/tmp/PlotX"' -persist; }
Test it with echo -e "10 12 15 16 19 21 20 24" | tr " " "\n" | PlotX
And example of use: ls -s1 | sort -n | sed -e "s/ / /g" -e "s/^ //" | cut -d" " -f1 | grep -v total | PlotX to get a histogram of file sizes.
Instant plot of X/Y numerical data
function PlotXY { cat >/tmp/PlotXY; gnuplot -e 'plot "/tmp/PlotXY" using 1:2' -persist; }
Test it with echo "2,10 4,12 5,15 8,16 9,19 11,21 15,20 16,24" | tr ", " "\t\n" | PlotXY
And for an example of use: cut -d" " -f10 access.log | grep "^[0-9]*$" | sort -n | uniq -c | PlotXY to plot file size vs number of downloads from a log file.
Instant file search in subdirectories
function S { find -type f -iname "*$1*" | grep -v .svn | sort | grep --color -i "$1"; }
For instance S 1941 or S "in the park"
Putting binary bytes in a file
echo -ne "\x00\x01\xFF" >file will put the bytes 0, 1 and 255 in file.
Check it by performing a hex dump with alias hd='od -Ax -tx1z -v'; hd file
Directory where the script itself is located
ScriptDir=$(cd "$(dirname "$0")"; pwd)
How to find all the images which have an exif rotation tag
find -iname \*.jpg | xargs jhead | grep "^File name\|^Orientation" | grep -B1 "^Orientation"
Then clear the rotation tags with jhead -norot File.jpg or, better, rotate and clear with jhead -autorot File.jpg
How to remove color ANSI sequences from a text output
sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g"

GCC tips and tricks

List default gcc options
gcc -Q -v
List all macros used by gcc
gcc -E -dD -xc /dev/null or echo | gcc -dM -E - -march=native
List all optimizations used by gcc at a given level
gcc -O3 -Q --help=optimizers | grep -v disabled
List default target-specific compiler options
gcc -march=native -Q --help=target
Find the architecture-specific options
gcc -### -E - -march=native 2>&1 | sed -r '/cc1/!d;s/(")|(^.* - )|( -mno-[^\ ]+)//g'

Getting offline Geoportail 1:25000 french maps on android phones

  1. Install mobac: sudo aptitude install mobile-atlas-creator
  2. Put the following Geoportail.bsh file in the directory ~/.config/mobac/mapsources/ which appears after the first run of mobac.
  3. Run mobile-atlas-creator, select [IGN Geoportail maps] in [Map Sources], select the zoom levels 15, 13, 11 and 9. 15 is the actual 1:25000, 16 would be the same as 15 but zoomed which you may want on very high resolution screens. Add a new Atlas in OruxMaps format, give it a name. Select an area by dragging on the map (start small, check the number of tiles that will be downloaded). Press [Add selection]. Press [Create Atlas] and wait a few hours.
  4. You can now find your tiles in the directory defined in [Settings][Directories][Atlas output directory]
  5. While you are in [Settings] you may want to limit the download speed.
  6. Now install OruxMaps on your android phone, run it once to create the storage directories, then connect your phone to your PC and copy the whole directory created above under /Internal/oruxmaps/mapfiles/...
  7. ...or if you want them on the external SD card, under /SD card/Android/data/com.orux.oruxmaps/mapfiles (or /SD card/Android/data/com.orux.oruxmapsDonate/mapfiles, depending on which version you have) and then open OruxMaps, go in Settings / Global Settings / Maps / Maps Directories, navigate to /SD card/Android/data/com.orux.oruxmaps/ and double-tap mapfiles to select it.
  8. Now when you start OruxMaps and zoom at one of those levels within the area, the map should appear.
  9. This discussion also has other .bsh files for Geoportail satellite images, coastal maps and cadastral maps.

Note that if the area you selected is too large (in terms of horizontal pixels), OruxMaps won't use the map. Bug ? Computing limitation ? I couldn't get the entire french Alps at once, had to divide them in narrower areas.


The Parent Experiment

A couple months after I started using Ubuntu as my main PC, I was in my parents home, upgrading their old computer. New mobo, new HD. Of course XP refused to restart after I clone the disk. So I start the painful install / configure / patch process. After 24 hours and everything nearly ready, SP3 crashes during its post-install. The whole install is now a brick. It's sunday afternoon and I leave in an hour...

I go in a rage, I put my Kubuntu install CD and after 30 minutes I have the full system and all its applications up and configured. I spend the rest of the allotted time to customize the system: obvious icons on the desktop to firefox, Redmond theme to ease the transition, kmail (with the old messages imported), Xsane for the scanner, OpenOffice, Dolphin, Gimp, my image import script and the few other apps they are likely to use such as ViewScan. I rename some of them to make it clear what they do or to the same name they had in Windows. I setup ssh so that I can login remotely to fix issues. And I leave, only telling them that with the new upgrade things will be a 'little different'. I cross my fingers on this understatement, waiting for the complaints calls to pour in.

After 2 weeks, I get the first call: "Yeah, it works fine but we don't have Skype anymore". Clickety click, ssh login, install. Done. "Is that it?". "Yeah, pretty much". I consider this experiment widely successful.