Automatically hardlinking duplicate files under /usr/share/doc with APT //at 20:43 //by abe

On my everyday netbook (a very reliable first generation ASUS EeePC 701 4G) the disk (4 GB as the product name suggests :-) is nearly always close to full.

TL;DWTR? Jump directly to the HowTo. :-)

So I came up with a few techniques to save some more disk space. Installing localepurge was one of the earliest. Another one was to implement aptitude filters to do interactively what deborphan does non-interactively. Yet another one is to use du and friends a lot – ncdu is definitely my favourite du-like tool in the meantime.

Using du and friends I often noticed how much disk space /usr/share/doc takes up. But since I value the contents of /usr/share/doc a lot, I condemn how Nokia solved that on the N900: They let APT delete all files and directories under /usr/share/doc (including the copyright files!) via some package named docpurge. I also dislike Ubuntu’s “solution” to truncate the shipped changelog files (you can still get the remainder of the files on the web somewhere) as they’re an important source of information for me.

So when aptitude showed me that some package suddenly wanted to use up quite some more disk space, I noticed that the new package version included the upstream changelog twice. So I started searching for duplicate files under /usr/share/doc.

There are quite some tools to find duplicate files in Debian. hardlink seemed most appropriate for this case.

First I just looked for duplicate files per package, which even on that less than four gigabytes installation on my EeePC found nine packages which shipped at least one file twice.

As recommended I rather opted for an according Lintian check (see bugs. Niels Thykier kindly implemented such a check in Lintian and its findings are as reported as tags “duplicate-changelog-files” (Severity: normal, from Lintian 2.5.2 on) and “duplicate-files” (Severity: minor, experimental, from Lintian 2.5.0 on).

Nevertheless, some source packages generate several binary packages and all of them (of course) ship the same, in some cases quite large (Debian) changelog file. So I found myself running hardlink /usr/share/doc now and then to gain some more free disk space. But as I run Sid and package upgrades happen more than daily, I came to the conclusion that I should run this command more or less after each aptitude run, i.e. automatically.

Having taken localepurge’s APT hook as example, I added the following content as /etc/apt/apt.conf.d/98-hardlink-doc to my system:

// Hardlink identical docs, changelogs, copyrights, examples, etc

DPkg
{
Post-Invoke {"if [ -x /usr/bin/hardlink ]; then /usr/bin/hardlink -t /usr/share/doc; else exit 0; fi";};
};

So now installing a package which contains duplicate files looks like this:

~ # aptitude install perl-tk
The following NEW packages will be installed:
  perl-tk 
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,522 kB of archives. After unpacking 6,783 kB will be used.
Get: 1 http://ftp.ch.debian.org/debian/ sid/main perl-tk i386 1:804.029-1.2 [2,522 kB]
Fetched 2,522 kB in 1s (1,287 kB/s)  
Selecting previously unselected package perl-tk.
(Reading database ... 121849 files and directories currently installed.)
Unpacking perl-tk (from .../perl-tk_1%3a804.029-1.2_i386.deb) ...
Processing triggers for man-db ...
Setting up perl-tk (1:804.029-1.2) ...
Mode:     real
Files:    15423
Linked:   3 files
Compared: 14724 files
Saved:    7.29 KiB
Duration: 4.03 seconds
localepurge: Disk space freed in /usr/share/locale: 0 KiB
localepurge: Disk space freed in /usr/share/man: 0 KiB
localepurge: Disk space freed in /usr/share/gnome/help: 0 KiB
localepurge: Disk space freed in /usr/share/omf: 0 KiB

Total disk space freed by localepurge: 0 KiB

Sure, that wasn’t the most space saving example, but on some installations I saved around 100 MB of disk space that way – and I still haven’t found a case where this caused unwanted damage. (Use of this advice on your own risk, though. Pointers to potential problems welcome. :-)

Finding libraries not marked as automatically installed with aptitude //at 17:26 //by abe

This is a direct followup on my blog posting Finding packages for deinstallation on the commandline with aptitude.

In the meantime on more alias for finding obosolete packages made it into my zsh configuration. It’s an alias to find installed libraries, …-data, …-common and other usually only automatically installed packages, which are not marked as being installed automatically nevertheless:

alias aptitude-review-unmarkauto-libraries='aptitude -o "Aptitude::Pkg-Display-Limit=( ^lib !-dev$ !-dbg$ !-utils$ !-tools$ !-bin$ !-doc$ !^libreoffice | -data$ | -common$ | -base$ !^r-base ) !~M"'

And yes, this pattern is slightly larger than those from the previous posting, so here’s the used filter in a little bit more readable way:

(
  ^lib
    !-dev$
    !-dbg$
    !-utils$
    !-tools$
    !-bin$
    !-doc$
    !^libreoffice | 
  -data$ | 
  -common$ | 
  -base$
    !^r-base
)
!~M

It matches all non-automatically installed packages whose name starts with “lib”, but is neither a debug symbols package, a development header package, a documentation package, a package containing supplied commands, nor a libreoffice package.

Additionally it matches all non-automatically installed packages ending in -data, -common, or -base, but excludes r-base packages.

Of course you can then mark any erroneously unmarked library by pressing “M” (Shift-m).

If you press “g” for “Go” afterwards and wonder why nothing to remove shows up, be reminded that the filter limit is active in this view, too. So press “l” for “Limit” and then Ctrl-u to erase the current filter limit of this view and press enter to set the new (now empty) filter, et voilà…

Hope this is of help for some others, too.

Finding packages for deinstallation on the commandline with aptitude //at 20:18 //by abe

Although I often don’t agree with Erich (especially if GNOME is involved ;-), he recently posted something on Planet Debian which I found very helpful.

I also own a netbook where disk space is sacre. It’s an ASUS EeePC 701 with just 4GB disk space. And it runs Debian Sid, so dependencies change often, leaving packages installed which formerly had hard dependencies on, but are now left with just recommendations pointing to it.

Quite a few times I asked myself if it’s possible to find those packages and if so, how to do it. Well, I don’t have to ask myself that anymore, since Erich recently posted the appropriate filter patterns for my favourite package manager aptitude for this task in his posting “Finding packages for deinstallation”. Thanks, Erich!

Since those filters aren’t very easy to remember, I’d like to extend the usefulness of his posting towards the commandline. I for myself added the following aliases to my shell setup:

alias aptitude-just-recommended='aptitude -o "Aptitude::Pkg-Display-Limit=!?reverse-depends(~i) ~M !?essential"'
alias aptitude-also-via-dependency='aptitude -o "Aptitude::Pkg-Display-Limit=~i !~M ?reverse-depends(~i) !?essential"'

As youam suggested on IRC, I also added the filter !?essential since we won’t touch essential packages when cleaning up the list of installed packages anyway.

Hope this helps further.

Useful but Unknown Unix Tools: htop //at 02:20 //by abe

You probably know about “top”, in Debian and Ubuntu part of the procps package.

Ever wanted to see CPU and memory usage as bars and not numbers?

Ever wanted to kill a process from inside top by just selecting its row instead of having to type its pid?

Ever tried to press a cursor key inside top? It makes a noise and says “Unknown command - try ‘h’ for help”. Short said: top is not that interactive.

Ever wanted top to be more colorful?

Well, there is a solution to all these issues. It’s called htop (Debian package) and is some kind of colorful, ncurses based cross-over between top and a Midnight Commander for processes:

More screenshots of htop at screenshot.debian.net and at the project’s site at SourceForge.

It can do many things, top can’t do:

• Different color themes including fore- and background (includes also a monochrome theme for people considering ANSI colors being eye cancer :-)
• Highlighting the current user’s processes.
• Scrolling up, down and sidewards.
• Interactively selecting processes with cursor keys as well as with the mouse.
• Kill or renice the selected process without having to enter its pid
• Show the CPU usage for each core and a CPU usage summary at the same time.
• Show CPU, memory and swap usage as textual bars, either encoded in colors or characters.
• Easy interactive configuring while running.
• Automatically saves the current state (sort order, color scheme, etc.)
• Easy access to the most often used functions through F-keys like in Midnight Commander.

One more cool thing about htop: It’s also available on the Nokia N900 as “app”.

The only thing I found so far that top can do, but htop can’t, is top’s batch mode (e.g. with “top -b -n 1”) where it runs non-interactively and its output can even be piped to other processes. The Xymon/Hobbit monitoring system uses that as input for some machine statistics.

ratpoison and focus follows mouse //at 00:22 //by abe

I use ratpoison as window manager on my ASUS EeePC netbook “nemo” for more than two years now. But although I’m very happy with ratpoison in the EeePC, there are two feature wishes which have been refused by upstream: One is more flexibel window name matching for the unmanage command. The other one is “focus follows mouse” between ratpoison frames.

Well, I always guessed that it was possible, but it took until now to find outhow to implement “focus follows mouse” for ratpoison.

There’s an ancient but still useful tool called Not a Window Manager (nawm) which is a small awk-like interpreter offering mostly window handling functions.

The following .nawmrc implements “focus follows mouse” in nawm:

window newwin;  # stores window to raise
window lastwin; # stores previous window to prevent race conditions
leave {
    lastwin = currentwindow;
}
enter {
    newwin = pointerwindow();
    if (name(newwin) != "" && newwin != lastwin) {
        raise newwin;
        sync;
    }
}

The leave hook is necessary to prevent flapping between two windows if switched between them via ratpoison’s commands.

I also had to add the following hook to my .ratpoisonrc to work around some cases where ratpoison’s own window switching didn’t work anymore. Only happened with more than one frame — with one frame banishing the mouse cursor was annoying, so I filtered that case:

addhook switchwin exec if [ `ratpoison -c fdump|fgrep -o frame|wc -l` -gt 1 ]; then ratpoison -c banish; fi

Unfortunately nawm has been removed from Debian Sid about a year ago due to being buggy and orphaned. There was not upstream development for seven years or so either.

So for the moment you can get nawm either from Debian Lenny or from snapshot.debian.org.

But I had to fix a segfault in nawm when calling name() on a window without name to be able to use it at all, so you will probably have to rebuild it anyway with the following patch:

diff -u nawm-0.0.20030130/builtins.c nawm-0.0.20030130-patched/builtins.c
--- nawm-0.0.20030130/builtins.c        2010-10-25 06:00:02.000000000 +0200
+++ nawm-0.0.20030130-patched/builtins.c        2010-10-25 04:15:25.000000000 +0200
@@ -546,8 +546,12 @@
     *name = gcstrdup("");
   else
     {
-      *name = gcstrdup((char *)nm);
-      XFree(nm);
+      if ((char *)nm) {
+        *name = gcstrdup((char *)nm);
+        XFree(nm);
+      } else {
+        *name = gcstrdup("");
+      }
     }
 }

And yes, I’m thinking about adopting and reintroducing the nawm package into Debian Sid.

But I’d prefer if anyone could give me a hint how to do this with more current and still maintained tools (or a patch against ratpoison :-). I looked into suckless-tools, but I haven’t found anything in there which provides hooks on X events. And the Perl module Tk seems to be able to set X event hooks, but only within the application being written itself.

Still happy with the ASUS EeePC 701 //at 16:02 //by abe

Recently Eric asked on the LUG Vorarlberg mailing list about netbook experience. I wrote a lengthy reply summarizing my experiences with the ASUS EeePC 701. And I thought this is something I probably should share with more people than only one LUG:

I ordered an ASUS EeePC 701 (4G) with US keyboard layout at digitec in Spring 2008, got it approximately one month later and posted a first resumé after one month in my blog.

I’m still very happy with the EeePC 701, despite two commonly mentioned drawbacks (the small screen resolution and the small SSD – which I both don’t see as real problems) and some other minor issues.

What matters

• Very robust and compact case. And thanks to a small fan being the only moving part inside, the EeePC 701 is also very robust against mobile use.
• Very pleasing always-in-my-daypack size (despite the 7" screen it’s the typical 9" netbook size) and easily held with one hand.
• Black. No glossy display. Neither clear varnish nor piano laquer. Short: No bath room tile. Textured surface, small scratches don’t stick out and don’t matter.
• Debian (previously Lenny, now Sid) runs fine on it, even the webcam works out-of-the-box.
• Despite all those neat features, it was fscking cheap at that time. And it was available without Windows.

Nice to have

• There’s power on the USB sockets even if the EeePC is turned off but the power supply is plugged in.
• The speakers are impressingly good and loud for their size. (But my demands with regards to audio are probably not too high, so audiophiles shouldn’t run to ebay because of this. ;-)
• It has three external USB sockets.

What doesn’t matter

• The small 7" 800×480 screen: I like small fonts and do most things inside a terminal anyway. And even with 800×480, those terminals are still much bigger than 80×25 characters. Only some applications and webpages have no heart for small screens.
• The small disk size: Quite a lot of programs fit on 4 GB of disk space. Additionally I use tmpfs a lot. And music and video files are either on a external 500 GB Western Digital 2.5" “My Passport” disk (which I need quite seldomly) or much more come via sshfs and IPv6 from my home server anyway. :-)
• The small keyboard: I just don’t have any problems with the size or layout (right shift right of the cursor up key, etc.) of the keyboard. Well, maybe except that any standard sized keyboard feels extremely large after having used the EeePC exclusively for some weeks. ;-)
• The to 630 MHz underclocked 900 MHz Intel Celeron: It’s enough for most of the things I do with the EeePC. Also the original 512 MB RAM are somehow ok, but for using tmpfs, but no swap space at all, 1 GB or 2 GB are surely the better choice.
• A battery runtime of 2.5h to 3h is fine for me.

What’s not so nice

• The “n” key needs to be pressed slighty stronger than other keys, otherwise no “n” appears. So if one of my texts in average misses more “n” than other letters, I typed it on the EeePC. ;-)
• Home, End, Page-Up, and Page-Down need the Fn key. This means that these keys can only be used with two hands (or one very big hand and I have quite small hands). This is usually no problem and you get used to it. It’s just annoying if you hold the EeePC with one hand and try to type with the other.
• What looks like a single mouse button is a seesaw and therefore two mouse buttons below one button. This makes it quite hard to press both at the same time, e.g. for emulating a middle mouse button press. It usually works in about half of all cases I tried it. My solution was to bind some key combination to emulate a middle mouse button in my window manager, ratpoison:

  bind y ratclick 2

  And that mouse button bar already fell off two times.
• The battery reports only in 10% steps, and reporting in percentage instead of mAh is an ACPI standard violation because reporting in percentage is only allowed for non-rechargable batteries. It also doesn’t report any charging and discharging rates. But in the meanwhile nearly all battery meter can cope with these hardware bugs. This was quite a problem in the early days.
• Now, after approximately 1.5 years, the battery slowly fritzes: When charging there are often only seconds between 10% and 40%. Rigorously using up all power of the battery helped a little bit. Looks like some kind of memory effect althought the battery is labeled Li-Ion and not Ni-MH and Li-Ion batteries are said to have no memory effect.
• The SD card reader only works fine if you once completed the setup of the original firmware or set the corresponding BIOS switch appropriately. No idea why.

Similar models

Technically, most of this also counts for the EeePC 900SD (not 901) which only differs in screen, resolution and disk size as well as CPU, but not on the the case. So same size, same robustness, same battery, same mainboard, bigger screen, resolution, disk and faster CPU. (The 901 has a different CPU, a different battery, and a different, glossy and partially chromed case.) See Wikipedia for the technical specifications of all EeePC models.

ASUS’ only big FAILure

Stopping to sell most EeePCs with Linux and cowardly teaming up with Microsoft after having shown big courage to come out with a Linux only netbook. Well, you probably already know, but it’s better without Windows…

So basically you no more get these really neat netbooks from ASUS anymore and you get nearly no netbooks with Linux from ASUS in the stores anymore. It’s a shame.

Would I buy it again?

Sure.

Well, maybe I would also buy the 900SD, 900AX (replacing the harddisk with an SSD) or 702 (8G) instead of the 701, but basically they’re very similar. See Wikipedia for the differences between these EeePC models. And of course I still prefer the versions without Windows.

But despite the low price, the EeePC 701 is surprisingly robust and still works as on the first day (ok, except battery, the mouse button bar and the “n” key ;-), so I recently bought a second power supply (only white ones were available *grrrr*) and ordered a bigger third party battery plus an adapter to load the battery directly from the (second) power supply without EeePC inbetween.

What desktop do I use on the EeePC?

None.

I use ratpoison as window manager, uxterm, urxvt, and yeahconsole as terminal emulators (running zsh with grml based .zshrc even as root’s login shell :-), wicd-curses as network manager and xmobar (previously dzen2) with i3status as text-only panel. Installed editors are GNU Emacs 23, GNU Zile and nvi. (No vim. :-)

And of course a netbook wouldn’t be a netbook if it wouldn’t have a lot of network applications installed. For me the most important ones are: ssh, scp, autossh, sshfs, miredo, conkeror, git, hg, and rsync.

/var/cache/apt/ on tmpfs //at 16:44 //by abe

My ASUS EeePC 701 (4G) “nemo” running Debian Sid has a 4 GB SSD as main disk, which is on the one hand quite full (mostly with software I use, but also local working copies of software I work on) and on the other hand an SSD, so I always try to reduce the amount of write to disk without losing convenience. Similar issues have systems which run off a CF or SD card or maybe even an USB stick.

Since I ordered a 2 GB RAM bar together with the EeePC, I not bound to the 512 MB which it had originally. But on the other hand I seldom needed more than 1 GB of RAM. Usually I needed between 400 MB and 1 GB of RAM. So it’s quite obvious to use tmpfs on as many places as possible.

Making /tmp, /var/run and /var/lock tmpfs were the most obvious directories to mount as tmpfs. Especially /var/run on tmpfs brought up a few bugs a while ago (mostly init.d scripts relying on /var/run/$PACKAGENAME/’s existence), but it’s no hassles to use nowadays. Even in Debian Stable such bugs got fixed.

Next target to explore for was /var/cache. According to the FHS, /var/cache is intended for cached data from applications. […] The application must be able to regenerate or restore the data. So it should be safe to put anything under /var/cache on tmpfs.

One directory in there which gets written quite often and with a lot of data on Debian Unstable is /var/cache/apt and its subdirectories, especially /var/cache/apt/archives. If you update your Sid installation daily, all new or updated .debs will be downloaded to /var/cache/apt first.

So I put /var/cache/apt on tmpfs by putting the following line into /etc/fstab:

tmpfs /var/cache/apt tmpfs defaults,noexec,nosuid,nodev,mode=0755 0 0

But despite FHS stating that anything under /var/cache must be reproducible by the application, apt is puking and refusing to work:

!447 Z31 ?0 L1 root@nemo:pts/0 (-zsh) 16:13:10 [~] # apt-get update
E: Archive directory /var/cache/apt/archives/partial is missing.
!448 Z32 ?100 L1 root@nemo:pts/0 (-zsh) 16:13:17 [~] # 

If you create /var/cache/apt/archives/partial, it will also argue about /var/cache/apt/partial.

Of course the workaround is simple: Just put mkdir -p /var/cache/apt/partial /var/cache/apt/archives/partial in /etc/rc.local.

But nevertheless, this is a bug in apt – which already has been reported by madduck earlier this year (#523920). Unfortunately the APT maintainers have not yet even commented on this FHS violation and therefore also a Debian Policy (Section 9.1.1) violation.

One more thought about /var/cache/apt vs only /var/cache/apt/archives: apt-file also caches its data under /var/cache/apt. So if you want to use apt-file after a reboot and have /var/cache/apt mounted as tmpfs, you have to run apt-file update first and it will download all Contents files (can be dozens of megabytes) and not only the differences to previously downloaded Contents files.

So if you use apt-file a lot, you probably go better with making only /var/cache/apt/archives tmpfs and not whole /var/cache/apt.