New web browsers in Wheezy //at 16:14 //by abe

Since there is so much nice new stuff in Debian Wheezy, I have to split up my contributions to Mika’s #newinwheezy game on Planet Debian.

Here’s the next bunch, this time web browsers:

dillo

The FLTK-based lightweight GUI web browser Dillo comes with its own rendering engine (no JavaScript, incomplete CSS support) was already in Debian before, but was removed before the release of Debian Squeeze, because Dillo 2 relied on FLTK 2.x which had an unclear license situation back then and never made it into Debian. In the meanwhile Dillo 3 relies on FLTK 1.3 as FLTK upstream abandoned the 2.0 branch and continued development on the 1.3 branch. So I brought Dillo back into Debian with its 3.0.x release.

netsurf

The RiscOS-originating lightweight GUI web browser Netsurf was already in Debian, too, but didn’t make it into Debian Squeeze as it needed the Lemon parser generator (part of the SQLite source) to build back then and a change in Lemon caused Netsurf to no more build properly in the wrong moment. Netsurf supports CSS 2.1, but has no JavaScript support either. I’d consider its rendering engine more complete than Dillo’s.

surf and xxxterm

Surf and XXXTerm are both simple and minimalistic webkit-based browsers. Surf is easy to embed in other applications and XXXTerm features vi-like keybindings for heavy keyboard users.

To be continued… ;-)

New SSH-related stuff in Wheezy //at 15:28 //by abe

Mika had the nice idea of doing a #newinwheezy game on Planet Debian, so let’s join:

There are (at least) two new SSH related tools new in Debian Wheezy:

mosh

is the “mobile shell”, an UDP based remote shell terminal which works better than SSH in case of lag, packet loss or other forms of bad connection. I wrote about mosh in more detail about a year ago. mosh is also available for Debian Squeeze via squeeze-backports.

sshuttle

is somewhere between port-forwarding and VPN. It allows forward arbitrary TCP connections over an SSH connection without the need to configure individual port forwardings. It does not need root access on the server-side either. I wrote about sshuttle in more detail about a year ago.

To be continued… ;-)

Up to date Aptitude Documentation Online //at 12:51 //by abe

Aptitude ships documentation in 7 languages as HTML files. However the latest version available online was 0.4.11.2 from 2008 and hosted on the server by the previous, now unfortunately inactive Aptitude maintainer, and only covered 5 languages.

This lack of up to date online documentation even caused others to put more up to date versions online. Nevertheless they age, too, and the one I’m aware is not up to date for Wheezy.

So the idea was born to keep an up to date version online on Aptitude’s Alioth webspace (which currently redirects to a subdirectory of the previous maintainer’s personal website). But unfortunately we, the current Aptitude Team, are still lacking administrative rights on Aptitude’s Alioth project, which would be necessary to assign new team members who could work on that.

As an intermediate step, there’s now a (currently ;-) up to date Aptitude User’s Manual online in all 7 languages at

http://people.debian.org/~abe/aptitude/

and English at

http://people.debian.org/~abe/aptitude/en/

As this location could also suffer from the same MIA issues as any other “personal” copy, the plan is to move this to somewhere under http://aptitude.alioth.debian.org/ as soon as we have full access to Aptitude’s Alioth project.

Our plans for then are:

• Redirects from http://people.debian.org/~abe/aptitude/ to e.g. http://aptitude.alioth.debian.org/doc/, so that all your links to or bookmarks of http://people.debian.org/~abe/aptitude/ are still valid. (This unfortunately won’t work for jump marks to specific sections, just per file, i.e. chapter.)
• Set up a cron-job, which keeps the documentation in sync with the version of Aptitude in Unstable (and maybe also with Aptitude in Stable).

P.S.: Anyone interested in doing a German translation of the Aptitude User’s Manual? Sources are in DocBook, i.e. XML, and available via Git.

Suggestions for the GNOME Team //at 23:01 //by abe

Thanks to Erich Schubert’s blog posting on Planet Debian I became aware of the 2012 GNOME User Survey at Phoronix.

Like back in 2006 I still use some GNOME applications, so I do consider myself as “GNOME user” in the widest sense and hence I filled out that survey. Additionally I have to live with GNOME 3 as a system administrator of workstations, and that’s some kind of usage, too. ;-)

The last question in the survey was Do you have any comments or suggestions for the GNOME team? — Sure I have. And since I tried to give constructive feedback instead of only ranting, here’s my answer to that question as I submitted it in the survey, too, just spiced up with some hyperlinks and highlighting:

Don’t try to change the users. Give the users more possibilities to change GNOME if they don’t agree with your own preferences and decisions. (The trend to castrate the user was already starting with GNOME 2 and GNOME 3 made that worse IMHO.)

If you really think that you need less configurability because some non-power-users are confused or challenged by too many choices, then please give the other users at least the chance to enable more configuration options. A very good example in that hindsight was Kazehakase (RIP) who offered several user interfaces (novice, intermediate and power user or such). The popular text-mode web browser Lynx does the same, too, btw.

GNOME lost me mostly with the change to GNOME 2. The switch from Galeon 1.2 to 1.3/2.0 was horrible and the later switch to Epiphany made things even worse on the browser side. My short trip to GNOME as desktop environment ended with moving back to FVWM (configurable without tons of clicking, especially after moving to some other computer) and for the browser I moved on to Kazehakase back then. Nowadays I’m living very well with Awesome and Ratpoison as window managers, Conkeror as web browser (which are all very configurable) and a few selected GNOME applications like Liferea (luckily still quite configurable despite I miss Gecko’s about:config since the switch to WebKit), GUCharmap and Gnumeric.

For people switching from Windows I nowadays recommend XFCE or maybe LXDE on low-end computers. I likely would recommend GNOME 2, too, if it still would exist. With regards to MATE I’m skeptical about its persistance and future, but I’m glad it exists as it solves a lot of problems and brings in just a few new ones. Cinnamon as well as SolusOS are based on the current GNOME libraries and are very likely the more persistent projects, but also very likely have the very same multi-head issues we’re all barfing about at work with Ubuntu Precise. (Heck, am I glad that I use Awesome at work, too, and all four screens work perfectly as they did with FVWM before.)

Thanks to Dirk Deimeke for his (German written) pointer to Marcus Moeller’s interview with Ikey Doherty (in German, too) about his Debian-/GNOME-based distribution SolusOS.

zutils: zcat and friends on Steroids //at 01:18 //by abe

I recently wrote about tools to handle archives conveniently. If you just have to handle compressed text files, there are some widely known shortcut commands to mimic common commands on files compressed with a specific compression format.

In Debian and derivatives, those tools are part of the according package for that compression utility, i.e. the zcat command is part of the gzip package and the xzfgrep command is part of the xz-utils package.

But despite this matrix is quite easy to remember, the situation has a few drawbacks:

• Those tools can only handle the format they’re written for (which btw. means that all xz-tools can also handle lzma-compressed files as lzma is xz’s predecessor)
• zcat and the other cat variants can’t even recognize non-compressed files and throw an error instead of just showing their contents.
• I always tend to think that lzcat and friends are for lzip-based compression as xzcat can handle lzma-compressed files anyway.

This is where the zutils project comes in: zutils provides the functionality of most of these utilities, too, but with one big difference: You don’t have to remember, think about or type which compression method has been used for your data, just use zcat, zcmp, zdiff, zgrep, zegrep, or zfgrep and it works — independently of what compression method has been used — if any — or if there are different compression types mixed in the parameters to the same command:

$ zfgrep foobar bla.txt fnord.gz hurz.xz quux.lz bar.lzma

Especially if you use logrotate and let logrotate compress old logs, it’s very comfortable that one command suffices to concatenate all the available logfiles, including the current uncompressed one:

$ zcat /var/log/syslog* | …

Additionally, zutils’ versions of these tools also support lzip-compressed files.

The zutils package is available in Debian starting with Wheezy and in Ubuntu since Oneiric. When being installed, it replaces the original z* utilities from the gzip package by diverting them away.

The only drawback so far is that there neither a zless nor a zmore utility from the zutils project, so zless bla.txt fnord.gz hurz.xz quux.lz bar.lzma will not work as expected even after installing zutils as it is still the one from the gzip package and hence it will show you just the first two files in plain text, but not the remaining ones.

Finding similar but not identical files //at 17:10 //by abe

There are quite some tools to find duplicate files in Debian (Ua is not even packaged for Debian!!!1!eleven! SCNR — via Chrütertee) and depending on the task I use either hardlink (see this blog posting), fdupes (if I need output with all identical files on one line; see example below), or duff (if it has to be performant).

But for code deduplication in historically grown code you sometimes need a tool which does not only find identical files, but also those which just differ in a few blanks or blank lines.

I found two tools in Debian which can give you some kind of percentage of similarity: simhash (which is btw. orphaned; upstream homepage) and similarity-tester (upstream homepage).

simhash has the shorter name and hecne sounds more usable on the command-line. But it seems only be able to compare two files at once and also only after first computing and writing down its similarity hash to a file. Not really usable for those one-liner cases on the command-line.

similarity-tester has the longer name (and one which made me suspect that it may be a GUI tool), but provides what I was looking for:

$ find . -type f | sim_text -ipTt 75

This lists all files in the current directory which have at 75% (“-t 75”) in common with another file in the list of files. The option “-i” causes sim_text to read the files to compare from standard input; “-p” causes sim_text to just output the similarity percentage; and “-T” suppresses the per-file list of found tokens.

I used similarity-tester’s “sim_text” tool to compare natural langauge as most of the files, I had to test, are shell scripts. But similarity-tester also provides tools to test the similarity of code in specific programming languages, namely C, Java, Pascal, Modula-2, Lisp and Miranda.

Example output from the xen-tools project (after I already did a lot of code deduplication):

./intrepid/30-disable-gettys consists for 100 % of ./edgy/30-disable-gettys material
./edgy/30-disable-gettys consists for 100 % of ./intrepid/30-disable-gettys material
./common/90-make-fstab-rpm consists for 98 % of ./centos-5/90-make-fstab material
./centos-5/90-make-fstab consists for 98 % of ./common/90-make-fstab-rpm material
./gentoo/55-create-dev consists for 91 % of ./dapper/55-create-dev material
./dapper/55-create-dev consists for 90 % of ./gentoo/55-create-dev material
./gentoo/55-create-dev consists for 88 % of ./common/55-create-dev material
./common/90-make-fstab-deb consists for 87 % of ./common/90-make-fstab-rpm material
./common/90-make-fstab-rpm consists for 85 % of ./common/90-make-fstab-deb material
./common/30-disable-gettys consists for 81 % of ./karmic/30-disable-gettys material
./intrepid/80-install-kernel consists for 78 % of ./edgy/80-install-kernel material
./edgy/30-disable-gettys consists for 76 % of ./karmic/30-disable-gettys material
./karmic/30-disable-gettys consists for 76 % of ./edgy/30-disable-gettys material
./common/50-setup-hostname-rpm consists for 76 % of ./gentoo/50-setup-hostname material

Depending on the length of possible filenames and amount of files this can be made more readable using the column utility from the bsdmainutils package and reversed by using tac from the coreutils package:

$ find . -type f | sim_text -ipTt 75 | tac | column -t
./common/50-setup-hostname-rpm  consists  for  76   %  of  ./gentoo/50-setup-hostname    material
./karmic/30-disable-gettys      consists  for  76   %  of  ./edgy/30-disable-gettys      material
./edgy/30-disable-gettys        consists  for  76   %  of  ./karmic/30-disable-gettys    material
./intrepid/80-install-kernel    consists  for  78   %  of  ./edgy/80-install-kernel      material
./common/30-disable-gettys      consists  for  81   %  of  ./karmic/30-disable-gettys    material
./common/90-make-fstab-rpm      consists  for  85   %  of  ./common/90-make-fstab-deb    material
./common/90-make-fstab-deb      consists  for  87   %  of  ./common/90-make-fstab-rpm    material
./gentoo/55-create-dev          consists  for  88   %  of  ./common/55-create-dev        material
./dapper/55-create-dev          consists  for  90   %  of  ./gentoo/55-create-dev        material
./gentoo/55-create-dev          consists  for  91   %  of  ./dapper/55-create-dev        material
./centos-5/90-make-fstab        consists  for  98   %  of  ./common/90-make-fstab-rpm    material
./common/90-make-fstab-rpm      consists  for  98   %  of  ./centos-5/90-make-fstab      material
./edgy/30-disable-gettys        consists  for  100  %  of  ./intrepid/30-disable-gettys  material
./intrepid/30-disable-gettys    consists  for  100  %  of  ./edgy/30-disable-gettys      material

Compared to that, fdupes only finds the two 100% identical files:

$ fdupes -r1 . 
./intrepid/30-disable-gettys ./edgy/30-disable-gettys 

But fdupes helped me already a lot to find the first bunch of identical files in xen-tools. :-)

unburden-home-dir uploaded to Sid //at 02:54 //by abe

Most popular web browsers cause quite a lot of I/O on a user’s home directory and their cache’s also take up quite some disk space – with Google’s Chrome/Chromium you can’t even configure how much disk space should be used for the cache.

This causes unnecessary network traffic and no more makes sense if the home directory itself comes over the network, e.g. via NFS or Samba. And on laptops it spins up the disks and unnecessarily costs battery power and therefore lowers the potential battery life.

Such caches also costs scarce disk space on SSDs or flash cards as common in laptops, netbooks and other mobile devices, and often get backed up without any real use.

To take some of this burden off our NFS servers at work I started to develop an Xsession.d hook which moves off such caches to the local disk and puts in symbolic links instead into the user’s home directory when the user locally logs in.

This hook quickly became a standalone Perl script named unburden-home-dir and the Xsession.d hook just a wrapper around it. Due to some unsolved issues I didn’t feel it’s good enough for Debian Unstable, so I uploaded it just to Debian Experimental back then.

Pietro Abate’s recent blog posting about unburden-home-dir on Planet Debian gave me the right kick to make another try to solve the remaining issues.

And the mental distance gained over the time indeed helped and I could fix the remaining issues. So I added some polish to the package and uploaded it to Debian Unstable.

If you used the previous version from experimental, you have to take care of a few things:

• Previously some configuration files sported unburden_home_dir as base name while others used unburden-home-dir as base name as that’s also the package name. Now all configuration files use the package name, i.e. unburden-home-dir as base name.
• “Conffiles” under /etc/ should be renamed by dpkg automatically, but per-user configuration files ($HOME/.unburden_home_dir and $HOME/.unburden_home_dir_list) must be manually renamed to $HOME/.unburden-home-dir and $HOME/.unburden-home-dir.list.
• By adding UNBURDEN_HOME=yes to $HOME/.unburden-home-dir every user can decide himself if he wants the Xsession.d hook to be used when he logs in under X. On managed workstations with many users this eases testing of unburden-home-dir with just a few users a lot.

You can follow the development of unburden-home-dir also on GitHub and on Gitorious as well as on Ohloh.

Enjoy!