German-written Debian Package Management Book //at 01:28 //by abe

Thursday was our big day: After more than 2.5 years of working in the hidden, ups and downs, Frank Hofmann and myself were able to announce the availability of our book project Debian Package Management under a free license (Creative Commons Attribution ShareAlike 4.0 International License, short “CC BY-SA 4.0”) during a Lightning Talk at DebConf15 in Heidelberg.

This became possible because we found Onyx Neon, a publishing company which is specialised on books with contents under free licenses. Its founder does not only have a faible for Perl but also for Debian. (Since the question already came up: We also thought about self-publishing, e.g. via Lulu or Epubli — and it would have been our fallback solution —, but we prefer the professionalism and services of a real publisher. I’m though happy to share what I found out about self-publishing in the past few months.)

The source code of the book is written in the AsciiDoc format and available on GitHub.

The book is still work in progress. But if you want, you can already build an e-book out of the publically available source code:

sudo apt-get install asciidoc dblatex git
git clone git://github.com/dpmb/dpmb.git
cd dpmb
make

(Works fine on Debian 7 Wheezy, Debian 8 Jessie and Ubuntu 14.04 LTS Trusty. Does not work on Ubuntu 12.04 LTS Precise.)

If you find an error in the book, please file an issue on GitHub. If you also know how to fix the error, please for the Git repository on GitHub, fix the error in your Git repository and file a pull request. (The first pull request already happenend and has been applied.)

Initially there will be only a German written issue as e-book (at least in HTML, PDF and EPUB formats, maybe also KF8/MOBI and EPUB3) and at some point in the future also as printed book at Onyx Neon. But we’re also planning a translation to English as well as a Debian package.

If your want to get informed when we publish a printed book, a translation or an official e-book release, please subscribe to one of our mailing lists: There’s one in German and one in English.

Debian-Paketmanagement-Buch //at 00:47 //by abe

Donnerstag war der große Tag: Nach über zweieinhalb Jahren Arbeit im Verborgenen, vielen Höhen und einigen Tiefen, konnten Frank Hofmann und ich heute im Rahmen eines Lightning Talks auf der DebConf15 in Heidelberg die Verfügbarkeit unseres Buch-Projektes Debian-Paketmanagement unter einer freien Lizenz (Creative Commons Namensnennung - Weitergabe unter gleichen Bedingungen 4.0 International Lizenz, kurz “CC BY-SA 4.0”) bekanntgeben.

Möglich wurde dies dadurch, daß wir mit Onyx Neon einen Verlag gefunden haben, der sich auf Bücher unter freien Lizenzen spezialisiert hat und deren Besitzer nicht nur ein Faible für Perl sondern auch für Debian haben. (Da die Frage bereits aufkam: Wir hatten auch an Selbst-Publikation via z.B. Lulu oder Epubli gedacht — und es wäre auch unser Fallback gewesen —, ziehen aber die Professionalität und Dienstleistungen eines richtigen Verlages vor.)

Der Quellcode des Buches ist im AsciiDoc-Format geschrieben und auf GitHub verfügbar.

Das Buch ist noch in der Mache, sozusagen Work In Progress. Aber wer möchte, kann sich trotzdem bereits jetzt aus dem online verfügbaren Quellcode ein E-Book (elektronisches Buch) generieren:

sudo apt-get install asciidoc dblatex git
git clone git://github.com/dpmb/dpmb.git
cd dpmb
make

(Funktioniert auf Debian 7 Wheezy, Debian 8 Jessie und Ubuntu 14.04 LTS Trusty. Funktioniert nicht auf Ubuntu 12.04 LTS Precise.)

Wer Fehler findet, darf gerne auf GitHub einen Fehlerreport öffnen und wer auch noch weiß, wie man ihn flicken kann, der kann das Git-Repository forken, den Fehler darin flicken und dann einen sog. Pull-Request stellen. (Den ersten Pull-Request haben wir bereits bekommen und gemerget.)

Initial wird es eine deutschsprachige Ausgabe als E-Book (mindestens HTML, PDF und EPUB, vielleicht auch KF8/MOBI und EPUB3) und ab einem bestimmten Zeitpunkt auch als gedrucktes Buch im Verlag Onyx Neon geben. Es ist aber auch eine Übersetzung ins Englische geplant. Ebenso ist eine Paketierung für Debian in Form eines Debian-Paketes angedacht.

Wer informiert werden will, sobald ein gedrucktes Buch, eine Übersetzung oder ein offizielles E-Book verfügbar ist, der kann unsere Mailinglisten abonnieren: Deutsch oder Englisch

PDiffs are still useful //at 03:26 //by abe

… probably just not as default.

I do agree with Richi and with Michael that disabling PDiffs by default gives the big majority of Debian Testing or Unstable users a speedier package list update.

I’m though not sure, if disabling PDiffs by default would

• also have an performance impact on our mirrors — it surely would have a traffic impact on the mirrors;
• really bring a benefit for Debian Stable users as Debian Stable changes seldomly and hence there are not that many PDiffs to download and apply — at least I can’t remember being annoyed by PDiffs anywhere else than on Debian Testing and Debian Unstable. Even the repositories with security updates don’t change that often.

Additionally I want to remind you that PDiffs per se are nothing bad and should be continued to be supported:

• Because there are still areas, even in “civilized” countries, where only small bandwidth is available and where using PDiffs reduces the download time a lot. Yes, also in Germany. BTDT. Only until recently there was no Fibre, no DSL, very bad UTMS reception and otherwise just EDGE at my parents’ home. (LTE was available far too expensive until recently.) And I was very happy about not having to download 30 MB or such just for seeing if there are updates at all, because 25 kB/s was the fastest download rate I could get (peaks, not average).
• Because it seems to be in fashion with big ISP near-to-monoplists, especially in Germany, to cut off your nice bandwidth if you transfer too many Megabytes. Keyword “Drosselkom”. If you happen to be a customer of such a shitty ISP, you may be happy to reduce your traffic amount by using PDiffs instead of downloading the full package list every time.

So yes, disabling PDiffs by default is probably ok, but the feature must be kept available for those who haven’t 100 MBit/s fibre connection into their homes or are sitting just one hop away from the next Debian mirror (like me at work :-).

Oh, and btw., for the very same reasons I’m also a big fan of debdelta which is approximately the same as PDiffs, just not for package lists but for binary packages. Using debdelta I was able to speed up my download rates over EDGE to up to virtual 100 kB/s, i.e. by factor four (depending on the packages). Just imagine a LibreOffice minor update at 15 kB/s average download rate. ;-)

And all these experiences were not made with a high-performance CPU but with the approximately 5 year old Intel Atom processor of my ASUS EeePC 900A. So I used PDiffs and debdelta even despite having a slight performance penalty on applying the diffs and deltas.

Automatically hardlinking duplicate files under /usr/share/doc with APT //at 20:43 //by abe

On my everyday netbook (a very reliable first generation ASUS EeePC 701 4G) the disk (4 GB as the product name suggests :-) is nearly always close to full.

TL;DWTR? Jump directly to the HowTo. :-)

So I came up with a few techniques to save some more disk space. Installing localepurge was one of the earliest. Another one was to implement aptitude filters to do interactively what deborphan does non-interactively. Yet another one is to use du and friends a lot – ncdu is definitely my favourite du-like tool in the meantime.

Using du and friends I often noticed how much disk space /usr/share/doc takes up. But since I value the contents of /usr/share/doc a lot, I condemn how Nokia solved that on the N900: They let APT delete all files and directories under /usr/share/doc (including the copyright files!) via some package named docpurge. I also dislike Ubuntu’s “solution” to truncate the shipped changelog files (you can still get the remainder of the files on the web somewhere) as they’re an important source of information for me.

So when aptitude showed me that some package suddenly wanted to use up quite some more disk space, I noticed that the new package version included the upstream changelog twice. So I started searching for duplicate files under /usr/share/doc.

There are quite some tools to find duplicate files in Debian. hardlink seemed most appropriate for this case.

First I just looked for duplicate files per package, which even on that less than four gigabytes installation on my EeePC found nine packages which shipped at least one file twice.

As recommended I rather opted for an according Lintian check (see bugs. Niels Thykier kindly implemented such a check in Lintian and its findings are as reported as tags “duplicate-changelog-files” (Severity: normal, from Lintian 2.5.2 on) and “duplicate-files” (Severity: minor, experimental, from Lintian 2.5.0 on).

Nevertheless, some source packages generate several binary packages and all of them (of course) ship the same, in some cases quite large (Debian) changelog file. So I found myself running hardlink /usr/share/doc now and then to gain some more free disk space. But as I run Sid and package upgrades happen more than daily, I came to the conclusion that I should run this command more or less after each aptitude run, i.e. automatically.

Having taken localepurge’s APT hook as example, I added the following content as /etc/apt/apt.conf.d/98-hardlink-doc to my system:

// Hardlink identical docs, changelogs, copyrights, examples, etc

DPkg
{
Post-Invoke {"if [ -x /usr/bin/hardlink ]; then /usr/bin/hardlink -t /usr/share/doc; else exit 0; fi";};
};

So now installing a package which contains duplicate files looks like this:

~ # aptitude install perl-tk
The following NEW packages will be installed:
  perl-tk 
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,522 kB of archives. After unpacking 6,783 kB will be used.
Get: 1 http://ftp.ch.debian.org/debian/ sid/main perl-tk i386 1:804.029-1.2 [2,522 kB]
Fetched 2,522 kB in 1s (1,287 kB/s)  
Selecting previously unselected package perl-tk.
(Reading database ... 121849 files and directories currently installed.)
Unpacking perl-tk (from .../perl-tk_1%3a804.029-1.2_i386.deb) ...
Processing triggers for man-db ...
Setting up perl-tk (1:804.029-1.2) ...
Mode:     real
Files:    15423
Linked:   3 files
Compared: 14724 files
Saved:    7.29 KiB
Duration: 4.03 seconds
localepurge: Disk space freed in /usr/share/locale: 0 KiB
localepurge: Disk space freed in /usr/share/man: 0 KiB
localepurge: Disk space freed in /usr/share/gnome/help: 0 KiB
localepurge: Disk space freed in /usr/share/omf: 0 KiB

Total disk space freed by localepurge: 0 KiB

Sure, that wasn’t the most space saving example, but on some installations I saved around 100 MB of disk space that way – and I still haven’t found a case where this caused unwanted damage. (Use of this advice on your own risk, though. Pointers to potential problems welcome. :-)

Daily Snapshot .debs of Conkeror //at 22:57 //by abe

Keeping track with packaging software which is under heavy development can be time-consuming. I noticed this while packaging Conkeror, because there was quite a demand for up-to-date packages, especially from upstream themself.

So recently on the IRC channel #conkeror the idea of automatically built Debian packages came up. After a few hours of experimenting and a few days of steadily optimizing, I can proudly present daily built snapshot packages of Conkeror for currently Lenny and Sid, ready to be included in your sources.list:

deb     http://noone.org/conkeror-nightly-debs lenny main
deb-src http://noone.org/conkeror-nightly-debs lenny main

deb     http://noone.org/conkeror-nightly-debs sid main
deb-src http://noone.org/conkeror-nightly-debs sid main

The binary package conkeror-spawn-process-helper is currently only built for the i386 architecture, but other architectures may follow.

The packages probably work also on any other Debian based distribution (e.g. Ubuntu) which includes XULRunner version 1.9.

Surely they are not of the usual Debian quality, but they should do it for staying up-to-date with the Conkeror development just by using your favourite APT frontend.

The script which generates those packages is also available in the Conkeror git repository at repo.or.cz.

The APTable archive is generated with reprepro. Packages and the repository are signed with the passphrase-less GnuPG key 373B76B4 which is used only for the Conkeror nightly builds. (If anyone knows a better solution for automatic builds than a passphrase-less key, please tell me. :-)

P.S.: I really like the new keybindings “<<”, “>>” and “G”. :-)

Useful but Unknown Unix Tools: netselect //at 00:05 //by abe

Ever wondered which mirror of your favourite Linux distribution is the fastest at your location?

Check it with netselect (code at GitHub). It checks for the number of hops and ping times to given hosts and tells you which one is the fastest of them:

# netselect -vv ftp.de.debian.org ftp2.de.debian.org \
                ftp.ch.debian.org ftp.nl.debian.org ftp.debian.org
Running netselect to choose 1 out of 5 addresses.
.......................................................
ftp.de.debian.org                       25 ms  16 hops   90% ok ( 9/10) [   72]
ftp2.de.debian.org                      17 ms  17 hops   90% ok ( 9/10) [   51]
ftp.ch.debian.org                        0 ms   3 hops   90% ok ( 9/10) [    0]
ftp.nl.debian.org                       22 ms  15 hops   90% ok ( 9/10) [   62]
ftp.debian.org                          22 ms  15 hops   90% ok ( 9/10) [   60]
    0 ftp.ch.debian.org

And if you’re too lazy to optimize your sources.list with netselect manually, just use the netselect-apt package. It will do it for you.