PDiffs are still useful //at 03:26 //by abe

… probably just not as default.

I do agree with Richi and with Michael that disabling PDiffs by default gives the big majority of Debian Testing or Unstable users a speedier package list update.

I’m though not sure, if disabling PDiffs by default would

• also have an performance impact on our mirrors — it surely would have a traffic impact on the mirrors;
• really bring a benefit for Debian Stable users as Debian Stable changes seldomly and hence there are not that many PDiffs to download and apply — at least I can’t remember being annoyed by PDiffs anywhere else than on Debian Testing and Debian Unstable. Even the repositories with security updates don’t change that often.

Additionally I want to remind you that PDiffs per se are nothing bad and should be continued to be supported:

• Because there are still areas, even in “civilized” countries, where only small bandwidth is available and where using PDiffs reduces the download time a lot. Yes, also in Germany. BTDT. Only until recently there was no Fibre, no DSL, very bad UTMS reception and otherwise just EDGE at my parents’ home. (LTE was available far too expensive until recently.) And I was very happy about not having to download 30 MB or such just for seeing if there are updates at all, because 25 kB/s was the fastest download rate I could get (peaks, not average).
• Because it seems to be in fashion with big ISP near-to-monoplists, especially in Germany, to cut off your nice bandwidth if you transfer too many Megabytes. Keyword “Drosselkom”. If you happen to be a customer of such a shitty ISP, you may be happy to reduce your traffic amount by using PDiffs instead of downloading the full package list every time.

So yes, disabling PDiffs by default is probably ok, but the feature must be kept available for those who haven’t 100 MBit/s fibre connection into their homes or are sitting just one hop away from the next Debian mirror (like me at work :-).

Oh, and btw., for the very same reasons I’m also a big fan of debdelta which is approximately the same as PDiffs, just not for package lists but for binary packages. Using debdelta I was able to speed up my download rates over EDGE to up to virtual 100 kB/s, i.e. by factor four (depending on the packages). Just imagine a LibreOffice minor update at 15 kB/s average download rate. ;-)

And all these experiences were not made with a high-performance CPU but with the approximately 5 year old Intel Atom processor of my ASUS EeePC 900A. So I used PDiffs and debdelta even despite having a slight performance penalty on applying the diffs and deltas.

Showing packages newer than in archive with aptitude //at 22:14 //by abe

I happens quite often that I install a manually built, newer version of some package on a machine. Occassionally I forget to remove it or to downgrade it to the version in the APT repo.

$ apt-show-versions | fgrep newer

easily finds those packages.

But usually when doing such a check, I want this list of packages in my aptitude TUI to have a look at the other versions of that package and to take actions. And I don’t want to manually search for each of the package manually.

This can be done with the following “one-liner”:

# aptitude -o "Aptitude::Pkg-Display-Limit=( `apt-show-versions | fgrep newer | awk -F '[ :]' '{printf "~n ^"$1"$ | "}' | sed -e 's/| *$//'` )"

It uses apt-show-version’s output, searches for the right packages, takes the first column and transforms it into an aptitude search pattern matching all packages whose name is exactly one of the listed packages.

But this solution is quite ugly and slow. So I wondered if this is also doable with pure aptitude search patterns which likely would also be faster.

And after some playing around I found the following working aptitude search term:

~i ?any-version(!~O.) !~U !~o

This matches all packages which which are installed and which have a version which has no origin, i.e. no associated APT repository. Since this also matches all hold packages as well as all packages not available in any archive, I use !~U !~o to exclude those packages from that list again.

Since nobody can remember that nor wants to type that everytime needed, I added the following alias to my setup:

alias aptitude-newer-than-in-archive='aptitude -o "Aptitude::Pkg-Display-Limit=~i ?any-version(!~O.) !~U !~o"'

Only caveat so far:

It seems to also match packages from APT repos which haven’t set an “Origin”. This should not happen with any Debian or Ubuntu APT repository, but seems to happen occasionally with privately run APT repositories.

And using ~A instead of ~O, i.e. ~i ?any-version(!~A.), does not work for this case either, despite it matches installed packages of which versions not in any available archive exist. But unfortunately aptitude seems to remember in some way if a package was in some archive in the past, so this only shows packages installed with dpkg -i, but not packages removed from e.g. unstable but with older versions still being available in stable.

Next Debian Meetup in Zurich //at 20:16 //by abe

The first Debian meetup in Zurich last month was quite a success and I look forward to further Debian meetups in Zurich — every first Tuesday of the month.

The next meetup will be

on Tuesday, 2013-Dec-03 starting at 18:30 CET
at St. Gallerhof, Konradstrasse 2, 8005 Zurich

Please note that this is a different location than last month.

Everybody who is interested in Debian is welcome to join us. Registering is not necessary.

There was also some interest in Debian meetups in other Swiss cities, namely Bern and somewhere around Lac Leman. In case you want a Debian meetup elsewhere in Switzerland, too, or if you’re interested in any Debian meetup in Switzerland, feel free to join the Swiss Debian Community Mailing List and help organising other Swiss Debian meetups.

Debian-Stammtisch in Zürich //at 12:49 //by abe

Wir (Michael Stapelberg und ich) wollen in Zürich einen regelmässigen Debian-Stammtisch, eine Debian-Lokalgruppe etablieren.

Das erste Treffen findet statt:

• am Dienstag, den 5. November 2013, ab 19 Uhr (CET)
• im Gloria, Josefstrasse 59, 8005 Zürich.

Jeder, der sich für Debian interessiert, ist eingeladen. Eine Anmeldung ist nicht notwendig.

Als regelmässigen Termin visieren wir den ersten Dienstag im Monat an.

Diesen Termin haben wir gewählt, weil er unseren Recherchen nach keine Terminkonflikte mit den Lokalgruppen der LUGS in Zürich und Winterthur, der FSFE in Zürich, dem CCC ZH, den Tuxeros oder dem Webtuesday gibt. In Kauf genommen haben wir Kollisionen mit wöchentlichen Treffs in der Hackerspaces Ruum42 in St. Gallen und Reaktor23 in Waldshut-Tiengen.

Es kam aber dennoch bereits die erste Meldung bzgl. potentieller Terminkonflikte rein. Wir diskutieren gerne noch über den Termin, sowohl am Stammtisch selbst, aber auch auf der Debian.ch Community-Mailingliste. Bevorzugte Sprache auf der Mailingliste ist Englisch, Deutsch ist aber auch in Ordnung.

Potentielle Terminänderungen werden primär auf der Debian.ch Community-Mailingliste und im Debian-Wiki auf der Lokalgruppen-Seite bekanntgegeben.

Die Termine wollen wir aber auch über die LUGS-Termine-Liste, die es auch als iCal oder E-Mail-Reminder gibt, publizieren. Danach sollten die Termine auch auf Freie Termine erscheinen.

Hui, ist das lange her, daß ich hier was auf Deutsch geschrieben habe. :-)

Searching in Screen’s copy mode //at 23:43 //by abe

I’m using GNU Screen daily for definitely more than a decade and I became maintainer of Debian’s screen package nearly exactly two years ago. Nevertheless it still happens occassionally that I discover features yet unknown to me. Recently I had one of these moments again:

I looked for a specific line in the long output of a command which has run inside a Screen session. For that I entered Screen’s copy mode with Ctrl-A [ and scrolled around with arrow keys and page-up and -down keys.

But didn’t find it. I thought, it would be cool if I can search for the string I’m looking for. Intuïtively I typed /, the search string and pressed enter. And it worked! It jumped to the next occurrence of that string.

Of course I immediately had to check if tmux has such a feature, too. And it indeed has, but it seems to be a less sophisticated implementation:

Being able to do incremental search like with GNU Emacs gave me yet another reason for continuing to use Screen and not to switch Tmux. ;-)

How to make wget honour Content-Disposition headers //at 16:12 //by abe

Download links often point to CGI scripts which actually generate (or just fetch, i.e. proxy) the actual file to be downloaded, e.g. URLs like http://www.example.com/download.cgi?file=foobar.txt.

Most of such CGI scripts send the real file name in the Content-Disposition header as specified in the MIME Specification.

All browsers I know (well, at least those I use regularily :-) handle that perfectly and propose the file name sent in the Content-Disposition header as file name for saving the downloaded name which is usually exactly what I want.

All browsers do that, …, just not my favourite commandline download tool GNU Wget … Downloading the above URL with wget would look like this with default settings:

$ wget 'http://www.example.com/download.cgi?file=foobar.txt'
--2013-10-02 16:04:16--  http://www.example.com/download.cgi?file=foobar
Resolving www.example.com (www.example.com)... 93.184.216.119, 2606:2800:220:6d:26bf:1447:1097:aa7
Connecting to www.switch.ch (www.example.com)|2606:2800:220:6d:26bf:1447:1097:aa7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2020 (2.0K) [text/plain]
Saving to: `download.cgi?file=foobar.txt'

100%[============================================>] 2,020       --.-K/s   in 0s

2013-10-02 16:04:24 (12.5 MB/s) - `download.cgi?file=foobar.txt' saved [2020/2020]

Meh!

But luckily Wget can do that, it’s just not enabled by default — because it’s an experimental and possibly buggy feature, at least according to the man page. Well, works for me! :-)

You can easily enabled it by default for either your user or the whole system by placing the following line in your ~/.wgetrc or /etc/wgetrc:

content-disposition = on

Given the CGI script sends an appropriate Content-Disposition header, the above output now looks like this:

$ wget 'http://www.example.com/download.cgi?file=foobar.txt'
--2013-10-02 16:04:16--  http://www.example.com/download.cgi?file=foobar
Resolving www.example.com (www.example.com)... 93.184.216.119, 2606:2800:220:6d:26bf:1447:1097:aa7
Connecting to www.switch.ch (www.example.com)|2606:2800:220:6d:26bf:1447:1097:aa7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2020 (2.0K) [text/plain]
Saving to: `foobar.txt'

100%[============================================>] 2,020       --.-K/s   in 0s

2013-10-02 16:04:24 (12.5 MB/s) - `foobar.txt' saved [2020/2020]

Now Wget does what I mean!

You can also set this as flag on the commandline, but typing wget --content-disposition … everytime is surely not what I want. ;-)

New web browsers in Wheezy //at 16:14 //by abe

Since there is so much nice new stuff in Debian Wheezy, I have to split up my contributions to Mika’s #newinwheezy game on Planet Debian.

Here’s the next bunch, this time web browsers:

dillo

The FLTK-based lightweight GUI web browser Dillo comes with its own rendering engine (no JavaScript, incomplete CSS support) was already in Debian before, but was removed before the release of Debian Squeeze, because Dillo 2 relied on FLTK 2.x which had an unclear license situation back then and never made it into Debian. In the meanwhile Dillo 3 relies on FLTK 1.3 as FLTK upstream abandoned the 2.0 branch and continued development on the 1.3 branch. So I brought Dillo back into Debian with its 3.0.x release.

netsurf

The RiscOS-originating lightweight GUI web browser Netsurf was already in Debian, too, but didn’t make it into Debian Squeeze as it needed the Lemon parser generator (part of the SQLite source) to build back then and a change in Lemon caused Netsurf to no more build properly in the wrong moment. Netsurf supports CSS 2.1, but has no JavaScript support either. I’d consider its rendering engine more complete than Dillo’s.

surf and xxxterm

Surf and XXXTerm are both simple and minimalistic webkit-based browsers. Surf is easy to embed in other applications and XXXTerm features vi-like keybindings for heavy keyboard users.

To be continued… ;-)

New SSH-related stuff in Wheezy //at 15:28 //by abe

Mika had the nice idea of doing a #newinwheezy game on Planet Debian, so let’s join:

There are (at least) two new SSH related tools new in Debian Wheezy:

mosh

is the “mobile shell”, an UDP based remote shell terminal which works better than SSH in case of lag, packet loss or other forms of bad connection. I wrote about mosh in more detail about a year ago. mosh is also available for Debian Squeeze via squeeze-backports.

sshuttle

is somewhere between port-forwarding and VPN. It allows forward arbitrary TCP connections over an SSH connection without the need to configure individual port forwardings. It does not need root access on the server-side either. I wrote about sshuttle in more detail about a year ago.

To be continued… ;-)

Rendering Markdown, Asciidoc and Friends automatically while Editing //at 15:41 //by abe

Partially because of Markdown being Github’s markup format of choice, I enjoy writing documents in simple markup formats more and more.

There’s though one common annoyance with these formats compared to writing plain HTML…

The Annoyance

They need to be rendered (i.e. more or less compiled) before you can view your outpourings rendered, e.g. in the web browser. So the workflow usually is:

1. Saving the current file in your favourite editor
2. Switch to terminal with commandline
3. Cursor up, Enter
4. Switch to your favourite web browser
5. Hit the reload button

Using a Specialized Editor with Live Preview

One choice would be to use a specific editor with live rendering. The one I know in Debian (from Wheezy on) is ReText (Debian package retext). It supports Markdown and reStructuredText.

But as with most simple GUI editors, I miss there many of the advanced editing commands possible with Emacs.

Using Emacs’ Markdown Mode

Then there is the Markdown Mode for Emacs (part of Debian’s emacs-goodies-el package), where you can get a “preview” by pressing C-c C-c p. But for some reason this takes several seconds, opens a new buffer and window with the rendered HTML code and then starts (hardcoded) Firefox (which is not my preferred web browser). And if you do that a second time without closing Firefox first, it won’t just reload the file but will open a new tab. You might think that just hitting reload should suffice. But no, the new tab has a different file name, so reload doesn’t help. Additionally it may not use my preferred Markdown implementation. Meh.

Well, I probably could fix all those issues with Markdown Mode, it’s only Emacs Lisp. Heck, the called command is even configurable. But fixing at least four issues to fix one workflow annoyance? Maybe some other time, but not as long there are other nice choices…

Using inotifywait to Render on Write

So everytime you save the currently edited file, you immediately want to rerender the same HTML file from it. This can be easily automated by using Linux’ inotify kernel subsystem which notices changes to the filesystem, and reports those to applications which ask for it.

One such tool is inotifywait which can either output all or just specific events, or just exit if the first requested event occurs. With the latter it’s easy to write a while loop on the commandline which regenerates a file after every write access. I use either Pandoc or Asciidoc for that since both generate full HTML pages including header and footer, but you can use that also with Markdown to render just the HTML body. Most browsers render it correctly anyway:

while inotifywait -q -e modify index.md; do pandoc -s -f markdown -t html -o index.html index.md; done
while inotifywait -q -e modify index.txt; do asciidoc index.txt; done
while inotifywait -q -e modify index.md; do markdown index.md > index.html; done

This solution is even editor- and build-system-agnostic (But not operating-system-agnostic.)

inotifywait is part of inotify-tools, a useful set of commandline tools to interface with inotify. They’re packaged in Debian as inotify-tools, too.

Using mdpress for Markdown plus Impress.js based Slides

The ruby-written mdpress is a special case of the previous case. It’s a commandline tool to convert Markdown into Impress.js based slide shows and it has an option named --automatic which causes it to keep running and automatically update the presentation as soon as changes are made to the Markdown file.

mdpress is not yet in Debian, but there’s an ITP for it and Impress.js itself recently entered Debian as libjs-impress. Nevertheless, two dependencies (highlight.js, ITP‘ed, ruby-launchy, ITP‘ed) are still missing in Debian.

Up to date Aptitude Documentation Online //at 12:51 //by abe

Aptitude ships documentation in 7 languages as HTML files. However the latest version available online was 0.4.11.2 from 2008 and hosted on the server by the previous, now unfortunately inactive Aptitude maintainer, and only covered 5 languages.

This lack of up to date online documentation even caused others to put more up to date versions online. Nevertheless they age, too, and the one I’m aware is not up to date for Wheezy.

So the idea was born to keep an up to date version online on Aptitude’s Alioth webspace (which currently redirects to a subdirectory of the previous maintainer’s personal website). But unfortunately we, the current Aptitude Team, are still lacking administrative rights on Aptitude’s Alioth project, which would be necessary to assign new team members who could work on that.

As an intermediate step, there’s now a (currently ;-) up to date Aptitude User’s Manual online in all 7 languages at

http://people.debian.org/~abe/aptitude/

and English at

http://people.debian.org/~abe/aptitude/en/

As this location could also suffer from the same MIA issues as any other “personal” copy, the plan is to move this to somewhere under http://aptitude.alioth.debian.org/ as soon as we have full access to Aptitude’s Alioth project.

Our plans for then are:

• Redirects from http://people.debian.org/~abe/aptitude/ to e.g. http://aptitude.alioth.debian.org/doc/, so that all your links to or bookmarks of http://people.debian.org/~abe/aptitude/ are still valid. (This unfortunately won’t work for jump marks to specific sections, just per file, i.e. chapter.)
• Set up a cron-job, which keeps the documentation in sync with the version of Aptitude in Unstable (and maybe also with Aptitude in Stable).

P.S.: Anyone interested in doing a German translation of the Aptitude User’s Manual? Sources are in DocBook, i.e. XML, and available via Git.