PDiffs are still useful //at 03:26 //by abe

… probably just not as default.

I do agree with Richi and with Michael that disabling PDiffs by default gives the big majority of Debian Testing or Unstable users a speedier package list update.

I’m though not sure, if disabling PDiffs by default would

• also have an performance impact on our mirrors — it surely would have a traffic impact on the mirrors;
• really bring a benefit for Debian Stable users as Debian Stable changes seldomly and hence there are not that many PDiffs to download and apply — at least I can’t remember being annoyed by PDiffs anywhere else than on Debian Testing and Debian Unstable. Even the repositories with security updates don’t change that often.

Additionally I want to remind you that PDiffs per se are nothing bad and should be continued to be supported:

• Because there are still areas, even in “civilized” countries, where only small bandwidth is available and where using PDiffs reduces the download time a lot. Yes, also in Germany. BTDT. Only until recently there was no Fibre, no DSL, very bad UTMS reception and otherwise just EDGE at my parents’ home. (LTE was available far too expensive until recently.) And I was very happy about not having to download 30 MB or such just for seeing if there are updates at all, because 25 kB/s was the fastest download rate I could get (peaks, not average).
• Because it seems to be in fashion with big ISP near-to-monoplists, especially in Germany, to cut off your nice bandwidth if you transfer too many Megabytes. Keyword “Drosselkom”. If you happen to be a customer of such a shitty ISP, you may be happy to reduce your traffic amount by using PDiffs instead of downloading the full package list every time.

So yes, disabling PDiffs by default is probably ok, but the feature must be kept available for those who haven’t 100 MBit/s fibre connection into their homes or are sitting just one hop away from the next Debian mirror (like me at work :-).

Oh, and btw., for the very same reasons I’m also a big fan of debdelta which is approximately the same as PDiffs, just not for package lists but for binary packages. Using debdelta I was able to speed up my download rates over EDGE to up to virtual 100 kB/s, i.e. by factor four (depending on the packages). Just imagine a LibreOffice minor update at 15 kB/s average download rate. ;-)

And all these experiences were not made with a high-performance CPU but with the approximately 5 year old Intel Atom processor of my ASUS EeePC 900A. So I used PDiffs and debdelta even despite having a slight performance penalty on applying the diffs and deltas.

Showing packages newer than in archive with aptitude //at 22:14 //by abe

I happens quite often that I install a manually built, newer version of some package on a machine. Occassionally I forget to remove it or to downgrade it to the version in the APT repo.

$ apt-show-versions | fgrep newer

easily finds those packages.

But usually when doing such a check, I want this list of packages in my aptitude TUI to have a look at the other versions of that package and to take actions. And I don’t want to manually search for each of the package manually.

This can be done with the following “one-liner”:

# aptitude -o "Aptitude::Pkg-Display-Limit=( `apt-show-versions | fgrep newer | awk -F '[ :]' '{printf "~n ^"$1"$ | "}' | sed -e 's/| *$//'` )"

It uses apt-show-version’s output, searches for the right packages, takes the first column and transforms it into an aptitude search pattern matching all packages whose name is exactly one of the listed packages.

But this solution is quite ugly and slow. So I wondered if this is also doable with pure aptitude search patterns which likely would also be faster.

And after some playing around I found the following working aptitude search term:

~i ?any-version(!~O.) !~U !~o

This matches all packages which which are installed and which have a version which has no origin, i.e. no associated APT repository. Since this also matches all hold packages as well as all packages not available in any archive, I use !~U !~o to exclude those packages from that list again.

Since nobody can remember that nor wants to type that everytime needed, I added the following alias to my setup:

alias aptitude-newer-than-in-archive='aptitude -o "Aptitude::Pkg-Display-Limit=~i ?any-version(!~O.) !~U !~o"'

Only caveat so far:

It seems to also match packages from APT repos which haven’t set an “Origin”. This should not happen with any Debian or Ubuntu APT repository, but seems to happen occasionally with privately run APT repositories.

And using ~A instead of ~O, i.e. ~i ?any-version(!~A.), does not work for this case either, despite it matches installed packages of which versions not in any available archive exist. But unfortunately aptitude seems to remember in some way if a package was in some archive in the past, so this only shows packages installed with dpkg -i, but not packages removed from e.g. unstable but with older versions still being available in stable.

Next Debian Meetup in Zurich //at 20:16 //by abe

The first Debian meetup in Zurich last month was quite a success and I look forward to further Debian meetups in Zurich — every first Tuesday of the month.

The next meetup will be

on Tuesday, 2013-Dec-03 starting at 18:30 CET
at St. Gallerhof, Konradstrasse 2, 8005 Zurich

Please note that this is a different location than last month.

Everybody who is interested in Debian is welcome to join us. Registering is not necessary.

There was also some interest in Debian meetups in other Swiss cities, namely Bern and somewhere around Lac Leman. In case you want a Debian meetup elsewhere in Switzerland, too, or if you’re interested in any Debian meetup in Switzerland, feel free to join the Swiss Debian Community Mailing List and help organising other Swiss Debian meetups.

Searching in Screen’s copy mode //at 23:43 //by abe

I’m using GNU Screen daily for definitely more than a decade and I became maintainer of Debian’s screen package nearly exactly two years ago. Nevertheless it still happens occassionally that I discover features yet unknown to me. Recently I had one of these moments again:

I looked for a specific line in the long output of a command which has run inside a Screen session. For that I entered Screen’s copy mode with Ctrl-A [ and scrolled around with arrow keys and page-up and -down keys.

But didn’t find it. I thought, it would be cool if I can search for the string I’m looking for. Intuïtively I typed /, the search string and pressed enter. And it worked! It jumped to the next occurrence of that string.

Of course I immediately had to check if tmux has such a feature, too. And it indeed has, but it seems to be a less sophisticated implementation:

Being able to do incremental search like with GNU Emacs gave me yet another reason for continuing to use Screen and not to switch Tmux. ;-)

How to make wget honour Content-Disposition headers //at 16:12 //by abe

Download links often point to CGI scripts which actually generate (or just fetch, i.e. proxy) the actual file to be downloaded, e.g. URLs like http://www.example.com/download.cgi?file=foobar.txt.

Most of such CGI scripts send the real file name in the Content-Disposition header as specified in the MIME Specification.

All browsers I know (well, at least those I use regularily :-) handle that perfectly and propose the file name sent in the Content-Disposition header as file name for saving the downloaded name which is usually exactly what I want.

All browsers do that, …, just not my favourite commandline download tool GNU Wget … Downloading the above URL with wget would look like this with default settings:

$ wget 'http://www.example.com/download.cgi?file=foobar.txt'
--2013-10-02 16:04:16--  http://www.example.com/download.cgi?file=foobar
Resolving www.example.com (www.example.com)... 93.184.216.119, 2606:2800:220:6d:26bf:1447:1097:aa7
Connecting to www.switch.ch (www.example.com)|2606:2800:220:6d:26bf:1447:1097:aa7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2020 (2.0K) [text/plain]
Saving to: `download.cgi?file=foobar.txt'

100%[============================================>] 2,020       --.-K/s   in 0s

2013-10-02 16:04:24 (12.5 MB/s) - `download.cgi?file=foobar.txt' saved [2020/2020]

Meh!

But luckily Wget can do that, it’s just not enabled by default — because it’s an experimental and possibly buggy feature, at least according to the man page. Well, works for me! :-)

You can easily enabled it by default for either your user or the whole system by placing the following line in your ~/.wgetrc or /etc/wgetrc:

content-disposition = on

Given the CGI script sends an appropriate Content-Disposition header, the above output now looks like this:

$ wget 'http://www.example.com/download.cgi?file=foobar.txt'
--2013-10-02 16:04:16--  http://www.example.com/download.cgi?file=foobar
Resolving www.example.com (www.example.com)... 93.184.216.119, 2606:2800:220:6d:26bf:1447:1097:aa7
Connecting to www.switch.ch (www.example.com)|2606:2800:220:6d:26bf:1447:1097:aa7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2020 (2.0K) [text/plain]
Saving to: `foobar.txt'

100%[============================================>] 2,020       --.-K/s   in 0s

2013-10-02 16:04:24 (12.5 MB/s) - `foobar.txt' saved [2020/2020]

Now Wget does what I mean!

You can also set this as flag on the commandline, but typing wget --content-disposition … everytime is surely not what I want. ;-)

New web browsers in Wheezy //at 16:14 //by abe

Since there is so much nice new stuff in Debian Wheezy, I have to split up my contributions to Mika’s #newinwheezy game on Planet Debian.

Here’s the next bunch, this time web browsers:

dillo

The FLTK-based lightweight GUI web browser Dillo comes with its own rendering engine (no JavaScript, incomplete CSS support) was already in Debian before, but was removed before the release of Debian Squeeze, because Dillo 2 relied on FLTK 2.x which had an unclear license situation back then and never made it into Debian. In the meanwhile Dillo 3 relies on FLTK 1.3 as FLTK upstream abandoned the 2.0 branch and continued development on the 1.3 branch. So I brought Dillo back into Debian with its 3.0.x release.

netsurf

The RiscOS-originating lightweight GUI web browser Netsurf was already in Debian, too, but didn’t make it into Debian Squeeze as it needed the Lemon parser generator (part of the SQLite source) to build back then and a change in Lemon caused Netsurf to no more build properly in the wrong moment. Netsurf supports CSS 2.1, but has no JavaScript support either. I’d consider its rendering engine more complete than Dillo’s.

surf and xxxterm

Surf and XXXTerm are both simple and minimalistic webkit-based browsers. Surf is easy to embed in other applications and XXXTerm features vi-like keybindings for heavy keyboard users.

To be continued… ;-)

New SSH-related stuff in Wheezy //at 15:28 //by abe

Mika had the nice idea of doing a #newinwheezy game on Planet Debian, so let’s join:

There are (at least) two new SSH related tools new in Debian Wheezy:

mosh

is the “mobile shell”, an UDP based remote shell terminal which works better than SSH in case of lag, packet loss or other forms of bad connection. I wrote about mosh in more detail about a year ago. mosh is also available for Debian Squeeze via squeeze-backports.

sshuttle

is somewhere between port-forwarding and VPN. It allows forward arbitrary TCP connections over an SSH connection without the need to configure individual port forwardings. It does not need root access on the server-side either. I wrote about sshuttle in more detail about a year ago.

To be continued… ;-)