Blogging is futile

New web browsers in Wheezy

abe+blog@deuxchevaux.org (Axel Beckert) — Thu, 02 May 2013 16:14:21 +0200

Since there is so much nice new stuff in Debian Wheezy, I have to split up my contributions to Mika’s #newinwheezy game on Planet Debian.

Here’s the next bunch, this time web browsers:

dillo: The FLTK-based lightweight GUI web browser Dillo comes with its own rendering engine (no JavaScript, incomplete CSS support) was already in Debian before, but was removed before the release of Debian Squeeze, because Dillo 2 relied on FLTK 2.x which had an unclear license situation back then and never made it into Debian. In the meanwhile Dillo 3 relies on FLTK 1.3 as FLTK upstream abandoned the 2.0 branch and continued development on the 1.3 branch. So I brought Dillo back into Debian with its 3.0.x release.
netsurf: The RiscOS-originating lightweight GUI web browser Netsurf was already in Debian, too, but didn’t make it into Debian Squeeze as it needed the Lemon parser generator (part of the SQLite source) to build back then and a change in Lemon caused Netsurf to no more build properly in the wrong moment. Netsurf supports CSS 2.1, but has no JavaScript support either. I’d consider its rendering engine more complete than Dillo’s.
surf and xxxterm: Surf and XXXTerm are both simple and minimalistic webkit-based browsers. Surf is easy to embed in other applications and XXXTerm features vi-like keybindings for heavy keyboard users.

To be continued… ;-)

New SSH-related stuff in Wheezy

abe+blog@deuxchevaux.org (Axel Beckert) — Thu, 02 May 2013 15:28:22 +0200

Mika had the nice idea of doing a #newinwheezy game on Planet Debian, so let’s join:

There are (at least) two new SSH related tools new in Debian Wheezy:

mosh: is the “mobile shell”, an UDP based remote shell terminal which works better than SSH in case of lag, packet loss or other forms of bad connection. I wrote about mosh in more detail about a year ago. mosh is also available for Debian Squeeze via squeeze-backports.
sshuttle: is somewhere between port-forwarding and VPN. It allows forward arbitrary TCP connections over an SSH connection without the need to configure individual port forwardings. It does not need root access on the server-side either. I wrote about sshuttle in more detail about a year ago.

To be continued… ;-)

Rendering Markdown, Asciidoc and Friends automatically while Editing

abe+blog@deuxchevaux.org (Axel Beckert) — Sun, 10 Mar 2013 15:41:58 +0100

Partially because of Markdown being Github’s markup format of choice, I enjoy writing documents in simple markup formats more and more.

There’s though one common annoyance with these formats compared to writing plain HTML…

The Annoyance

They need to be rendered (i.e. more or less compiled) before you can view your outpourings rendered, e.g. in the web browser. So the workflow usually is:

Saving the current file in your favourite editor
Switch to terminal with commandline
Cursor up, Enter
Switch to your favourite web browser
Hit the reload button

Using a Specialized Editor with Live Preview

One choice would be to use a specific editor with live rendering. The one I know in Debian (from Wheezy on) is ReText (Debian package retext). It supports Markdown and reStructuredText.

But as with most simple GUI editors, I miss there many of the advanced editing commands possible with Emacs.

Using Emacs’ Markdown Mode

Then there is the Markdown Mode for Emacs (part of Debian’s emacs-goodies-el package), where you can get a “preview” by pressing C-c C-c p. But for some reason this takes several seconds, opens a new buffer and window with the rendered HTML code and then starts (hardcoded) Firefox (which is not my preferred web browser). And if you do that a second time without closing Firefox first, it won’t just reload the file but will open a new tab. You might think that just hitting reload should suffice. But no, the new tab has a different file name, so reload doesn’t help. Additionally it may not use my preferred Markdown implementation. Meh.

Well, I probably could fix all those issues with Markdown Mode, it’s only Emacs Lisp. Heck, the called command is even configurable. But fixing at least four issues to fix one workflow annoyance? Maybe some other time, but not as long there are other nice choices…

Using inotifywait to Render on Write

So everytime you save the currently edited file, you immediately want to rerender the same HTML file from it. This can be easily automated by using Linux’ inotify kernel subsystem which notices changes to the filesystem, and reports those to applications which ask for it.

One such tool is inotifywait which can either output all or just specific events, or just exit if the first requested event occurs. With the latter it’s easy to write a while loop on the commandline which regenerates a file after every write access. I use either Pandoc or Asciidoc for that since both generate full HTML pages including header and footer, but you can use that also with Markdown to render just the HTML body. Most browsers render it correctly anyway:

while inotifywait -q -e modify index.md; do pandoc -s -f markdown -t html -o index.html index.md; done
while inotifywait -q -e modify index.txt; do asciidoc index.txt; done
while inotifywait -q -e modify index.md; do markdown index.md > index.html; done

This solution is even editor- and build-system-agnostic (But not operating-system-agnostic.)

inotifywait is part of inotify-tools, a useful set of commandline tools to interface with inotify. They’re packaged in Debian as inotify-tools, too.

Using mdpress for Markdown plus Impress.js based Slides

The ruby-written mdpress is a special case of the previous case. It’s a commandline tool to convert Markdown into Impress.js based slide shows and it has an option named --automatic which causes it to keep running and automatically update the presentation as soon as changes are made to the Markdown file.

mdpress is not yet in Debian, but there’s an ITP for it and Impress.js itself recently entered Debian as libjs-impress. Nevertheless, two dependencies (highlight.js, ITP‘ed, ruby-launchy, ITP‘ed) are still missing in Debian.

Up to date Aptitude Documentation Online

abe+blog@deuxchevaux.org (Axel Beckert) — Sun, 10 Mar 2013 12:51:26 +0100

Aptitude ships documentation in 7 languages as HTML files. However the latest version available online was 0.4.11.2 from 2008 and hosted on the server by the previous, now unfortunately inactive Aptitude maintainer, and only covered 5 languages.

This lack of up to date online documentation even caused others to put more up to date versions online. Nevertheless they age, too, and the one I’m aware is not up to date for Wheezy.

So the idea was born to keep an up to date version online on Aptitude’s Alioth webspace (which currently redirects to a subdirectory of the previous maintainer’s personal website). But unfortunately we, the current Aptitude Team, are still lacking administrative rights on Aptitude’s Alioth project, which would be necessary to assign new team members who could work on that.

As an intermediate step, there’s now a (currently ;-) up to date Aptitude User’s Manual online in all 7 languages at

http://people.debian.org/~abe/aptitude/

and English at

http://people.debian.org/~abe/aptitude/en/

As this location could also suffer from the same MIA issues as any other “personal” copy, the plan is to move this to somewhere under http://aptitude.alioth.debian.org/ as soon as we have full access to Aptitude’s Alioth project.

Our plans for then are:

Redirects from http://people.debian.org/~abe/aptitude/ to e.g. http://aptitude.alioth.debian.org/doc/, so that all your links to or bookmarks of http://people.debian.org/~abe/aptitude/ are still valid. (This unfortunately won’t work for jump marks to specific sections, just per file, i.e. chapter.)
Set up a cron-job, which keeps the documentation in sync with the version of Aptitude in Unstable (and maybe also with Aptitude in Stable).

P.S.: Anyone interested in doing a German translation of the Aptitude User’s Manual? Sources are in DocBook, i.e. XML, and available via Git.

Suggestions for the GNOME Team

abe+blog@deuxchevaux.org (Axel Beckert) — Wed, 21 Nov 2012 23:01:02 +0100

Thanks to Erich Schubert’s blog posting on Planet Debian I became aware of the 2012 GNOME User Survey at Phoronix.

Like back in 2006 I still use some GNOME applications, so I do consider myself as “GNOME user” in the widest sense and hence I filled out that survey. Additionally I have to live with GNOME 3 as a system administrator of workstations, and that’s some kind of usage, too. ;-)

The last question in the survey was Do you have any comments or suggestions for the GNOME team? — Sure I have. And since I tried to give constructive feedback instead of only ranting, here’s my answer to that question as I submitted it in the survey, too, just spiced up with some hyperlinks and highlighting:

Don’t try to change the users. Give the users more possibilities to change GNOME if they don’t agree with your own preferences and decisions. (The trend to castrate the user was already starting with GNOME 2 and GNOME 3 made that worse IMHO.)
If you really think that you need less configurability because some non-power-users are confused or challenged by too many choices, then please give the other users at least the chance to enable more configuration options. A very good example in that hindsight was Kazehakase (RIP) who offered several user interfaces (novice, intermediate and power user or such). The popular text-mode web browser Lynx does the same, too, btw.
GNOME lost me mostly with the change to GNOME 2. The switch from Galeon 1.2 to 1.3/2.0 was horrible and the later switch to Epiphany made things even worse on the browser side. My short trip to GNOME as desktop environment ended with moving back to FVWM (configurable without tons of clicking, especially after moving to some other computer) and for the browser I moved on to Kazehakase back then. Nowadays I’m living very well with Awesome and Ratpoison as window managers, Conkeror as web browser (which are all very configurable) and a few selected GNOME applications like Liferea (luckily still quite configurable despite I miss Gecko’s about:config since the switch to WebKit), GUCharmap and Gnumeric.
For people switching from Windows I nowadays recommend XFCE or maybe LXDE on low-end computers. I likely would recommend GNOME 2, too, if it still would exist. With regards to MATE I’m skeptical about its persistance and future, but I’m glad it exists as it solves a lot of problems and brings in just a few new ones. Cinnamon as well as SolusOS are based on the current GNOME libraries and are very likely the more persistent projects, but also very likely have the very same multi-head issues we’re all barfing about at work with Ubuntu Precise. (Heck, am I glad that I use Awesome at work, too, and all four screens work perfectly as they did with FVWM before.)

Thanks to Dirk Deimeke for his (German written) pointer to Marcus Moeller’s interview with Ikey Doherty (in German, too) about his Debian-/GNOME-based distribution SolusOS.

zutils: zcat and friends on Steroids

abe+blog@deuxchevaux.org (Axel Beckert) — Wed, 21 Nov 2012 01:18:11 +0100

I recently wrote about tools to handle archives conveniently. If you just have to handle compressed text files, there are some widely known shortcut commands to mimic common commands on files compressed with a specific compression format.

	gzip	bzip2	lzma	xz
cat	zcat	bzcat	lzcat	xzcat
cmp	zcmp	bzcmp	lzcmp	xzcmp
diff	zdiff	bzdiff	lzdiff	xzdiff
grep	zgrep	bzgrep	lzgrep	xzgrep
egrep	zegrep	bzegrep	lzegrep	xzegrep
fgrep	zfgrep	bzfgrep	lzfgrep	xzfgrep
more	zmore	bzmore	lzmore	xzmore
less	zless	bzless	lzless	xzless

In Debian and derivatives, those tools are part of the according package for that compression utility, i.e. the zcat command is part of the gzip package and the xzfgrep command is part of the xz-utils package.

But despite this matrix is quite easy to remember, the situation has a few drawbacks:

Those tools can only handle the format they’re written for (which btw. means that all xz-tools can also handle lzma-compressed files as lzma is xz’s predecessor)
zcat and the other cat variants can’t even recognize non-compressed files and throw an error instead of just showing their contents.
I always tend to think that lzcat and friends are for lzip-based compression as xzcat can handle lzma-compressed files anyway.

This is where the zutils project comes in: zutils provides the functionality of most of these utilities, too, but with one big difference: You don’t have to remember, think about or type which compression method has been used for your data, just use zcat, zcmp, zdiff, zgrep, zegrep, or zfgrep and it works — independently of what compression method has been used — if any — or if there are different compression types mixed in the parameters to the same command:

$ zfgrep foobar bla.txt fnord.gz hurz.xz quux.lz bar.lzma

Especially if you use logrotate and let logrotate compress old logs, it’s very comfortable that one command suffices to concatenate all the available logfiles, including the current uncompressed one:

$ zcat /var/log/syslog* | …

Additionally, zutils’ versions of these tools also support lzip-compressed files.

The zutils package is available in Debian starting with Wheezy and in Ubuntu since Oneiric. When being installed, it replaces the original z* utilities from the gzip package by diverting them away.

The only drawback so far is that there neither a zless nor a zmore utility from the zutils project, so zless bla.txt fnord.gz hurz.xz quux.lz bar.lzma will not work as expected even after installing zutils as it is still the one from the gzip package and hence it will show you just the first two files in plain text, but not the remaining ones.

deepgrep: grep nested archives with one command

abe+blog@deuxchevaux.org (Axel Beckert) — Sat, 17 Nov 2012 02:00:12 +0100

Several months ago, I wrote about grep everything and listed grep-like tools which can grep through compressed files or specific data formats. The blog posting sparked several magazine articles and talks by Frank Hofmann and me.

Frank recently noticed that we though missed one more or less mighty tool so far. We missed it, because it’s mostly unknown, undocumented and hidden behind a package name which doesn’t suggest a real recursive “grep everything”:

deepgrep

deepgrep is part of the Debian package strigi-utils, a package which contains utilities related to the KDE desktop search Strigi.

deepgrep especially eases the searching through tar balls, even nested ones, but can also search through zip files and OpenOffice.org/LibreOffice documents (which are actually zip files).

deepgrep seems to support at least the following archive and compression formats:

tar
ar, and hence deb
rpm (but not cpio)
gzip/gz
bzip2/bz2
zip, and hence jar/war and OpenOffice.org/LibreOffice documents
MIME messages (i.e. files attached to e-mails)

A search in an archive which is deeply nested looks like this:

$ deepgrep bar foo.ar
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz/foo.txt:foobar
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz/foo.txt:bar

deepgrep though neither seems to support any LZMA based compression (lzma, xz, lzip, 7z), nor does it support lzop, rzip, compress (.Z suffix), cab, cpio, xar, or rar.

Further current drawbacks of deepgrep:

Nearly no commandline options, especially none of the common grep options
No man-page or other documentation
Exit code not related to search results, you have to check the output to see if something has been found

deepfind

If you just need the file names of the files in nested archives, the package also contains the tool deepfind which does nothing else than to list all files and directories in a given set of archives or directories:

$ deepfind foo.ar
foo.ar
foo.ar/foo.tar
foo.ar/foo.tar/foo.tar.gz
foo.ar/foo.tar/foo.tar.gz/foo.zip
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz/foo.txt

As with deepgrep, deepfind does not implement any common options of it’s normal sister tool find.

[The following part has been added on 17-Nov-2012]

As with deepgrep, it also doesn’t seem to support any of the more modern or more exotic compression formats, i.e. it fails on modern debian binary packages which use xz compression on the data part:

deepfind xulrunner-18.0_18.0\~a2+20121109042012-1_amd64.deb
xulrunner-18.0_18.0~a2+20121109042012-1_amd64.deb
xulrunner-18.0_18.0~a2+20121109042012-1_amd64.deb/debian-binary
xulrunner-18.0_18.0~a2+20121109042012-1_amd64.deb/control.tar.gz
xulrunner-18.0_18.0~a2+20121109042012-1_amd64.deb/control.tar.gz/triggers
xulrunner-18.0_18.0~a2+20121109042012-1_amd64.deb/control.tar.gz/preinst
xulrunner-18.0_18.0~a2+20121109042012-1_amd64.deb/control.tar.gz/md5sums
xulrunner-18.0_18.0~a2+20121109042012-1_amd64.deb/control.tar.gz/postinst
xulrunner-18.0_18.0~a2+20121109042012-1_amd64.deb/control.tar.gz/control
xulrunner-18.0_18.0~a2+20121109042012-1_amd64.deb/data.tar.xz

[End of part added at 17-Nov-2012]

Dependencies

The package strigi-utils doesn’t pull in the complete Strigi framework (i.e. no daemon), just a few libraries (libstreams, libstreamanalyzer, and libclucene). On Wheezy it also pulls in some audio/video decoding libraries which may make some server administrators less happy.

Conclusion

Both tools are quite limited to some basic use cases, but can be worth a fortune if you have to work with nested archives. Nevertheless the claim in the Debian package description of strigi-utils that they’re “enhanced” versions of their well known counterparts is IMHO disproportionate.

Most of the missing features and documentation can be explained by the primary purpose of these tools: Being backend for desktop searches. I guess, there wasn’t much need for proper commandline usage yet. Until now. ;-)

42.zip

And yes, I was curious enough to let deepfind have a look at 42.zip (the one from SecurityFocus, unzip seems not able to unpack 42.zip from unforgettable.dk due a missing version compatibility) and since it just traverses the archive sequentially, it has no problem with that, needing just about 5 MB of RAM and a lot of time:

[…]
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page e.zip
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page e.zip/0.dll
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page f.zip
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page f.zip/0.dll
deepfind 42.zip  11644.12s user 303.89s system 97% cpu 3:24:02.46 total

I though won’t try deepgrep on 42.zip. ;-)