Stoppt die Vorratsdatenspeicherung! Jetzt klicken &handeln! Willst du auch an der Aktion teilnehmen? Hier findest du alle relevanten Infos und Materialien:
Jump to menu and information about this site.

Wednesday·24·November·2010

Useful but Unknown Unix Tools: Convert UTF-8 text files to PostScript with paps //at 02:23 //by abe

from the Keysinging dept.

Sometime you get an UTF-8 encoded text file you want to print. But most text to postscript converters (often invoked automatically by your print server) can only render ISO-Latin-1 text files properly and so you get the notorious ä et al on your printout.

This is especially annoying at PGP/GnuPG keysigning parties where nowadays most people have the names in their UIDs encoded in UTF-8.

Fortunately there is paps (Debian package), a Pango based commandline tool to convert UTF-8 encoded text files into PostScript.

See the paps home page for a neat example.

Update 02:10 (CET): Funnily the explicitly as HTML entities “Ô and “¤” written “ä” above got rendered as “ä” in Liferea, but only in the Planet Debian and Planet Symlink feeds, and only until I put that “ä” in this paragraph, because both got converted to their ISO-Latin-1 8-bit equivalent bytes, so that without the “ä”, “ä”, converted to 8-bit ISO-Latin-1 characters looks also like an UTF-8 “ä”. And Liferea seems to guess the character set somehow and if it validates as UTF-8, it uses UTF-8 even if it isn’t UTF-8. This is a strange Planet.

Useful but Unknown Unix Commandline Options: sort -h //at 01:12 //by abe

from the human-sortable dept.

The GNU coreutils command “du” knows about the option “-h” to output human readable (or at least human friendly) values with unit prefixes, e.g. k, M or G.

The GNU coreutils command “sort” also can sort by numbers for quite a long time using the option “-n”, but that doesn’t work on the output of “du -h”. So you usually just did one of the following commands, but couldn’t easily combine them:

$ du -h
$ du | sort -n

For approximately a year, GNU sort now knows about another command line option named “-h”. You guessed it probably: “sort -h” can sort human readable values with SI prefixes, e.g.

$ du -h | sort -h | tail -15
34M     ./ttf-mplus-033/debian/ttf-mplus
34M     ./ttf-mplus-033/debian/ttf-mplus/usr
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share/fonts
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share/fonts/truetype
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share/fonts/truetype/ttf-mplus
35M     ./ttf-mplus-034
57M     ./ttf-mplus-029
60M     ./php5-5.2.6/ext
60M     ./ttf-mplus-030
63M     ./ttf-mplus-031
65M     ./ttf-mplus-032
67M     ./ttf-mplus-033
81M     ./php5-5.2.6
1.5G    .
$

You can get this feature already in Debian Unstable (Sid) and Testing (Squeeze, the upcoming stable release), and Ubuntu Maverick and Natty, but not yet in the current Debian Stable release (Lenny) nor in the last Ubuntu LTS release (Lucid Lynx).

Useful but Unknown Unix Tools: friends of du //at 01:12 //by abe

from the du-on-steroids dept.

You probably all know “du” from GNU Coreutils. But there’s more than just “du” around in Debian.

du | xdu

First there is xdu, which displays du output piped into it graphically, even if “du” is running on some other box with the output coming over an ssh connection:

$ du | xdu
$ ssh server-without-xlibs du some-directory | xdu

Clicking on any directory except the topmost zooms into that directory. Pressing “n” sorts by size and “s” toggles the display of sizes in numbers.

xdiskusage

xdiskusage seems to be a fork of xdu or at least shares some GUI code with it as it looks and behaves quite similar.

But the GUI looks nicer, more modern and you don’t need to pipe the output of du into it. It collects the disk usage of a given directory itself.

ncdu

And then there is ncdu, the “ncurses du”. (It has nothing to do with the German party “CDU”. You can easily remember that as “Not CDU”. ;-) It shows you the disk usage of files and subdirectories in the given directory already sorted by size, optionally displaying also ascii art bars and/or percentage (by pressing “g” one or more times) for better comparision between file and directory sizes. And all easily and intuitively to navigate in an ncurses text-mode user interface.

ncdu 1.3 ~ Use the arrow keys to navigate, press ? for help
--- /home/abe/debian/webwml ------------------------------------
                             /..
   70.2MB [20.0% ##########] /english
   40.9MB [11.7% #####     ] /french
   35.0MB [10.0% ####      ] /german
   30.2MB [ 8.6% ####      ] /swedish
   26.6MB [ 7.6% ###       ] /japanese
   23.2MB [ 6.6% ###       ] /spanish
   21.5MB [ 6.1% ###       ] /portuguese
   19.8MB [ 5.6% ##        ] /danish
   15.2MB [ 4.3% ##        ] /italian
    9.8MB [ 2.8% #         ] /russian
    8.8MB [ 2.5% #         ] /polish
    6.5MB [ 1.8%           ] /finnish
    5.8MB [ 1.6%           ] /chinese
    4.4MB [ 1.3%           ] /catalan
    4.3MB [ 1.2%           ] /dutch
    3.7MB [ 1.0%           ] /korean
    3.4MB [ 1.0%           ] /ukrainian
    3.0MB [ 0.9%           ] /czech
    2.6MB [ 0.7%           ] /croatian
    2.6MB [ 0.7%           ] /bulgarian
    2.4MB [ 0.7%           ] /norwegian
    1.7MB [ 0.5%           ] /hungarian
    1.6MB [ 0.4%           ] /romanian
    1.2MB [ 0.3%           ] /greek
    1.2MB [ 0.3%           ] /turkish
    1.1MB [ 0.3%           ] /slovak
  576.0kB [ 0.2%           ] /Perl
  556.0kB [ 0.2%           ] /arabic
  428.0kB [ 0.1%           ] /lithuanian
 Total disk usage: 351.1MB  Apparent size: 351.1MB  Items: 70653                                                      

And yes, I uploaded the two screenshots shown above also to screenshots.debian.net as both tools had no screenshots available yet.

Perfect Team: autossh and GNU Screen //at 01:06 //by abe

from the shell-functions-for-road-warriors dept.

SSH is definitely one of my most often used tools, not only for system administration at work but also on the road with my netbook, an EeePC 701 running Debian Sid.

On the road, it often happens that I have a flaky WLAN or UMTS connection, so I often have to kill (via <Enter>~.) and reconnect my SSH session due to a changed IP address or so.

First step against problems arising from using SSH over unreliable network connections is of course GNU Screen. Second step is use SSH keys and ssh-agent to not needing to type the password on every reconnect.

But it’s still very annoying to kill the SSH connection and call ssh again manually. For luck there is autossh, a wrapper around SSH which regularily checks via two tunnels connect to each other on the remote side if the connection is still alive, and if not, it kills the ssh and starts a new one with the same parameters (i.e. tunnels, port forwardings, commands to call, etc.).

It’s quite obvious that this is perfect to be combined with screen’s -R and -d options (Reattach if a detached screen is around, else start a new screen; detach a currently attached screen), so I found myself very often typing (or fetching it from the commandline history :-):

autossh -t sym.noone.org 'screen -Rd'

-t is necessary to allocate a terminal device on the remote machine which is not done by default if you directly call a command via ssh.

In comparision to OpenSSH, autossh needs the single quotes, because otherwise it would parse -Rd as options to parse to ssh and bail out. That’s not a real problem, but when you’re used to just type ssh -t sym.noone.org screen -Rd without any quotes, you’ll run into this then and when.

Update, 25-May-2010, 14:55: As Carsten Hey points out, autossh also supports the -- option to declare that all following options and parameters must be passed to ssh itself. (End of Update)

Typing that often and mistyping it then and when cries for an shell alias or an shell function. So I came up with the following shell function:

asc() {
    autossh -x -a -t "$@" 'screen -RdU'
}

I used a function instead of an alias in case of autossh will in future regard all parameters given after the command as part of the command as ssh does.

The additional options -x and -a disable X and SSH Agent forwarding which both don’t work if you reattach to an already running screen.

And if you’re using Zsh as I do, you can even add some more format string magic to set the window title more or less to the expanded alias, eh, function:

function asc() {
    # Set the title to something more obvious, e.g. the expanded
    # alias, eh, function
    print -Pn "\e]0;%n@%m: autossh -t $* 'screen -RdU'\a";
    autossh -x -a -t "$@" 'screen -RdU'
}
compdef asc=ssh

Update, 25-May-2010, 14:59: As Hauke points out in a comment, Zsh users should also declare that asc should have the same tab completion as ssh itself. The example above has been updated accordingly. (End of Update)

In the meantime on the EeePC I use asc on the commandline more often than ssh itself. And I nearly no more type autossh. (The most common exception here is autossh hostname tail -F /path/to/some/logfile.)

Using that function you can also add common ssh options for tunneling, etc. — I use it most often like this:

asc -D 1080 sym.noone.org

This opens a SOCKS proxy on localhost, port 1080 and that way I can surf via the host I’m connecting to by SSH.

There’s one small drawback though: You didn’t expect that I can just invent some new three letter command without a namespace clash, did you? There is a free game called Advanced Strategic Command whose binary (and Debian package) is named asc, too. If you have that game installed, you can always call it using its full path, e.g. /usr/games/asc on Debian.

P.S.: My whole grml based .zshrc is also available via git at git.noone.org as well as on github.

Useful but Unknown Unix Tools: convmv //at 01:05 //by abe

from the UTF-8-is-good-but-the-migration-to-it-sucks-(less-with-convmv) dept.

Ogg and MP3 files often contain the interpreter and song title in the file name. Which is (besides quoting blanks, etc.) usually no issue if you have English interpreters or song titles.

But it may become a character set issue, if you have songs with e.g. French, German, Greek, or Russian interpreters or song titles: They may not be in your preferred character set and therefore don’t display properly on your system. Worst case: You just see questions marks.

If you face such problems, convmv (man page, Debian package) is your tool, because convmv converts filenames from one encoding to another:

$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 ????
$ convmv -f latin1 -t utf-8 *
Your Perl version has fleas #37757 #49830 
Starting a dry run without changes...
mv "./??"       "./äöüß"
No changes to your files done. Use --notest to finally rename the files.
$ convmv --notest -f latin1 -t utf-8 *
Your Perl version has fleas #37757 #49830 
mv "./??"       "./äöüß"
Ready!
$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 äöüß
$ 

Of course it can also recursively rename your whole Ogg collection. And it can do it interactively, too:

$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 ????
$ convmv -i --notest -f latin1 -t utf-8 *
Your Perl version has fleas #37757 #49830 
mv "./??"       "./äöüß" (y/n) y

Ready!
$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 äöüß
$ 

So convmv takes away at least one UTF-8 migration fear.

Update 23:03: Nearly forgot, but Mowgli reminded me: There is also convmvfs (Debian package), a FUSE based filesystem which converts file names on access.

Useful but Unknown Unix Tools: WebDAV client for the commandline //at 01:04 //by abe

from the cadaver-is-no-cadaver dept.

Ever wondered how to test or connect to a WebDAV server on the commandline? Don’t want to use a bloaty GUI file manager to connect to a WebDAV server?

Then use cadaver (Debian package), a WebDAV client reminiscent of the classic ftp commandline client (plus tab completion :-).

$ cadaver https://webdav.phys.ethz.ch/unixhome/abe/vortraege
Authentication required for D-PHYS on server `webdav.phys.ethz.ch':
Username: abe
Password: 
dav:/unixhome/abe/vortraege/> ls
Listing collection `/unixhome/abe/vortraege/': succeeded.
Coll:   wml                                    0  Mar 13  2007
dav:/unixhome/abe/vortraege/> quit
Connection to `webdav.phys.ethz.ch' closed.

There is also nd which works more like ncftpget/ncftpput/wget/wput, but I didn’t get the current version in Debian to work with our WebDAV server. Last time I used nd was back then at ECOS times more than half a decade ago.

Useful but Unknown Unix Tools: Swiss Army Knife for SMTP //at 01:04 //by abe

from the forget-telnet-on-port-25 dept.

Which mail server administrator hasn’t tested his servers using telnet on port 25, speaking SMTP himself? And which of them made a typo and had to retype everything due to missing readline support of telnet? If you want to check SMTP over SSL aka SMTPS, you had to use “openssl s_client -connect yourserver:465”. And what did you use to check TLS support on port 25?

Just forget about those hassles and use swaks (Debian package), the Swiss Army Knife for SMTP. It does all the annoying stuff (using the correct syntax, typing fast enough, SSL, TLS, etc.) for you while still giving you full control over the SMTP session, especially showing you all input and output:

$ swaks -t abe@noone.org -s sym.noone.org -q RCPT
=== Trying sym.noone.org:25...
=== Connected to sym.noone.org.
<-  220 sym.noone.org ESMTP Postfix (Debian/GNU)
 -> EHLO nemo.deuxchevaux.org
<-  250-sym.noone.org
<-  250-PIPELINING
<-  250-SIZE 10240000
<-  250-ENHANCEDSTATUSCODES
<-  250 8BITMIME
 -> MAIL FROM:<abe@nemo.deuxchevaux.org>
<-  250 2.1.0 Ok
 -> RCPT TO:<abe@noone.org>
<** 450 4.2.0 <abe@nemo.deuxchevaux.org>: Sender address rejected: Greylisted, see http://postgrey.schweikert.ch/help/noone.org.html
 -> QUIT
<-  221 2.0.0 Bye
=== Connection closed with remote host.

Since the moment I discovered swaks, I never telnetted to port 25 anymore.

Useful but Unknown Unix Tools //at 01:04 //by abe

from the let's-start-a-column dept.

Most of my talks are either talks about commandline basics (Commandline Helpers about GNU Coreutils and Findutils usage; Understanding Shell Quoting) or collections of small tips and tricks, e.g. about SSH, History Expansion, or commandline tools. Valentin Haenel called this second type of talks “nugget talks”. I like that term.

My newest talk (German only so far, next occasion at the LUGS meeting in Zurich on 16th of December 2010) is such a nugget talk and is about useful but unknown tools. While preparing that talk I noticed that the amount of neat but less known tools I collected for possible inclusion into that talk would be enough for a whole day or so. I noticed that I’ll never be able to give a talk about all that stuff.

To not letting these tips rot in that file uncommented, I decided that I’ll write (more or less) short blog postings about these tools, similar to and inspired by Myon’s blog postings about Cool Unix Features as well as the no more existing or at least broken Debian Package of the Day and Debian Package a Day.

While Myon mostly posted long-time but still unknown features, my postings will mostly cover useful but unknown packages available in Debian and most debian-derived distributions like grml and Ubuntu. Some postings may also just highlight some features of common tools or packages like e.g. GNU Coreutils.

You can subscribe to these tips via RSS or just view all of them in your web browser at http://noone.org/blog/English/Computer/Debian/CoolTools/.

Multiple Move & Co. //at 00:08 //by abe

from the variations dept.

nion’s blog made me notice that many people don’t know mmv (multiple move), which approximately works like this:

mmv '*.htm' #1.html
mmv '*.foo.*' #1.#2.bla

Additionally, mmv also can copy, link or even append files when called as mcp, mln or mad respectively with the appropriate command line options.

When I told nion in IRC on #debian.de about mmv, HE pointed me to the Perl script /usr/bin/rename, which is in Debian’s perl package and therefore installed on nearly every Debian system by default. It moves files by applying perl subsitutions to file names:

rename 's/\.htm$/.html/' *.htm
rename 'y/A-Z/a-z/' *

Being curious, if the newly found tool is not only available in Debian, I looked on a SuSE 9.0 box and indeed, I also found there a /usr/bin/rename. But — surprise, surprise — it’s not a Perl script but an ELF binary. And although it does similar things than mmv and Debian’s rename, it is the simplest of the three commands:

rename .htm .html *.htm
rename foo foo00 foo?
rename foo foo0 foo??

Note to my self: Nice add-on for your command line efficiency talk.

autossh vs TCP resetter //at 00:08 //by abe

from the lick-my-ass-script-kiddie dept.

LUG-Camp 2007 in Interlaken is nearly over, and I’m reading my mail as usual using ssh, screen and mutt on the server. But the ssh connection resets every few minutes. According to the LUSC people (who are running the gateway) some script kiddie is running a TCP resetter somewhere in the network.

I remembered that I read about autossh in the Debian package list once a while and that it sounded cool but I had no use for it yet. Until now.

I’m writing this over the same crashing ssh connection and I’m typing without taking big notice of the quite often occurring connection resets:

autossh noone.org -t 'screen -rd'

It just works. :-)

Group packages by origin in aptitude //at 00:06 //by abe

from the reasons-for-loving-debian dept.

I always wondered how others recognise non-Debian packages in the aptitude package tree. I also missed the additional priority level in the hierachy well-known from good old dselect.

For the last one, I quickly found out that you can set the priority as subsection — it’s straight forward after you’ve read the documentation: Just add ,priority at the end of the default grouping method for package views under “Options → UI Options” in the aptitude menu.

Getting the origin as given in the Release file of the repository a package originates from is a little bit more difficult. You need to use the pattern() group function with the appropriate search pattern: pattern(~O)

Since already the default default grouping method for package views doesn’t fit into the dialog, I nowadays just edit /etc/apt/apt.conf directly for changes on aptitude’s default grouping method for package views. It now looks like this on several of my machines:

Aptitude::UI {
  Default-Grouping "filter(missing),status,section(subdir,passthrough),pattern(~O),section(topdir),priority";
};

In aptitude this looks like this:

[...]
  --- text - Text processing utilities
  --\ utils - Various system utilities
    --- Debian
    --- Mowgli
    --- volatile.debian.org
  --\ web - Web browsers, servers, proxies, and other tools
    --- Debian
    --- Opera Software ASA
  --\ x11 - The X window system and related software
    --\ Debian
      --- contrib - Programs which depend on software not in Debian
      --\ main - The main Debian archive
        --- Priority optional
        --- Priority extra
      --- non-free - Programs which are not free software
    --- Mowgli
[...]

Unfortunately this doesn’t work with all non-Debian repositories since a few repository maintainer, e.g. those from Emdebian, arrogate to just keep “Debian” as their packages’ origin. This could be solved, if there’s a possibility to group by e.g. repository URL (host and/or path).

Another problem I haven’t solved yet is that grouping by origin does neither work with locally created nor virtual packages nor tasks — probably since all of them lack an origin. Those branches are just empty or don’t even show up anymore with this configuration. I probably have to dig a little bit more in the aptitude documentation to resolve this.

Now playing: E-Rotic — Max don’t have sex with your ex

Useful but Unknown Unix Tools: netselect //at 00:05 //by abe

from the tune-your-sources.list dept.

Ever wondered which mirror of your favourite Linux distribution is the fastest at your location?

Check it with netselect (code at GitHub). It checks for the number of hops and ping times to given hosts and tells you which one is the fastest of them:

# netselect -vv ftp.de.debian.org ftp2.de.debian.org \
                ftp.ch.debian.org ftp.nl.debian.org ftp.debian.org
Running netselect to choose 1 out of 5 addresses.
.......................................................
ftp.de.debian.org                       25 ms  16 hops   90% ok ( 9/10) [   72]
ftp2.de.debian.org                      17 ms  17 hops   90% ok ( 9/10) [   51]
ftp.ch.debian.org                        0 ms   3 hops   90% ok ( 9/10) [    0]
ftp.nl.debian.org                       22 ms  15 hops   90% ok ( 9/10) [   62]
ftp.debian.org                          22 ms  15 hops   90% ok ( 9/10) [   60]
    0 ftp.ch.debian.org

And if you’re too lazy to optimize your sources.list with netselect manually, just use the netselect-apt package. It will do it for you.

Tag Cloud

2CV, aha, Apache, APT, aptitude, ASUS, Automobiles, autossh, Berlin, bijou, Blogging, Blosxom, Blosxom Plugin, Browser, BSD, CDU, Chemnitz, Citroën, CLI, CLT, Conkeror, CX, deb, Debian, Doofe Parteien, E-Mail, eBay, EeePC, Emacs, Epiphany, Etch, ETH Zürich, Events, Experimental, Firefox, Fläsch, FreeBSD, FVWM, Galeon, Gecko, git, GitHub, GNOME, GNU, GNU Coreutils, GNU Screen, Google, GPL, grep, grml, gzip, Hackerfunk, Hacks, Hardware, Heise, HTML, identi.ca, IRC, irssi, Jabber, JavaShit, Kazehakase, Lenny, Liferea, Linux, LinuxTag, LUGS, Lynx, maol, Meme, Microsoft, Mozilla, Music, mutt, Myon, München, nemo, Nokia, nuggets, Open Source, Opera, packaging, Pentium I, Perl, Planet Debian, Planet Symlink, Quiz, Rant, ratpoison, Religion, RIP, Sarcasm, Sarge, Schweiz, screen, Shell, Sid, Spam, Squeeze, SSH, Stöckchen, SuSE, Symlink, Symlink-Artikel, Tagging, Talk, taz, Text Mode, ThinkPad, Ubuntu, USA, USB, UUUCO, UUUT, VCFe, Ventilator, Vintage, Wahlen, Wheezy, Wikipedia, Windows, WML, Woody, WTF, X, Xen, zsh, Zürich, ÖPNV

Calendar

 2010 
Months
Nov
 November 
Mo Tu We Th Fr Sa Su
24
         

Tattletale Statistics

Blog postings by posting time
Blog posting times this month



Search


Advanced Search


Categories


Recent Postings

13 most recent of 277 postings total shown.


Recent Comments

Hackergotchi of Axel Beckert

About...

This is the blog or weblog of Axel Stefan Beckert (aka abe or XTaran) who thought, he would never start blogging... (He also once thought, that there is no reason to switch to this new ugly Netscape thing because Mosaïc works fine. That was about 1996.) Well, times change...

He was born 1975 at Villingen-Schwenningen, made his Abitur at Schwäbisch Hall, studied Computer Science with minor Biology at University of Saarland at Saarbrücken (Germany) and now lives in Zürich (Switzerland), working at the IT Support Group (ISG) of the Departement of Physics at ETH Zurich.

Links to internal pages are orange, links to related pages are blue, links to external resources are green and links to Wikipedia articles, Internet Movie Database (IMDb) entries or similar resources are bordeaux. Times are CET respective CEST (which means GMT +0100 respective +0200).


RSS Feeds


Identity Archipelago


Picture Gallery


Button Futility

Valid XHTML Valid CSS
Valid RSS Any Browser
GeoURL
This content is licensed under a Creative Commons License (SA 3.0 DE). Some rights reserved. Hacker Emblem
Get Mozilla Firefox! Powered by Linux!
Typed with GNU Emacs Listed at Tux Mobil
XFN Friendly Button Maker

Blogroll

Blog or not?


People I know personally


Other blogs I like or read


Independent News


Interesting Planets


Web comics I like and read

Stalled Web comics I liked


Blogging Software

Blosxom Plugins I use

Bedside Reading

Just read

  • Bastian Sick: Der Dativ ist dem Genitiv sein Tod (Teile 1-3)
  • Neil Gaiman and Terry Pratchett: Good Omens (borrowed from Ermel)

Currently Reading

  • Douglas R. Hofstadter: Gödel, Escher, Bach
  • Neil Gaiman: Keine Panik (borrowed from Ermel)

Yet to read

  • Neil Stephenson: Cryptonomicon (borrowed from Ermel)

Always a good snack

  • Wolfgang Stoffels: Lokomotivbau und Dampftechnik (borrowed from Ermel)
  • Beverly Cole: Trains — The Early Years (getty images)

Postponed