Useful but Unknown Unix Tools: colored cal(endar) //at 02:09 //by abe

Another thing I regard as useful on the commandline are colors. (Others commonly refer to my color preferences as eye cancer, but that’s not relevant here. ;-) Colors help to easily distinguish between relevant and non-relevant things or to separate different things.

One colored command line program I use nearly everyday is the colored fork of bsdmainutils’s well know (or at least better known) “cal” program whose output looks like this:

$ cal
    November 2010    
Su Mo Tu We Th Fr Sa
    1  2  3  4  5  6
 7  8  9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30
$ 

Since the colored fork calls itself still “cal” (as the Debian source package is called), to avoid conflicts with bsdmainutils’s cal, the binary package and the binary itself are called ccal (short for colored cal) and its output looks like this:

$ ccal
    November 2010     
 Mo Tu We Th Fr Sa Su 
  1  2  3  4  5  6  7 
  8  9 10 11 12 13 14 
 15 16 17 18 19 20 21 
 22 23 24 25 26 27 28 
 29<30>               
                      
$ 

ccal also use the locale definitions (LC_TIME), if available, that’s why the one “screenshot” is Sun-Sat and the other one is Mon-Sun.

According to the man page, ccal has some more improvements over the classic cal:

But I must admit, I don’t use that feature. I just use it as on-demand calendar sheet.

The colored “screenshot” above is btw. generated with Adam Borowski’s ansi2html which popped up after I published my Intent to package “aha” (for Ansi HTML Adapter), the first ANSI colors to HTML converter I found on the net. (And yes I searched for it because of this and some yet to come blog postings. No spoilers here though. ;-)

While capturing colored output of “ls” and some other tools was easy, it needed a little hack to capture ccal’s output, because if you pipe ccal output to anything, it drops its colors. Always. The solution was to run it in GNU Screen, log the output and then pipe the log file through “aha” or “ansi2html”:

$ screen -c /dev/null -L ccal; cat screenlog.0 | ansi2html > ccal.html

Probably the output of every program which drops colors when writing to a pipe instead of a terminal can be fetched that way.

Update, 2:04 CET: As Adam Borowski points out in message 62 of #605380, “script” is the better tool for making tools believe they talk to a terminal:

$ script -q /dev/null -c "ccal" | ansi2html

Catching the output of a whole year’s calender (“ccal 2011”) failed with GNU screen though, as it was truncated on the 24th of September by GNU screen (as it was already truncated in the screenlog.0), so no “screenshot” of that for the moment. And with script, I’d get the whole output, but would have to manually fix the styles so they don’t get posted literally to Planet Debian. So look for yourself how the output of “ccal 2011” looks like by installing ccal. ;-)

Useful but Unknown Unix Commandline Options of cat //at 14:04 //by abe

Yesterday in Harald König’s commandline talk at LinuxDay.at, I first heard of cat’s -e option, and noticed that cat has quite some interesting options. From the man page of cat:

-A, --show-all

equivalent to -vET

-b, --number-nonblank

number nonempty output lines

-e

equivalent to -vE

-E, --show-ends

display $ at end of each line

-n, --number

number all output lines

-s, --squeeze-blank

suppress repeated empty output lines

-t

equivalent to -vT

-T, --show-tabs

display TAB characters as ^I

-v, --show-nonprinting

use ^ and M- notation, except for LFD and TAB

Some examples made on an UTF-8 terminal:

user@host:demo $ cat bla
ä  ö  ü ß   


abc
user@host:demo $ cat -Ab bla
     1  M-CM-$  M-CM-6  M-CM-<^IM-CM-^_   $
$
$
     2  abc$
user@host:demo $ cat -Ens bla
     1  ä  ö  ü ß   $
     2	$
     3  abc$
user@host:demo $ 

I hope this blog posting will reduce the percentage of “useless uses of cat” by raising the amount of “useful uses of cat”. :-)

Useful but Unknown Unix Tools: units //at 14:56 //by abe

Ever wondered how to easily convert e.g. 10 seamiles into kilometres? Use units:

$ units 
2411 units, 71 prefixes, 33 nonlinear units

You have: 10 seamiles
You want: km
        * 18.288
        / 0.054680665
You have: ^C
$

Of course this is interactive. There’s also a non-interactive mode:

$ units '10 seamiles' 'km'
        * 18.288
        / 0.054680665
$ 

The line with the asterisk means that 10 seamiles are 18.288 kilometres or 1 kilometre is the 0.054680665th part of 10 seamiles.

This quite non-intuitive output is caused by the fact that unit is designed to be used with units only:

$ units seamiles km
        * 1.8288
        / 0.54680665
$

Now this makes more sense: You have to multiply seamiles with 1.8288 to get kilometres and you have to multiply (not divide) kilometres with 0.54something to get seamiles.

But this output is still a little bit cumbersome, and annoying if you want to use it in shell scripts. But for luck, units knows some nice options, especially “-v” (“–verbose”) and “-t” (“–terse”):

$ units -v '10 seamiles' km
        10 seamiles = 18.288 km
        10 seamiles = (1 / 0.054680665) km
$ units -t '10 seamiles' km
18.288

Now that’s way easier to read and script!

You can also script more complex things like

$ units -v '100 attoparsec/microfortnight' m/s
        100 attoparsec/microfortnight = 2.5509901 m/s
        100 attoparsec/microfortnight = (1 / 0.39200466) m/s

Unfortunately not all common units are unambiguous for units:

units -v '100 km/h' m/s
conformability error
        100 km/h = 1.5091905e+38 s / kg m
        m/s = 1 m / s

Well, “h” seems not to be units unit for “hours”, so lets tell it explicity that we want km per hour:

units -v '100 km/hour' m/s
        100 km/hour = 27.777778 m/s
        100 km/hour = (1 / 0.036) m/s

Looks more like what I expected.

“units” behaves though a little bit strange when I try to convert litres per 100 kilometres into miles per gallon:

$ units -t '6L/100km' 'mpg'
conformability error
6e-08 m^2
425143.71 / m^2
$

Interestingly changing the verbosity helps already in this case:

$ units -v '6L/100km' 'mpg'
        reciprocal conversion
        1 / (6L/100km) = 39.202431 mpg
        1 / (6L/100km) = (1 / 0.025508622) mpg

Greetings from the Debian booth at LinuxDay.at and thanks to Y_Plentyn, rhalina and bzed for example ideas and the suggestion to write a blog posting about units. :-)

Useful but Unknown Unix Tools: awsetbg //at 19:03 //by abe

Have you ever wondered which tool you should use to set the background of your X11 desktop? There are tons of it and every single one does need different command line options. And if you picked one, you could be sure, you’ll once come to a machine or operating system, where that one isn’t available.

When I started to play around with the tiling window manager awesome, I also noticed that the awesome Debian package contains awesome’s own desktop background image setting tool named “awsetbg”.

“Yet another one!” I thought.

But I was wrong: Today I noticed the word “wrapper” in awsetbg’s man page. I became curious and had a peek into its source code. And indeed, it’s an awesome desktop background image setting POSIX bourne shell compatible shell-script: It knows about not less than 15 other desktop background image setting tools and can be used as frontend wrapper for those who don’t want to remember each tool’s command line syntax:

• Eterm’s Esetroot
• habak
• feh
• hsetroot
• chbg
• FVWM’s fvwm-root
• ImageMagick’s display
• qiv
• xv (non-free, .deb packages available from Mowgli)
• xsri
• xli
• xsetbg from xloadimage
• WindowMaker’s wmsetbg
• xsetroot from x11-xserver-utils
• IceWM’s icewmbg (currently disabled by default with comment “broken”)

Needless to say that most of them are also available in Debian. :-)

You may wonder what’s useful with it. Well, it does something you no more need to do yourself or in one of your own scripts: Checking which desktop background image setting tools are available on the machine you’re currently working and using the appropriate command line options for tiling, fullscreen, whatever.

Only drawback I found so far: In Debian, you need the awesome window manager package to be installed, if you want to get the script via APT. But on the other hand, it’s just a shell script. You can easily put it into your $HOME/bin or so. :-)

Useful but Unknown Unix Commandline Options: touch -d //at 20:42 //by abe

You may wonder how I manage to write one “Useful but Unknown Unix Something” blog posting per day. Well, I don’t. I write them in bursts, but don’t want to flood Planet Debian and Planet Symlink with more than one such posting a day.

The first bunch of postings were mostly slides (or part of a slide plus what I tell while showing the slide) from my Unknown but Useful Unix Tools talk and written somewhen last week. And this blog posting and yesterday’s blog posting were written in a row, too.

What does that have to do with Unix commandline tools? Well I use Blosxom as blogging engine and it’s based on simple text files lounging around in some directories, and the last modification time stamp is the posting’s date – which is cached after first time seen by the entries_index plugin so that I can fix typos without resetting the posting’s release date. Postings with a modification date in the future won’t be shown before that time.

So how do you set the last modification date to a date in the future? With “touch” from GNU Coreutils of course. Like “date”, “touch” also knows the option “-d” to explicity set a date and time instead of using the current date and time.

Now in my humble opinion the cool feature is that you can easily describe dates by giving touch values like “now + 1 hour”, “12:00 tomorrow” or so:

$ touch bla
$ ls -l
total 0
-rw-r--r-- 1 abe abe 0 2010-11-23 22:59 bla
$ touch -d 'now + 1 day + 23 hours - 42 minutes' bla
$ ls -l
total 0
-rw-r--r-- 1 abe abe 0 2010-11-25 21:17 bla
$

To see all the possibilities you have to describe relative dates with “touch”, “date”, and maybe other GNU Coreutils tools, check that you have “info” installed and call:

$ info '(coreutils.info.gz)Relative items in date strings'

Oh, and this whole story about setting modification dates manually to arbitrary values also means that the “Tattletale Statistics: Blogtimes November 2010” image in my blog’s side bar is absolute nonsense, at least this month. Well, never trust any statistic you haven’t faked yourself. ;-)

But on the other hand, the dates in the example may give a hint when I really wrote that blog posting. Or not. ;-)

And I wonder what my backup tools think about last modification times in the future. Well, they probably only check for modification dates newer than last backup and therefore should be fine. *phew*

Useful but Unknown Unix Tools: cut //at 15:55 //by abe

Just recently someone asked on #debian.de how to extract the 3rd to 8th character out of some string. That reminded me that the GNU Coreutils tool “cut” is also quite unknown.

The solution to his problem is:

$ echo Hurzlipuh | cut -c3-8
rzlipu

My favourite usage of “cut” is to restrict the output of programs which can’t do that themselves to 80 columns (common terminal width):

$ dpkg -l | grep -i macro | cut -c-80
ii  docbook-to-man                                       1:2.0.0-28             
ii  latex-cjk-common                                     4.8.2+git20090105-5    
ii  m4                                                   1.4.14-3               
ii  mp4h                                                 1.3.1-5                
rc  swfdec-gnome                                         2.28.0-1               

cut can cut out columns based on characters (-c), bytes (-b), fields (-f) delimited by tab (or the delimiter given by -d).

After many comments: Yes, “dpkg -l” is capable of restricting and even fitting its output to the number of available columns if $COLUMNS is passed as environment variable. (And $COLUMNS is set by default in the shell, but not exported as environment variable as it may change on terminal resize or WINCH signal.)

Useful but Unknown Unix Tools: Convert UTF-8 text files to PostScript with paps //at 02:23 //by abe

Sometime you get an UTF-8 encoded text file you want to print. But most text to postscript converters (often invoked automatically by your print server) can only render ISO-Latin-1 text files properly and so you get the notorious ä et al on your printout.

This is especially annoying at PGP/GnuPG keysigning parties where nowadays most people have the names in their UIDs encoded in UTF-8.

Fortunately there is paps (Debian package), a Pango based commandline tool to convert UTF-8 encoded text files into PostScript.

See the paps home page for a neat example.

Update 02:10 (CET): Funnily the explicitly as HTML entities “Ô and “¤” written “ä” above got rendered as “ä” in Liferea, but only in the Planet Debian and Planet Symlink feeds, and only until I put that “ä” in this paragraph, because both got converted to their ISO-Latin-1 8-bit equivalent bytes, so that without the “ä”, “ä”, converted to 8-bit ISO-Latin-1 characters looks also like an UTF-8 “ä”. And Liferea seems to guess the character set somehow and if it validates as UTF-8, it uses UTF-8 even if it isn’t UTF-8. This is a strange Planet.

Useful but Unknown Unix Commandline Options: sort -h //at 01:12 //by abe

The GNU coreutils command “du” knows about the option “-h” to output human readable (or at least human friendly) values with unit prefixes, e.g. k, M or G.

The GNU coreutils command “sort” also can sort by numbers for quite a long time using the option “-n”, but that doesn’t work on the output of “du -h”. So you usually just did one of the following commands, but couldn’t easily combine them:

$ du -h
$ du | sort -n

For approximately a year, GNU sort now knows about another command line option named “-h”. You guessed it probably: “sort -h” can sort human readable values with SI prefixes, e.g.

$ du -h | sort -h | tail -15
34M     ./ttf-mplus-033/debian/ttf-mplus
34M     ./ttf-mplus-033/debian/ttf-mplus/usr
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share/fonts
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share/fonts/truetype
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share/fonts/truetype/ttf-mplus
35M     ./ttf-mplus-034
57M     ./ttf-mplus-029
60M     ./php5-5.2.6/ext
60M     ./ttf-mplus-030
63M     ./ttf-mplus-031
65M     ./ttf-mplus-032
67M     ./ttf-mplus-033
81M     ./php5-5.2.6
1.5G    .
$

You can get this feature already in Debian Unstable (Sid) and Testing (Squeeze, the upcoming stable release), and Ubuntu Maverick and Natty, but not yet in the current Debian Stable release (Lenny) nor in the last Ubuntu LTS release (Lucid Lynx).

Useful but Unknown Unix Tools: friends of du //at 01:12 //by abe

You probably all know “du” from GNU Coreutils. But there’s more than just “du” around in Debian.

du | xdu

First there is xdu, which displays du output piped into it graphically, even if “du” is running on some other box with the output coming over an ssh connection:

$ du | xdu
$ ssh server-without-xlibs du some-directory | xdu

Clicking on any directory except the topmost zooms into that directory. Pressing “n” sorts by size and “s” toggles the display of sizes in numbers.

xdiskusage

xdiskusage seems to be a fork of xdu or at least shares some GUI code with it as it looks and behaves quite similar.

But the GUI looks nicer, more modern and you don’t need to pipe the output of du into it. It collects the disk usage of a given directory itself.

ncdu

And then there is ncdu, the “ncurses du”. (It has nothing to do with the German party “CDU”. You can easily remember that as “Not CDU”. ;-) It shows you the disk usage of files and subdirectories in the given directory already sorted by size, optionally displaying also ascii art bars and/or percentage (by pressing “g” one or more times) for better comparision between file and directory sizes. And all easily and intuitively to navigate in an ncurses text-mode user interface.

ncdu 1.3 ~ Use the arrow keys to navigate, press ? for help
--- /home/abe/debian/webwml ------------------------------------
                             /..
   70.2MB [20.0% ##########] /english
   40.9MB [11.7% #####     ] /french
   35.0MB [10.0% ####      ] /german
   30.2MB [ 8.6% ####      ] /swedish
   26.6MB [ 7.6% ###       ] /japanese
   23.2MB [ 6.6% ###       ] /spanish
   21.5MB [ 6.1% ###       ] /portuguese
   19.8MB [ 5.6% ##        ] /danish
   15.2MB [ 4.3% ##        ] /italian
    9.8MB [ 2.8% #         ] /russian
    8.8MB [ 2.5% #         ] /polish
    6.5MB [ 1.8%           ] /finnish
    5.8MB [ 1.6%           ] /chinese
    4.4MB [ 1.3%           ] /catalan
    4.3MB [ 1.2%           ] /dutch
    3.7MB [ 1.0%           ] /korean
    3.4MB [ 1.0%           ] /ukrainian
    3.0MB [ 0.9%           ] /czech
    2.6MB [ 0.7%           ] /croatian
    2.6MB [ 0.7%           ] /bulgarian
    2.4MB [ 0.7%           ] /norwegian
    1.7MB [ 0.5%           ] /hungarian
    1.6MB [ 0.4%           ] /romanian
    1.2MB [ 0.3%           ] /greek
    1.2MB [ 0.3%           ] /turkish
    1.1MB [ 0.3%           ] /slovak
  576.0kB [ 0.2%           ] /Perl
  556.0kB [ 0.2%           ] /arabic
  428.0kB [ 0.1%           ] /lithuanian
 Total disk usage: 351.1MB  Apparent size: 351.1MB  Items: 70653                                                      

And yes, I uploaded the two screenshots shown above also to screenshots.debian.net as both tools had no screenshots available yet.

Perfect Team: autossh and GNU Screen //at 01:06 //by abe

SSH is definitely one of my most often used tools, not only for system administration at work but also on the road with my netbook, an EeePC 701 running Debian Sid.

On the road, it often happens that I have a flaky WLAN or UMTS connection, so I often have to kill (via <Enter>~.) and reconnect my SSH session due to a changed IP address or so.

First step against problems arising from using SSH over unreliable network connections is of course GNU Screen. Second step is use SSH keys and ssh-agent to not needing to type the password on every reconnect.

But it’s still very annoying to kill the SSH connection and call ssh again manually. For luck there is autossh, a wrapper around SSH which regularily checks via two tunnels connect to each other on the remote side if the connection is still alive, and if not, it kills the ssh and starts a new one with the same parameters (i.e. tunnels, port forwardings, commands to call, etc.).

It’s quite obvious that this is perfect to be combined with screen’s -R and -d options (Reattach if a detached screen is around, else start a new screen; detach a currently attached screen), so I found myself very often typing (or fetching it from the commandline history :-):

autossh -t sym.noone.org 'screen -Rd'

-t is necessary to allocate a terminal device on the remote machine which is not done by default if you directly call a command via ssh.

In comparision to OpenSSH, autossh needs the single quotes, because otherwise it would parse -Rd as options to parse to ssh and bail out. That’s not a real problem, but when you’re used to just type ssh -t sym.noone.org screen -Rd without any quotes, you’ll run into this then and when.

Update, 25-May-2010, 14:55: As Carsten Hey points out, autossh also supports the -- option to declare that all following options and parameters must be passed to ssh itself. (End of Update)

Typing that often and mistyping it then and when cries for an shell alias or an shell function. So I came up with the following shell function:

asc() {
    autossh -x -a -t "$@" 'screen -RdU'
}

I used a function instead of an alias in case of autossh will in future regard all parameters given after the command as part of the command as ssh does.

The additional options -x and -a disable X and SSH Agent forwarding which both don’t work if you reattach to an already running screen.

And if you’re using Zsh as I do, you can even add some more format string magic to set the window title more or less to the expanded alias, eh, function:

function asc() {
    # Set the title to something more obvious, e.g. the expanded
    # alias, eh, function
    print -Pn "\e]0;%n@%m: autossh -t $* 'screen -RdU'\a";
    autossh -x -a -t "$@" 'screen -RdU'
}
compdef asc=ssh

Update, 25-May-2010, 14:59: As Hauke points out in a comment, Zsh users should also declare that asc should have the same tab completion as ssh itself. The example above has been updated accordingly. (End of Update)

In the meantime on the EeePC I use asc on the commandline more often than ssh itself. And I nearly no more type autossh. (The most common exception here is autossh hostname tail -F /path/to/some/logfile.)

Using that function you can also add common ssh options for tunneling, etc. — I use it most often like this:

asc -D 1080 sym.noone.org

This opens a SOCKS proxy on localhost, port 1080 and that way I can surf via the host I’m connecting to by SSH.

There’s one small drawback though: You didn’t expect that I can just invent some new three letter command without a namespace clash, did you? There is a free game called Advanced Strategic Command whose binary (and Debian package) is named asc, too. If you have that game installed, you can always call it using its full path, e.g. /usr/games/asc on Debian.

P.S.: My whole grml based .zshrc is also available via git at git.noone.org as well as on github.

Useful but Unknown Unix Tools: convmv //at 01:05 //by abe

Ogg and MP3 files often contain the interpreter and song title in the file name. Which is (besides quoting blanks, etc.) usually no issue if you have English interpreters or song titles.

But it may become a character set issue, if you have songs with e.g. French, German, Greek, or Russian interpreters or song titles: They may not be in your preferred character set and therefore don’t display properly on your system. Worst case: You just see questions marks.

If you face such problems, convmv (man page, Debian package) is your tool, because convmv converts filenames from one encoding to another:

$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 ????
$ convmv -f latin1 -t utf-8 *
Your Perl version has fleas #37757 #49830 
Starting a dry run without changes...
mv "./??"       "./äöüß"
No changes to your files done. Use --notest to finally rename the files.
$ convmv --notest -f latin1 -t utf-8 *
Your Perl version has fleas #37757 #49830 
mv "./??"       "./äöüß"
Ready!
$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 äöüß
$ 

Of course it can also recursively rename your whole Ogg collection. And it can do it interactively, too:

$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 ????
$ convmv -i --notest -f latin1 -t utf-8 *
Your Perl version has fleas #37757 #49830 
mv "./??"       "./äöüß" (y/n) y

Ready!
$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 äöüß
$ 

So convmv takes away at least one UTF-8 migration fear.

Update 23:03: Nearly forgot, but Mowgli reminded me: There is also convmvfs (Debian package), a FUSE based filesystem which converts file names on access.

Useful but Unknown Unix Tools: WebDAV client for the commandline //at 01:04 //by abe

Ever wondered how to test or connect to a WebDAV server on the commandline? Don’t want to use a bloaty GUI file manager to connect to a WebDAV server?

Then use cadaver (Debian package), a WebDAV client reminiscent of the classic ftp commandline client (plus tab completion :-).

$ cadaver https://webdav.phys.ethz.ch/unixhome/abe/vortraege
Authentication required for D-PHYS on server `webdav.phys.ethz.ch':
Username: abe
Password: 
dav:/unixhome/abe/vortraege/> ls
Listing collection `/unixhome/abe/vortraege/': succeeded.
Coll:   wml                                    0  Mar 13  2007
dav:/unixhome/abe/vortraege/> quit
Connection to `webdav.phys.ethz.ch' closed.

There is also nd which works more like ncftpget/ncftpput/wget/wput, but I didn’t get the current version in Debian to work with our WebDAV server. Last time I used nd was back then at ECOS times more than half a decade ago.

Useful but Unknown Unix Tools: Swiss Army Knife for SMTP //at 01:04 //by abe

Which mail server administrator hasn’t tested his servers using telnet on port 25, speaking SMTP himself? And which of them made a typo and had to retype everything due to missing readline support of telnet? If you want to check SMTP over SSL aka SMTPS, you had to use “openssl s_client -connect yourserver:465”. And what did you use to check TLS support on port 25?

Just forget about those hassles and use swaks (Debian package), the Swiss Army Knife for SMTP. It does all the annoying stuff (using the correct syntax, typing fast enough, SSL, TLS, etc.) for you while still giving you full control over the SMTP session, especially showing you all input and output:

$ swaks -t abe@noone.org -s sym.noone.org -q RCPT
=== Trying sym.noone.org:25...
=== Connected to sym.noone.org.
<-  220 sym.noone.org ESMTP Postfix (Debian/GNU)
 -> EHLO nemo.deuxchevaux.org
<-  250-sym.noone.org
<-  250-PIPELINING
<-  250-SIZE 10240000
<-  250-ENHANCEDSTATUSCODES
<-  250 8BITMIME
 -> MAIL FROM:<abe@nemo.deuxchevaux.org>
<-  250 2.1.0 Ok
 -> RCPT TO:<abe@noone.org>
<** 450 4.2.0 <abe@nemo.deuxchevaux.org>: Sender address rejected: Greylisted, see http://postgrey.schweikert.ch/help/noone.org.html
 -> QUIT
<-  221 2.0.0 Bye
=== Connection closed with remote host.

Since the moment I discovered swaks, I never telnetted to port 25 anymore.

Useful but Unknown Unix Tools //at 01:04 //by abe

Most of my talks are either talks about commandline basics (Commandline Helpers about GNU Coreutils and Findutils usage; Understanding Shell Quoting) or collections of small tips and tricks, e.g. about SSH, History Expansion, or commandline tools. Valentin Haenel called this second type of talks “nugget talks”. I like that term.

My newest talk (German only so far, next occasion at the LUGS meeting in Zurich on 16th of December 2010) is such a nugget talk and is about useful but unknown tools. While preparing that talk I noticed that the amount of neat but less known tools I collected for possible inclusion into that talk would be enough for a whole day or so. I noticed that I’ll never be able to give a talk about all that stuff.

To not letting these tips rot in that file uncommented, I decided that I’ll write (more or less) short blog postings about these tools, similar to and inspired by Myon’s blog postings about Cool Unix Features as well as the no more existing or at least broken Debian Package of the Day and Debian Package a Day.

While Myon mostly posted long-time but still unknown features, my postings will mostly cover useful but unknown packages available in Debian and most debian-derived distributions like grml and Ubuntu. Some postings may also just highlight some features of common tools or packages like e.g. GNU Coreutils.

You can subscribe to these tips via RSS or just view all of them in your web browser at http://noone.org/blog/English/Computer/Debian/CoolTools/.

Multiple Move & Co. //at 00:08 //by abe

nion’s blog made me notice that many people don’t know mmv (multiple move), which approximately works like this:

mmv '*.htm' #1.html
mmv '*.foo.*' #1.#2.bla

Additionally, mmv also can copy, link or even append files when called as mcp, mln or mad respectively with the appropriate command line options.

When I told nion in IRC on #debian.de about mmv, HE pointed me to the Perl script /usr/bin/rename, which is in Debian’s perl package and therefore installed on nearly every Debian system by default. It moves files by applying perl subsitutions to file names:

rename 's/\.htm$/.html/' *.htm
rename 'y/A-Z/a-z/' *

Being curious, if the newly found tool is not only available in Debian, I looked on a SuSE 9.0 box and indeed, I also found there a /usr/bin/rename. But — surprise, surprise — it’s not a Perl script but an ELF binary. And although it does similar things than mmv and Debian’s rename, it is the simplest of the three commands:

rename .htm .html *.htm
rename foo foo00 foo?
rename foo foo0 foo??

Note to my self: Nice add-on for your command line efficiency talk.

autossh vs TCP resetter //at 00:08 //by abe

LUG-Camp 2007 in Interlaken is nearly over, and I’m reading my mail as usual using ssh, screen and mutt on the server. But the ssh connection resets every few minutes. According to the LUSC people (who are running the gateway) some script kiddie is running a TCP resetter somewhere in the network.

I remembered that I read about autossh in the Debian package list once a while and that it sounded cool but I had no use for it yet. Until now.

I’m writing this over the same crashing ssh connection and I’m typing without taking big notice of the quite often occurring connection resets:

autossh noone.org -t 'screen -rd'

It just works. :-)

Group packages by origin in aptitude //at 00:06 //by abe

I always wondered how others recognise non-Debian packages in the aptitude package tree. I also missed the additional priority level in the hierachy well-known from good old dselect.

For the last one, I quickly found out that you can set the priority as subsection — it’s straight forward after you’ve read the documentation: Just add ,priority at the end of the default grouping method for package views under “Options → UI Options” in the aptitude menu.

Getting the origin as given in the Release file of the repository a package originates from is a little bit more difficult. You need to use the pattern() group function with the appropriate search pattern: pattern(~O)

Since already the default default grouping method for package views doesn’t fit into the dialog, I nowadays just edit /etc/apt/apt.conf directly for changes on aptitude’s default grouping method for package views. It now looks like this on several of my machines:

Aptitude::UI {
  Default-Grouping "filter(missing),status,section(subdir,passthrough),pattern(~O),section(topdir),priority";
};

In aptitude this looks like this:

[...]
  --- text - Text processing utilities
  --\ utils - Various system utilities
    --- Debian
    --- Mowgli
    --- volatile.debian.org
  --\ web - Web browsers, servers, proxies, and other tools
    --- Debian
    --- Opera Software ASA
  --\ x11 - The X window system and related software
    --\ Debian
      --- contrib - Programs which depend on software not in Debian
      --\ main - The main Debian archive
        --- Priority optional
        --- Priority extra
      --- non-free - Programs which are not free software
    --- Mowgli
[...]

Unfortunately this doesn’t work with all non-Debian repositories since a few repository maintainer, e.g. those from Emdebian, arrogate to just keep “Debian” as their packages’ origin. This could be solved, if there’s a possibility to group by e.g. repository URL (host and/or path).

Another problem I haven’t solved yet is that grouping by origin does neither work with locally created nor virtual packages nor tasks — probably since all of them lack an origin. Those branches are just empty or don’t even show up anymore with this configuration. I probably have to dig a little bit more in the aptitude documentation to resolve this.

Now playing: E-Rotic — Max don’t have sex with your ex

Useful but Unknown Unix Tools: netselect //at 00:05 //by abe

Ever wondered which mirror of your favourite Linux distribution is the fastest at your location?

Check it with netselect (code at GitHub). It checks for the number of hops and ping times to given hosts and tells you which one is the fastest of them:

# netselect -vv ftp.de.debian.org ftp2.de.debian.org \
                ftp.ch.debian.org ftp.nl.debian.org ftp.debian.org
Running netselect to choose 1 out of 5 addresses.
.......................................................
ftp.de.debian.org                       25 ms  16 hops   90% ok ( 9/10) [   72]
ftp2.de.debian.org                      17 ms  17 hops   90% ok ( 9/10) [   51]
ftp.ch.debian.org                        0 ms   3 hops   90% ok ( 9/10) [    0]
ftp.nl.debian.org                       22 ms  15 hops   90% ok ( 9/10) [   62]
ftp.debian.org                          22 ms  15 hops   90% ok ( 9/10) [   60]
    0 ftp.ch.debian.org

And if you’re too lazy to optimize your sources.list with netselect manually, just use the netselect-apt package. It will do it for you.