Useful but Unknown Unix Commandline Options of cat //at 14:04 //by abe

Yesterday in Harald König’s commandline talk at LinuxDay.at, I first heard of cat’s -e option, and noticed that cat has quite some interesting options. From the man page of cat:

-A, --show-all

equivalent to -vET

-b, --number-nonblank

number nonempty output lines

-e

equivalent to -vE

-E, --show-ends

display $ at end of each line

-n, --number

number all output lines

-s, --squeeze-blank

suppress repeated empty output lines

-t

equivalent to -vT

-T, --show-tabs

display TAB characters as ^I

-v, --show-nonprinting

use ^ and M- notation, except for LFD and TAB

Some examples made on an UTF-8 terminal:

user@host:demo $ cat bla
ä  ö  ü ß   


abc
user@host:demo $ cat -Ab bla
     1  M-CM-$  M-CM-6  M-CM-<^IM-CM-^_   $
$
$
     2  abc$
user@host:demo $ cat -Ens bla
     1  ä  ö  ü ß   $
     2	$
     3  abc$
user@host:demo $ 

I hope this blog posting will reduce the percentage of “useless uses of cat” by raising the amount of “useful uses of cat”. :-)

Useful but Unknown Unix Commandline Options: touch -d //at 20:42 //by abe

You may wonder how I manage to write one “Useful but Unknown Unix Something” blog posting per day. Well, I don’t. I write them in bursts, but don’t want to flood Planet Debian and Planet Symlink with more than one such posting a day.

The first bunch of postings were mostly slides (or part of a slide plus what I tell while showing the slide) from my Unknown but Useful Unix Tools talk and written somewhen last week. And this blog posting and yesterday’s blog posting were written in a row, too.

What does that have to do with Unix commandline tools? Well I use Blosxom as blogging engine and it’s based on simple text files lounging around in some directories, and the last modification time stamp is the posting’s date – which is cached after first time seen by the entries_index plugin so that I can fix typos without resetting the posting’s release date. Postings with a modification date in the future won’t be shown before that time.

So how do you set the last modification date to a date in the future? With “touch” from GNU Coreutils of course. Like “date”, “touch” also knows the option “-d” to explicity set a date and time instead of using the current date and time.

Now in my humble opinion the cool feature is that you can easily describe dates by giving touch values like “now + 1 hour”, “12:00 tomorrow” or so:

$ touch bla
$ ls -l
total 0
-rw-r--r-- 1 abe abe 0 2010-11-23 22:59 bla
$ touch -d 'now + 1 day + 23 hours - 42 minutes' bla
$ ls -l
total 0
-rw-r--r-- 1 abe abe 0 2010-11-25 21:17 bla
$

To see all the possibilities you have to describe relative dates with “touch”, “date”, and maybe other GNU Coreutils tools, check that you have “info” installed and call:

$ info '(coreutils.info.gz)Relative items in date strings'

Oh, and this whole story about setting modification dates manually to arbitrary values also means that the “Tattletale Statistics: Blogtimes November 2010” image in my blog’s side bar is absolute nonsense, at least this month. Well, never trust any statistic you haven’t faked yourself. ;-)

But on the other hand, the dates in the example may give a hint when I really wrote that blog posting. Or not. ;-)

And I wonder what my backup tools think about last modification times in the future. Well, they probably only check for modification dates newer than last backup and therefore should be fine. *phew*

Useful but Unknown Unix Tools: cut //at 15:55 //by abe

Just recently someone asked on #debian.de how to extract the 3rd to 8th character out of some string. That reminded me that the GNU Coreutils tool “cut” is also quite unknown.

The solution to his problem is:

$ echo Hurzlipuh | cut -c3-8
rzlipu

My favourite usage of “cut” is to restrict the output of programs which can’t do that themselves to 80 columns (common terminal width):

$ dpkg -l | grep -i macro | cut -c-80
ii  docbook-to-man                                       1:2.0.0-28             
ii  latex-cjk-common                                     4.8.2+git20090105-5    
ii  m4                                                   1.4.14-3               
ii  mp4h                                                 1.3.1-5                
rc  swfdec-gnome                                         2.28.0-1               

cut can cut out columns based on characters (-c), bytes (-b), fields (-f) delimited by tab (or the delimiter given by -d).

After many comments: Yes, “dpkg -l” is capable of restricting and even fitting its output to the number of available columns if $COLUMNS is passed as environment variable. (And $COLUMNS is set by default in the shell, but not exported as environment variable as it may change on terminal resize or WINCH signal.)

Useful but Unknown Unix Commandline Options: sort -h //at 01:12 //by abe

The GNU coreutils command “du” knows about the option “-h” to output human readable (or at least human friendly) values with unit prefixes, e.g. k, M or G.

The GNU coreutils command “sort” also can sort by numbers for quite a long time using the option “-n”, but that doesn’t work on the output of “du -h”. So you usually just did one of the following commands, but couldn’t easily combine them:

$ du -h
$ du | sort -n

For approximately a year, GNU sort now knows about another command line option named “-h”. You guessed it probably: “sort -h” can sort human readable values with SI prefixes, e.g.

$ du -h | sort -h | tail -15
34M     ./ttf-mplus-033/debian/ttf-mplus
34M     ./ttf-mplus-033/debian/ttf-mplus/usr
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share/fonts
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share/fonts/truetype
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share/fonts/truetype/ttf-mplus
35M     ./ttf-mplus-034
57M     ./ttf-mplus-029
60M     ./php5-5.2.6/ext
60M     ./ttf-mplus-030
63M     ./ttf-mplus-031
65M     ./ttf-mplus-032
67M     ./ttf-mplus-033
81M     ./php5-5.2.6
1.5G    .
$

You can get this feature already in Debian Unstable (Sid) and Testing (Squeeze, the upcoming stable release), and Ubuntu Maverick and Natty, but not yet in the current Debian Stable release (Lenny) nor in the last Ubuntu LTS release (Lucid Lynx).

Useful but Unknown Unix Tools: friends of du //at 01:12 //by abe

You probably all know “du” from GNU Coreutils. But there’s more than just “du” around in Debian.

du | xdu

First there is xdu, which displays du output piped into it graphically, even if “du” is running on some other box with the output coming over an ssh connection:

$ du | xdu
$ ssh server-without-xlibs du some-directory | xdu

Clicking on any directory except the topmost zooms into that directory. Pressing “n” sorts by size and “s” toggles the display of sizes in numbers.

xdiskusage

xdiskusage seems to be a fork of xdu or at least shares some GUI code with it as it looks and behaves quite similar.

But the GUI looks nicer, more modern and you don’t need to pipe the output of du into it. It collects the disk usage of a given directory itself.

ncdu

And then there is ncdu, the “ncurses du”. (It has nothing to do with the German party “CDU”. You can easily remember that as “Not CDU”. ;-) It shows you the disk usage of files and subdirectories in the given directory already sorted by size, optionally displaying also ascii art bars and/or percentage (by pressing “g” one or more times) for better comparision between file and directory sizes. And all easily and intuitively to navigate in an ncurses text-mode user interface.

ncdu 1.3 ~ Use the arrow keys to navigate, press ? for help
--- /home/abe/debian/webwml ------------------------------------
                             /..
   70.2MB [20.0% ##########] /english
   40.9MB [11.7% #####     ] /french
   35.0MB [10.0% ####      ] /german
   30.2MB [ 8.6% ####      ] /swedish
   26.6MB [ 7.6% ###       ] /japanese
   23.2MB [ 6.6% ###       ] /spanish
   21.5MB [ 6.1% ###       ] /portuguese
   19.8MB [ 5.6% ##        ] /danish
   15.2MB [ 4.3% ##        ] /italian
    9.8MB [ 2.8% #         ] /russian
    8.8MB [ 2.5% #         ] /polish
    6.5MB [ 1.8%           ] /finnish
    5.8MB [ 1.6%           ] /chinese
    4.4MB [ 1.3%           ] /catalan
    4.3MB [ 1.2%           ] /dutch
    3.7MB [ 1.0%           ] /korean
    3.4MB [ 1.0%           ] /ukrainian
    3.0MB [ 0.9%           ] /czech
    2.6MB [ 0.7%           ] /croatian
    2.6MB [ 0.7%           ] /bulgarian
    2.4MB [ 0.7%           ] /norwegian
    1.7MB [ 0.5%           ] /hungarian
    1.6MB [ 0.4%           ] /romanian
    1.2MB [ 0.3%           ] /greek
    1.2MB [ 0.3%           ] /turkish
    1.1MB [ 0.3%           ] /slovak
  576.0kB [ 0.2%           ] /Perl
  556.0kB [ 0.2%           ] /arabic
  428.0kB [ 0.1%           ] /lithuanian
 Total disk usage: 351.1MB  Apparent size: 351.1MB  Items: 70653                                                      

And yes, I uploaded the two screenshots shown above also to screenshots.debian.net as both tools had no screenshots available yet.

The Software Museum inside the Software Museum //at 13:40 //by abe

Most Linuxers know that Debian and most of its users prefer stable software over up-to-date software. So do I, but sometimes this goes a little bit too far, e.g. when I find software which has been compiled years before the first line of Linux kernel code has been written:

C:\>ls +version
GNU ls, Version 1.4.0.2 (compiled Sep 19 1990 12:43:10 for MS-DOS)

C:\>ls -alF gnu\
total 521
drwxrwxrwx   1 anonymou anonymou     4096 Mar 26 23:16 ./
drwxrwxrwx   1 anonymou anonymou     4096 Mar 26 23:17 ../
-rwxrwxrwx   1 anonymou anonymou    17868 Sep 19  1990 cat.exe*
-rwxrwxrwx   1 anonymou anonymou    20028 Sep 19  1990 cmp.exe*
-rwxrwxrwx   1 anonymou anonymou    26780 Sep 19  1990 cp.exe*
-rwxrwxrwx   1 anonymou anonymou    17948 Sep 19  1990 cut.exe*
-rwxrwxrwx   1 anonymou anonymou    27138 Sep 24  1990 grep.exe*
-rwxrwxrwx   1 anonymou anonymou    16572 Sep 19  1990 head.exe*
-rwxrwxrwx   1 anonymou anonymou    27756 Sep 19  1990 ls.exe*
-rwxrwxrwx   1 anonymou anonymou    23100 Sep 19  1990 mv.exe*
-rwxrwxrwx   1 anonymou anonymou    19820 Sep 23  1990 rm.exe*
-rwxrwxrwx   1 anonymou anonymou    37644 Sep 19  1990 tac.exe*
-rwxrwxrwx   1 anonymou anonymou    20188 Sep 19  1990 tail.exe*

C:\>

And yes, this looks like DOS. This is FreeDOS (1:0.0.b9r5a-3) inside of dosemu, packaged for Debian 4.0 Etch and installed from the original Debian archives.

BTW, the date looks quite authentic: According to the ChangeLog, Version 1.4 of the GNU Fileutils have been released on the 9th of September 1990. The oldest version of the GNU Fileutils (nowadays coreutils) available on the GNU FTP server is version 3.13 from July 1996, though.

I really wonder how many buffer overflows this version has. And I wonder if there’s really a scenario in which this combination (Debian → dosemu → FreeDOS → GNU fileutils) could be exploited.