Stoppt die Vorratsdatenspeicherung! Jetzt klicken &handeln! Willst du auch an der Aktion teilnehmen? Hier findest du alle relevanten Infos und Materialien:
Jump to menu and information about this site.

Thursday·25·November·2010

Useful but Unknown Unix Tools: cut //at 15:55 //by abe

from the the-first-cut-is-the-deepest dept.

Just recently someone asked on #debian.de how to extract the 3rd to 8th character out of some string. That reminded me that the GNU Coreutils tool “cut” is also quite unknown.

The solution to his problem is:

$ echo Hurzlipuh | cut -c3-8
rzlipu

My favourite usage of “cut” is to restrict the output of programs which can’t do that themselves to 80 columns (common terminal width):

$ dpkg -l | grep -i macro | cut -c-80
ii  docbook-to-man                                       1:2.0.0-28             
ii  latex-cjk-common                                     4.8.2+git20090105-5    
ii  m4                                                   1.4.14-3               
ii  mp4h                                                 1.3.1-5                
rc  swfdec-gnome                                         2.28.0-1               

cut can cut out columns based on characters (-c), bytes (-b), fields (-f) delimited by tab (or the delimiter given by -d).

After many comments: Yes, “dpkg -l” is capable of restricting and even fitting its output to the number of available columns if $COLUMNS is passed as environment variable. (And $COLUMNS is set by default in the shell, but not exported as environment variable as it may change on terminal resize or WINCH signal.)

Wednesday·24·November·2010

Useful but Unknown Unix Tools: Convert UTF-8 text files to PostScript with paps //at 02:23 //by abe

from the Keysinging dept.

Sometime you get an UTF-8 encoded text file you want to print. But most text to postscript converters (often invoked automatically by your print server) can only render ISO-Latin-1 text files properly and so you get the notorious ä et al on your printout.

This is especially annoying at PGP/GnuPG keysigning parties where nowadays most people have the names in their UIDs encoded in UTF-8.

Fortunately there is paps (Debian package), a Pango based commandline tool to convert UTF-8 encoded text files into PostScript.

See the paps home page for a neat example.

Update 02:10 (CET): Funnily the explicitly as HTML entities “Ô and “¤” written “ä” above got rendered as “ä” in Liferea, but only in the Planet Debian and Planet Symlink feeds, and only until I put that “ä” in this paragraph, because both got converted to their ISO-Latin-1 8-bit equivalent bytes, so that without the “ä”, “ä”, converted to 8-bit ISO-Latin-1 characters looks also like an UTF-8 “ä”. And Liferea seems to guess the character set somehow and if it validates as UTF-8, it uses UTF-8 even if it isn’t UTF-8. This is a strange Planet.

Useful but Unknown Unix Commandline Options: sort -h //at 01:12 //by abe

from the human-sortable dept.

The GNU coreutils command “du” knows about the option “-h” to output human readable (or at least human friendly) values with unit prefixes, e.g. k, M or G.

The GNU coreutils command “sort” also can sort by numbers for quite a long time using the option “-n”, but that doesn’t work on the output of “du -h”. So you usually just did one of the following commands, but couldn’t easily combine them:

$ du -h
$ du | sort -n

For approximately a year, GNU sort now knows about another command line option named “-h”. You guessed it probably: “sort -h” can sort human readable values with SI prefixes, e.g.

$ du -h | sort -h | tail -15
34M     ./ttf-mplus-033/debian/ttf-mplus
34M     ./ttf-mplus-033/debian/ttf-mplus/usr
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share/fonts
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share/fonts/truetype
34M     ./ttf-mplus-033/debian/ttf-mplus/usr/share/fonts/truetype/ttf-mplus
35M     ./ttf-mplus-034
57M     ./ttf-mplus-029
60M     ./php5-5.2.6/ext
60M     ./ttf-mplus-030
63M     ./ttf-mplus-031
65M     ./ttf-mplus-032
67M     ./ttf-mplus-033
81M     ./php5-5.2.6
1.5G    .
$

You can get this feature already in Debian Unstable (Sid) and Testing (Squeeze, the upcoming stable release), and Ubuntu Maverick and Natty, but not yet in the current Debian Stable release (Lenny) nor in the last Ubuntu LTS release (Lucid Lynx).

Useful but Unknown Unix Tools: friends of du //at 01:12 //by abe

from the du-on-steroids dept.

You probably all know “du” from GNU Coreutils. But there’s more than just “du” around in Debian.

du | xdu

First there is xdu, which displays du output piped into it graphically, even if “du” is running on some other box with the output coming over an ssh connection:

$ du | xdu
$ ssh server-without-xlibs du some-directory | xdu

Clicking on any directory except the topmost zooms into that directory. Pressing “n” sorts by size and “s” toggles the display of sizes in numbers.

xdiskusage

xdiskusage seems to be a fork of xdu or at least shares some GUI code with it as it looks and behaves quite similar.

But the GUI looks nicer, more modern and you don’t need to pipe the output of du into it. It collects the disk usage of a given directory itself.

ncdu

And then there is ncdu, the “ncurses du”. (It has nothing to do with the German party “CDU”. You can easily remember that as “Not CDU”. ;-) It shows you the disk usage of files and subdirectories in the given directory already sorted by size, optionally displaying also ascii art bars and/or percentage (by pressing “g” one or more times) for better comparision between file and directory sizes. And all easily and intuitively to navigate in an ncurses text-mode user interface.

ncdu 1.3 ~ Use the arrow keys to navigate, press ? for help
--- /home/abe/debian/webwml ------------------------------------
                             /..
   70.2MB [20.0% ##########] /english
   40.9MB [11.7% #####     ] /french
   35.0MB [10.0% ####      ] /german
   30.2MB [ 8.6% ####      ] /swedish
   26.6MB [ 7.6% ###       ] /japanese
   23.2MB [ 6.6% ###       ] /spanish
   21.5MB [ 6.1% ###       ] /portuguese
   19.8MB [ 5.6% ##        ] /danish
   15.2MB [ 4.3% ##        ] /italian
    9.8MB [ 2.8% #         ] /russian
    8.8MB [ 2.5% #         ] /polish
    6.5MB [ 1.8%           ] /finnish
    5.8MB [ 1.6%           ] /chinese
    4.4MB [ 1.3%           ] /catalan
    4.3MB [ 1.2%           ] /dutch
    3.7MB [ 1.0%           ] /korean
    3.4MB [ 1.0%           ] /ukrainian
    3.0MB [ 0.9%           ] /czech
    2.6MB [ 0.7%           ] /croatian
    2.6MB [ 0.7%           ] /bulgarian
    2.4MB [ 0.7%           ] /norwegian
    1.7MB [ 0.5%           ] /hungarian
    1.6MB [ 0.4%           ] /romanian
    1.2MB [ 0.3%           ] /greek
    1.2MB [ 0.3%           ] /turkish
    1.1MB [ 0.3%           ] /slovak
  576.0kB [ 0.2%           ] /Perl
  556.0kB [ 0.2%           ] /arabic
  428.0kB [ 0.1%           ] /lithuanian
 Total disk usage: 351.1MB  Apparent size: 351.1MB  Items: 70653                                                      

And yes, I uploaded the two screenshots shown above also to screenshots.debian.net as both tools had no screenshots available yet.

Useful but Unknown Unix Tools: convmv //at 01:05 //by abe

from the UTF-8-is-good-but-the-migration-to-it-sucks-(less-with-convmv) dept.

Ogg and MP3 files often contain the interpreter and song title in the file name. Which is (besides quoting blanks, etc.) usually no issue if you have English interpreters or song titles.

But it may become a character set issue, if you have songs with e.g. French, German, Greek, or Russian interpreters or song titles: They may not be in your preferred character set and therefore don’t display properly on your system. Worst case: You just see questions marks.

If you face such problems, convmv (man page, Debian package) is your tool, because convmv converts filenames from one encoding to another:

$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 ????
$ convmv -f latin1 -t utf-8 *
Your Perl version has fleas #37757 #49830 
Starting a dry run without changes...
mv "./??"       "./äöüß"
No changes to your files done. Use --notest to finally rename the files.
$ convmv --notest -f latin1 -t utf-8 *
Your Perl version has fleas #37757 #49830 
mv "./??"       "./äöüß"
Ready!
$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 äöüß
$ 

Of course it can also recursively rename your whole Ogg collection. And it can do it interactively, too:

$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 ????
$ convmv -i --notest -f latin1 -t utf-8 *
Your Perl version has fleas #37757 #49830 
mv "./??"       "./äöüß" (y/n) y

Ready!
$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 äöüß
$ 

So convmv takes away at least one UTF-8 migration fear.

Update 23:03: Nearly forgot, but Mowgli reminded me: There is also convmvfs (Debian package), a FUSE based filesystem which converts file names on access.

Useful but Unknown Unix Tools: WebDAV client for the commandline //at 01:04 //by abe

from the cadaver-is-no-cadaver dept.

Ever wondered how to test or connect to a WebDAV server on the commandline? Don’t want to use a bloaty GUI file manager to connect to a WebDAV server?

Then use cadaver (Debian package), a WebDAV client reminiscent of the classic ftp commandline client (plus tab completion :-).

$ cadaver https://webdav.phys.ethz.ch/unixhome/abe/vortraege
Authentication required for D-PHYS on server `webdav.phys.ethz.ch':
Username: abe
Password: 
dav:/unixhome/abe/vortraege/> ls
Listing collection `/unixhome/abe/vortraege/': succeeded.
Coll:   wml                                    0  Mar 13  2007
dav:/unixhome/abe/vortraege/> quit
Connection to `webdav.phys.ethz.ch' closed.

There is also nd which works more like ncftpget/ncftpput/wget/wput, but I didn’t get the current version in Debian to work with our WebDAV server. Last time I used nd was back then at ECOS times more than half a decade ago.

Useful but Unknown Unix Tools: Swiss Army Knife for SMTP //at 01:04 //by abe

from the forget-telnet-on-port-25 dept.

Which mail server administrator hasn’t tested his servers using telnet on port 25, speaking SMTP himself? And which of them made a typo and had to retype everything due to missing readline support of telnet? If you want to check SMTP over SSL aka SMTPS, you had to use “openssl s_client -connect yourserver:465”. And what did you use to check TLS support on port 25?

Just forget about those hassles and use swaks (Debian package), the Swiss Army Knife for SMTP. It does all the annoying stuff (using the correct syntax, typing fast enough, SSL, TLS, etc.) for you while still giving you full control over the SMTP session, especially showing you all input and output:

$ swaks -t abe@noone.org -s sym.noone.org -q RCPT
=== Trying sym.noone.org:25...
=== Connected to sym.noone.org.
<-  220 sym.noone.org ESMTP Postfix (Debian/GNU)
 -> EHLO nemo.deuxchevaux.org
<-  250-sym.noone.org
<-  250-PIPELINING
<-  250-SIZE 10240000
<-  250-ENHANCEDSTATUSCODES
<-  250 8BITMIME
 -> MAIL FROM:<abe@nemo.deuxchevaux.org>
<-  250 2.1.0 Ok
 -> RCPT TO:<abe@noone.org>
<** 450 4.2.0 <abe@nemo.deuxchevaux.org>: Sender address rejected: Greylisted, see http://postgrey.schweikert.ch/help/noone.org.html
 -> QUIT
<-  221 2.0.0 Bye
=== Connection closed with remote host.

Since the moment I discovered swaks, I never telnetted to port 25 anymore.

Tag Cloud

2CV, aha, Apache, APT, aptitude, ASUS, Automobiles, autossh, Berlin, bijou, Blogging, Blosxom, Blosxom Plugin, Browser, BSD, CDU, Chemnitz, Citroën, CLI, CLT, Conkeror, CX, deb, Debian, Doofe Parteien, E-Mail, eBay, EeePC, Emacs, Epiphany, Etch, ETH Zürich, Events, Experimental, Firefox, Fläsch, FreeBSD, FVWM, Galeon, Gecko, git, GitHub, GNOME, GNU, GNU Coreutils, GNU Screen, Google, GPL, grep, grml, gzip, Hackerfunk, Hacks, Hardware, Heise, HTML, identi.ca, IRC, irssi, Jabber, JavaShit, Kazehakase, Lenny, Liferea, Linux, LinuxTag, LUGS, Lynx, maol, Meme, Microsoft, Mozilla, Music, mutt, Myon, München, nemo, Nokia, nuggets, Open Source, Opera, packaging, Pentium I, Perl, Planet Debian, Planet Symlink, Quiz, Rant, ratpoison, Religion, RIP, Sarcasm, Sarge, Schweiz, screen, Shell, Sid, Spam, Squeeze, SSH, Stöckchen, SuSE, Symlink, Symlink-Artikel, Tagging, Talk, taz, Text Mode, ThinkPad, Ubuntu, USA, USB, UUUCO, UUUT, VCFe, Ventilator, Vintage, Wahlen, Wheezy, Wikipedia, Windows, WML, Woody, WTF, X, Xen, zsh, Zürich, ÖPNV

Calendar

← 2017 →
Months
Dec
 December →
Mo Tu We Th Fr Sa Su
       
15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31

Tattletale Statistics

Blog postings by posting time
Blog posting times this month



Search


Advanced Search


Categories


Recent Postings

13 most recent of 30 postings total shown.


Recent Comments

Hackergotchi of Axel Beckert

About...

Debian GNU/Linux is my favourite Linux distribution, being stable, flexible, consistent and having a great community. Although I'm not the biggest bug report writer, I try to contribute by staffing the Debian booth at events, carrying the necessary hardware there or even organising the whole booth.


RSS Feeds


Identity Archipelago


Picture Gallery


Button Futility

Valid XHTML Valid CSS
Valid RSS Any Browser
GeoURL
This content is licensed under a Creative Commons License (SA 3.0 DE). Some rights reserved. Hacker Emblem
Get Mozilla Firefox! Powered by Linux!
Typed with GNU Emacs Listed at Tux Mobil
XFN Friendly Button Maker

Blogroll

Blog or not?


People I know personally


Other blogs I like or read


Independent News


Interesting Planets


Web comics I like and read

Stalled Web comics I liked


Blogging Software

Blosxom Plugins I use

Bedside Reading

Just read

  • Bastian Sick: Der Dativ ist dem Genitiv sein Tod (Teile 1-3)
  • Neil Gaiman and Terry Pratchett: Good Omens (borrowed from Ermel)

Currently Reading

  • Douglas R. Hofstadter: Gödel, Escher, Bach
  • Neil Gaiman: Keine Panik (borrowed from Ermel)

Yet to read

  • Neil Stephenson: Cryptonomicon (borrowed from Ermel)

Always a good snack

  • Wolfgang Stoffels: Lokomotivbau und Dampftechnik (borrowed from Ermel)
  • Beverly Cole: Trains — The Early Years (getty images)

Postponed