Stoppt die Vorratsdatenspeicherung! Jetzt klicken &handeln! Willst du auch an der Aktion teilnehmen? Hier findest du alle relevanten Infos und Materialien:
Jump to menu and information about this site.

Sunday·06·October·2013

Searching in Screen’s copy mode //at 23:43 //by abe

from the It-would-be-neat-if-that-would-work.-Oh,-it-does-work! dept.

I’m using GNU Screen daily for definitely more than a decade and I became maintainer of Debian’s screen package nearly exactly two years ago. Nevertheless it still happens occassionally that I discover features yet unknown to me. Recently I had one of these moments again:

I looked for a specific line in the long output of a command which has run inside a Screen session. For that I entered Screen’s copy mode with Ctrl-A [ and scrolled around with arrow keys and page-up and -down keys.

But didn’t find it. I thought, it would be cool if I can search for the string I’m looking for. Intuïtively I typed /, the search string and pressed enter. And it worked! It jumped to the next occurrence of that string.

Of course I immediately had to check if tmux has such a feature, too. And it indeed has, but it seems to be a less sophisticated implementation:

Feature Key-binding in GNU Screen Key-binding in Tmux
Switch into copy/scroll mode
(needed for the remainder)
Ctrl-A [ Ctrl-B [
Search for string once, forward / + string + Enter Ctrl-S + string + Enter
Search for string once, backward ? + string + Enter Ctrl-R + string + Enter
Search for string again, forward / Enter Ctrl-S Enter
Search for string again, backward ? Enter Ctrl-R Enter
Incremental search for string, forward Ctrl-S + string -
Incremental search for string, backward Ctrl-R + string -
(Incremental) search for next occurrence, forward Ctrl-S again -
(Incremental) search for next occurrence, backward Ctrl-R again -

Being able to do incremental search like with GNU Emacs gave me yet another reason for continuing to use Screen and not to switch Tmux. ;-)

Wednesday·02·October·2013

How to make wget honour Content-Disposition headers //at 16:12 //by abe

from the DWIM dept.

Download links often point to CGI scripts which actually generate (or just fetch, i.e. proxy) the actual file to be downloaded, e.g. URLs like http://www.example.com/download.cgi?file=foobar.txt.

Most of such CGI scripts send the real file name in the Content-Disposition header as specified in the MIME Specification.

All browsers I know (well, at least those I use regularily :-) handle that perfectly and propose the file name sent in the Content-Disposition header as file name for saving the downloaded name which is usually exactly what I want.

All browsers do that, …, just not my favourite commandline download tool GNU Wget … Downloading the above URL with wget would look like this with default settings:

$ wget 'http://www.example.com/download.cgi?file=foobar.txt'
--2013-10-02 16:04:16--  http://www.example.com/download.cgi?file=foobar
Resolving www.example.com (www.example.com)... 93.184.216.119, 2606:2800:220:6d:26bf:1447:1097:aa7
Connecting to www.switch.ch (www.example.com)|2606:2800:220:6d:26bf:1447:1097:aa7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2020 (2.0K) [text/plain]
Saving to: `download.cgi?file=foobar.txt'

100%[============================================>] 2,020       --.-K/s   in 0s

2013-10-02 16:04:24 (12.5 MB/s) - `download.cgi?file=foobar.txt' saved [2020/2020]

Meh!

But luckily Wget can do that, it’s just not enabled by default — because it’s an experimental and possibly buggy feature, at least according to the man page. Well, works for me! :-)

You can easily enabled it by default for either your user or the whole system by placing the following line in your ~/.wgetrc or /etc/wgetrc:

content-disposition = on

Given the CGI script sends an appropriate Content-Disposition header, the above output now looks like this:

$ wget 'http://www.example.com/download.cgi?file=foobar.txt'
--2013-10-02 16:04:16--  http://www.example.com/download.cgi?file=foobar
Resolving www.example.com (www.example.com)... 93.184.216.119, 2606:2800:220:6d:26bf:1447:1097:aa7
Connecting to www.switch.ch (www.example.com)|2606:2800:220:6d:26bf:1447:1097:aa7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2020 (2.0K) [text/plain]
Saving to: `foobar.txt'

100%[============================================>] 2,020       --.-K/s   in 0s

2013-10-02 16:04:24 (12.5 MB/s) - `foobar.txt' saved [2020/2020]

Now Wget does what I mean!

You can also set this as flag on the commandline, but typing wget --content-disposition … everytime is surely not what I want. ;-)

Sunday·10·March·2013

Rendering Markdown, Asciidoc and Friends automatically while Editing //at 15:41 //by abe

from the also-small-tools-can-make-people-happy dept.

Partially because of Markdown being Github’s markup format of choice, I enjoy writing documents in simple markup formats more and more.

There’s though one common annoyance with these formats compared to writing plain HTML

The Annoyance

They need to be rendered (i.e. more or less compiled) before you can view your outpourings rendered, e.g. in the web browser. So the workflow usually is:

  1. Saving the current file in your favourite editor
  2. Switch to terminal with commandline
  3. Cursor up, Enter
  4. Switch to your favourite web browser
  5. Hit the reload button

Using a Specialized Editor with Live Preview

One choice would be to use a specific editor with live rendering. The one I know in Debian (from Wheezy on) is ReText (Debian package retext). It supports Markdown and reStructuredText.

But as with most simple GUI editors, I miss there many of the advanced editing commands possible with Emacs.

Using Emacs’ Markdown Mode

Then there is the Markdown Mode for Emacs (part of Debian’s emacs-goodies-el package), where you can get a “preview” by pressing C-c C-c p. But for some reason this takes several seconds, opens a new buffer and window with the rendered HTML code and then starts (hardcoded) Firefox (which is not my preferred web browser). And if you do that a second time without closing Firefox first, it won’t just reload the file but will open a new tab. You might think that just hitting reload should suffice. But no, the new tab has a different file name, so reload doesn’t help. Additionally it may not use my preferred Markdown implementation. Meh.

Well, I probably could fix all those issues with Markdown Mode, it’s only Emacs Lisp. Heck, the called command is even configurable. But fixing at least four issues to fix one workflow annoyance? Maybe some other time, but not as long there are other nice choices…

Using inotifywait to Render on Write

So everytime you save the currently edited file, you immediately want to rerender the same HTML file from it. This can be easily automated by using Linux’ inotify kernel subsystem which notices changes to the filesystem, and reports those to applications which ask for it.

One such tool is inotifywait which can either output all or just specific events, or just exit if the first requested event occurs. With the latter it’s easy to write a while loop on the commandline which regenerates a file after every write access. I use either Pandoc or Asciidoc for that since both generate full HTML pages including header and footer, but you can use that also with Markdown to render just the HTML body. Most browsers render it correctly anyway:

while inotifywait -q -e modify index.md; do pandoc -s -f markdown -t html -o index.html index.md; done
while inotifywait -q -e modify index.txt; do asciidoc index.txt; done
while inotifywait -q -e modify index.md; do markdown index.md > index.html; done

This solution is even editor- and build-system-agnostic (But not operating-system-agnostic.)

inotifywait is part of inotify-tools, a useful set of commandline tools to interface with inotify. They’re packaged in Debian as inotify-tools, too.

Using mdpress for Markdown plus Impress.js based Slides

The ruby-written mdpress is a special case of the previous case. It’s a commandline tool to convert Markdown into Impress.js based slide shows and it has an option named --automatic which causes it to keep running and automatically update the presentation as soon as changes are made to the Markdown file.

mdpress is not yet in Debian, but there’s an ITP for it and Impress.js itself recently entered Debian as libjs-impress. Nevertheless, two dependencies (highlight.js, ITP‘ed, ruby-launchy, ITP‘ed) are still missing in Debian.

Wednesday·11·April·2012

Tools for CLI Road Warriors: Remote Shells //at 19:44 //by abe

from the OTR-not-only-means-Off-The-Record dept.

Most of my private online life happens on netbooks and besides the web browser, SSH is my most used program — especially on netbooks. Accordingly I also have hosts on the net to which I connect via SSH. My most used program there is GNU Screen.

So yes, for things like e-mail, IRC, and Jabber I connect to a running screen session on some host with a permanent internet connection. On those hosts there is usually one GNU Screen instance running permanently with either mutt or irssi (which is also my Jabber client via a Bitlbee gateway).

But there are some other less well-known tools which I regard as useful in such a setup. The following two tools can both be seen as SSH for special occassions.

autossh

I already blogged about autossh, even twice, so I’ll just recap the most important features here:

autossh is a wrapper around SSH which regularily checks via two tunnels connect to each other on the remote side if the connection is still alive, and if not, it kills the ssh and starts a new one with the same parameters (i.e. tunnels, port forwardings, commands to call, etc.).

It’s quite obvious that this is perfect to be combined with screen’s -R and -d options.

I use autossh so often that I even adopted its Debian package.

mosh

Since last week there’s a new kid in town^WDebian Unstable: mosh targets the same problems as autossh (unreliable networks, roaming, suspending the computer, etc.) just with a completely different approach which partially even obsoletes the usage of GNU Screen or tmux:

While mosh uses plain SSH for authentication, authorization and key exchange the final connection is an AES-128 encrypted UDP connection on a random port and is independent of the client’s IP address.

This allows mosh to have the following advantages: The connection stays even if you’re switching networks or suspending your netbook. So if you’re just running a single text-mode application you don’t even need GNU Screen or tmux. (You still do if you want the terminal multiplexing feature of GNU Screen or tmux.)

Another nice feature, especially on unreliable WLAN connections or laggy GSM or UMTS connections is mosh’s output prediction based on its input (i.e. what is typed). Per line it tries to guess which server reaction a key press would cause and if it detects a lagging connection, it shows the predicted result underlined until it gets the real result from the server. This eases writing mails in a remote mutt or chatting in a remote irssi, especially if you noticed that you made a typo, but can’t remember how many backspaces you would have to type to fix it.

Mosh needs to be installed on both, client and server, but the server is only activated via SSH, so it has no port open unless a connection is started. And despite that (in Debian) mosh is currently just available in Unstable, the package builds fine on Squeeze, too. There’s also an PPA for Ubuntu and of course you can also get the source code, e.g. as git checkout from GitHub.

mosh is still under heavy development and new features and bug fixes get added nearly every day.

Thanks to Christine Spang for sponsoring and mentoring Keith’s mosh package in Debian.

Update: I gave a lightning talk about Mosh and AutoSSH in German at Easterhegg 2012. The slides are available online.

Wednesday·04·April·2012

Tools for CLI Road Warriors: Hidden Terminals //at 00:57 //by abe

from the Terminals-where-you-don't-expect-them dept.

Some networks have no connection to the outside except that they allow surfing through an HTTP(S) proxy. Sometimes you are happy and the HTTPS port (443) is unrestricted. The following server-side tools allow you to exploit these weaknesses and get you a shell on your server.

sslh

sslh is an SSH/SSL multiplexor. If a client connects to sslh, it checks if the clients speaks the SSH or the SSL protocol and then passes the connection to the according real port of SSL or some SSL enabled service, e.g. an HTTPS, OpenVPN, Tinc or XMPP server. That way it’s possible to connect to one of these services and SSH on the same port.

The usual scenario where this daemon is useful are firewalls which block SSH, force HTTP to go through a proxy, but allow HTTPS connections without restriction. In that case you let sslh listen on the HTTPS port (443) and to move the real HTTPS server (e.g. Apache) to listen on either a different port number (e.g. 442, 444 or 8443) or on another IP address, e.g. on localhost, port 443.

On an Debian or Ubuntu based Apache HTTPS server, you just have to do the following to run Apache on port 442 and sslh on port 443 instead:

  1. apt-get install sslh as root.
  2. Edit /etc/default/sslh, change RUN=no to RUN=yes and --ssl 127.0.0.1:443 to --ssl 127.0.0.1:442.
  3. Edit /etc/apache2/ports.conf and all files in /etc/apache2/sites-available/ which contain a reference to port 443 (which is only /etc/apache2/sites-available/default-ssl.conf in the default configuration) and change all occurrences of 443 to 442.
  4. service apache2 restart
  5. service sslh start

Now you should be able to ssh to your server on port 443 (ssh -p 443 your.server.example.org) while still being able to surf to https://your.server.example.org/.

sslh works as threaded or as preforking daemon, or via inetd. It also honors tcpwrapper configurations for sshd in /etc/hosts.allow and /etc/hosts.deny.

sslh is available as port or package at least in Gentoo, in FreeBSD, in Debian and in Ubuntu.

AjaxTerm

A completely different approach takes AjaxTerm. It provides a terminal inside a web browser with login and ssh being its server-side backend.

Properly safe-guarded by HTTPS plus maybe HTTP based authentication this can be an interesting emergency alternative to the more common — but also more often blocked — remote login mechanisms.

AjaxTerm is available as package at least in Debian and in Ubuntu.

Happily I never were forced to use either of them myself. :-)

Thursday·22·March·2012

Tools for CLI Road Warriors: Tunnels //at 19:49 //by abe

from the I'll-Tunnel-My-Way-Home dept.

Sometime the network you’re connected to is either untrusted (e.g. wireless) or castrated in some way. In both cases you want a tunnel to your trusted home base.

Following I’ll show you three completely different tunneling tools which may helpful while travelling.

sshuttle

sshuttle is a tool somewhere in between of automatic port forward and VPN. It tunnels arbitrary TCP connections and DNS through an SSH tunnel without requiring root access on the remote end of the SSH connection.

So it’s perfect for redirecting most of your traffic through an SSH tunnel to your favourite SSH server, e.g. to ensure your local privacy when you are online via a public, unencrypted WLAN (i.e. easy to sniff for everyone).

It runs on Linux and MacOS X and only needs a Python interpreter on the remote side. Requires root access (usually via sudo) on the client side, though.

It’s currently available at least in Debian Unstable and Testing (Wheezy) as well as in Ubuntu since 11.04 Natty.

Miredo

Miredo is an free and open-source implementation of Microsoft’s NAT-traversing Teredo IPv6 tunneling protocol for at least Linux, FreeBSD, NetBSD and MacOS X.

Miredo includes not only a Teredo client but also a Teredo server implementation. The developer of Miredo also runs a public Miredo server, so you don’t even need to install a server somewhere. If you run Debian or Ubuntu you just need to do apt-get install miredo as root and you have IPv6 connectivity. It’s that easy.

So it’s perfect to get a dynamic IPv6 tunnel for your laptop or mobile phone independently where you are and without the need to register any IPv6 tunnel or configure the Miredo client.

I usually use Miredo on my netbooks to be able to access my boxes at home (which are behind an IPv4 NAT router which is also an SixXS IPv6 tunnel endpoint) from whereever I am.

iodine

iodine is likely the most undermining tool in this set. It tunnels IPv4 over DNS, allowing you to make arbitrary network connections if you are on a network where nothing but DNS requests is allowed (i.e. only DNS packets reach the internet).

This is often the case on wireless LANs with landing page. They redirect all web traffic to the landing page. But the network’s routers try to avoid poisoning the client’s DNS cache with different DNS replies as they would get after the user is logged in. So DNS packets usually pass even the local network’s DNS servers unchanged, just TCP and other UDP packets are redirected until logging in.

With an iodine tunnel, it is possible get a network connection to the outside on such a network anyway. On startup iodine tries to automatically find the best parameters (MTU, request type, etc.) for the current environmenent. However that may fail if any DNS server in between imposes DNS request rate limits.

To be able to start such a tunnel you need to set up an iodine daemon somewhere on the internet. Choose a server which is not already a DNS server.

iodine is available in many distributions, e.g. in Debian and in Ubuntu.

Tuesday·20·March·2012

Happy Birthday GNU Screen! //at 23:46 //by abe

from the State-of-the-Screen dept.

According to this Usenet posting, GNU Screen became 25 years old today. (Found via Fefe.)

And no, it’s not dead. In contrary, the reaction on the mailing list to bug fixes with patches is usually impressingly prompt. :-)

I took this occassion and uploaded a current git snapshot of GNU Screen to Debian Experimental.

Bug #644788 (screen 4.1.0 can’t attach to a running or detached screen 4.0.3 session) is still an issue with that snapshot, but gladly upstream seems to work on a solution for it. There’s even talk about a 4.1.0 beta release soon — although that hasn’t happened yet.

Have fun!

Tag Cloud

2CV, aha, Apache, aptitude, ASUS, Automobiles, autossh, Berlin, bijou, Blogging, Blosxom, Blosxom Plugin, Browser, BSD, CDU, Chemnitz, Citroën, CLI, CLT, Conkeror, CX, deb, Debian, Doofe Parteien, E-Mail, eBay, EeePC, Emacs, Epiphany, Etch, ETH Zürich, Events, Experimental, Firefox, Fläsch, FreeBSD, FVWM, Galeon, Gecko, git, GitHub, GNOME, GNU, GNU Coreutils, GNU Screen, Google, GPL, grep, grml, gzip, Hacks, Hardware, Heise, HTML, identi.ca, IRC, irssi, Jabber, JavaShit, Kazehakase, Lenny, Liferea, Linux, LinuxTag, LUGS, Lynx, maol, Meme, Microsoft, Mozilla, Music, mutt, Myon, München, nemo, Nokia, nuggets, Open Source, Opera, Pentium I, Perl, Planet Debian, Planet Symlink, Quiz, Rant, ratpoison, Religion, RIP, Sarcasm, Sarge, Schweiz, screen, Shell, Sid, Spam, Squeeze, SSH, Stöckchen, SuSE, Symlink, Symlink-Artikel, Tagging, Talk, taz, Text Mode, ThinkPad, Ubuntu, USA, USB, UUUCO, UUUT, VCFe, Ventilator, Vintage, Wahlen, Wheezy, Wikipedia, Windows, WML, Woody, WTF, X, Xen, zsh, Zürich, ÖPNV

Calendar

 2014 →
Months
JulAug Sep Oct Nov Dec
 July →
Mo Tu We Th Fr Sa Su
 
23 24 25 26 27
28 29 30 31      

Tattletale Statistics

Blog postings by posting time
Blog posting times this month



Search


Advanced Search


Categories


Recent Postings

13 most recent of 17 postings total shown.


Recent Comments

Hackergotchi of Axel Beckert

About...

This is the blog or weblog of Axel Stefan Beckert (aka abe or XTaran) who thought, he would never start blogging... (He also once thought, that there is no reason to switch to this new ugly Netscape thing because Mosaïc works fine. That was about 1996.) Well, times change...

He was born 1975 at Villingen-Schwenningen, made his Abitur at Schwäbisch Hall, studied Computer Science with minor Biology at University of Saarland at Saarbrücken (Germany) and now lives in Zürich (Switzerland), working at the IT Support Group (ISG) of the Departement of Physics at ETH Zurich.

Links to internal pages are orange, links to related pages are blue, links to external resources are green and links to Wikipedia articles, Internet Movie Database (IMDb) entries or similar resources are bordeaux. Times are CET respective CEST (which means GMT +0100 respective +0200).


RSS Feeds


Identity Archipelago


Picture Gallery


Button Futility

Valid XHTML Valid CSS
Valid RSS Any Browser
GeoURL
This content is licensed under a Creative Commons License (SA 3.0 DE). Some rights reserved. Hacker Emblem
Get Mozilla Firefox! Powered by Linux!
Typed with GNU Emacs Listed at Tux Mobil
XFN Friendly Button Maker

Blogroll

Blog or not?


People I know personally


Other blogs I like or read


Independent News


Interesting Planets


Web comics I like and read

Stalled Web comics I liked


Blogging Software

Blosxom Plugins I use

Bedside Reading

Just read

  • Bastian Sick: Der Dativ ist dem Genitiv sein Tod (Teile 1-3)
  • Neil Gaiman and Terry Pratchett: Good Omens (borrowed from Ermel)

Currently Reading

  • Douglas R. Hofstadter: Gödel, Escher, Bach
  • Neil Gaiman: Keine Panik (borrowed from Ermel)

Yet to read

  • Neil Stephenson: Cryptonomicon (borrowed from Ermel)

Always a good snack

  • Wolfgang Stoffels: Lokomotivbau und Dampftechnik (borrowed from Ermel)
  • Beverly Cole: Trains — The Early Years (getty images)

Postponed