Posts Tagged ‘bash’

Deduplication of old filesystems

Sunday, December 18th, 2016

Modern filesystems, and even storage systems, might have built-in deduplication, but common filesystems still do not. So checking for redundant data and do deduplication when possible might save disk space.

Once up on a a time, there was a system, were we had this 6TB spool of binary files on an production ext4 filesystem, and the volume was running out of disk space. The owner of the data thought it likely that there were duplicates in the vast ammount of files, and wanted to check this up. We checked using fdupes, and yes, there were a lot of duplicates.

Read the rest of the post at Redpill Linpro’s sysadvent blog

Bash: Random numbers for fun and profit

Tuesday, December 13th, 2016

bash has many things that just works automagically. Did you know it has a built-in pseudorandom number generator? Let’s play some games! Read rest of the post here!

Poor man’s VPN via ssh socks proxy

Sunday, December 13th, 2015

This was also posted on Redpill Linpro’s Sysadvent blog

It is late night. You have just arrived at your Grandparents, when the SMS beeper goes off. There is a problem with a SAN controller, and the on-call person know you fixed it the last time. Now, if you only had documented it.

You know you have to fix this yourself, but you have no VPN access. You don’t even have an Internet connection, except your 3G mobile phone, and you really need access to that admin web gui. There is an emergency ssh port available, but no other port is open. X-forwarding over 3G? Not an option. ssh port-forwarding and fix /etc/hosts. Doable perhaps? VNC over ssh? Awkward. Enter the ssh socks proxy!

Emergency web access

Simply run:

$ ssh -D 1080 login.example.com

Now, you have a local port 1080 that creates a SOCKS proxy to the server side. Firefox has support for that proxy.

Settings -> Advanced -> Network -> Configure how Firefox connects to the Internet -> Manual settings, Socks: localhost, Port: 1080

If you need to resolve addresses from the server side, add that to the config. In the URL field, type about:config , then search for key

network.proxy.socks_remote_dns

Set it to true. That is all. You are now surfing as if Firefox was running locally on the login server. Remember to reset your settings after you have finished your session, or Firefox will not work properly when you close your SOCKS proxy ssh shell.

Not just surfing

But wait, there’s more. With a local SOCKS proxy, you may also use other programs, and they don’t even have to support SOCKS themselves. Install tsocks, and set localhost as the socks proxy host:

$ sudo yum install tsocks || sudo apt-get install tsocks
$ echo "server = 127.0.0.1" | sudo tee /etc/tsocks.conf

tsocks is a little gem of a program. It hooks into other programs, and redirects network traffic to the local SOCKS proxy. Now, while the ssh SOCKS proxy is still running (the ssh -D1080 command), just use tsocks to run your favourite program through the proxy:

# Log into a server on a closed network behind the firewall
$ tsocks ssh server.behind.firewall.example.com
# Run a local psql shell against a remote server through the SOCKS proxy
$ tsocks psql -U pg_admin_user -W -h database.behind.firewall.example.com -W template1

or to run a whole session of commands through the socks proxy, start with “. tsocks on” (note the leading dot), and stop it with “. tsocks off”

$. tsocks on
$ command
$ command
$ command 
$. tsocks off

To run Firefox through the SOCKS proxy, but without changing its configuration:

$ tsocks firefox http://ripe.net   # Stop firefox first

To check tsocks status, run

$ tsocks show

If the LD_PRELOAD variable is empty, tsocks is disabled for this shell.

Note that all Internet traffic is not routed via tsocks. For example, ICMP is not.

Bash process substitution

Saturday, December 12th, 2015

Also posted on Redpill Linpro’s sysadvent blog

In bash, we often use redirects (that is < and > ) to get output from a command to a file, or input from a file to a command. But sometimes, commands takes two or more files as input. Then our ordinary scheme does not work anymore.

Let’s say you want to diff(1) the output of two commands. For example, compare the contents of two directories. You may run the two commands, and redirect the output to files, then diff the files, and finally remove the files. Awkward.

 $ ls dir1 | sort > file1
 $ ls dir2 | sort > file2
 $ diff -u file1 file2
 $ rm file1 file2

Since diff can take stdin as one input via the special filename ‘-‘, we might cut down to one file, but this is still awkward.

 $ ls dir1 | sort > file1
 $ ls dir2 | sort | diff -u file1 -
 $ rm file1

Bash has (of course) a better solution: Process Substition, that is, treat the output (or input) of commands as files. Enter the process substitution operators:

 >(command list) # Input
 <(command list) # Output

Now, let us solve our diff challenge with a simple oneliner:

 $ diff -u <( ls dir1 | sort)  <( ls dir2 | sort )

Neat, isn’t it? I use this all the time!

Bonus: Avoid subshell scripting

The following bash shell loop is a pitfall often missed, leading to subtle bugs that are hard to spot. Pipe to a while loop runs in a subshell, so global variables goes out of scope when they are changed inside the loop.

 #!/bin/bash
 global=0

echo "Outside loop, global=$global"

 for n in 1 2 3; do echo $n; done | \
 while read i; do
     global=$i
     echo "Inside loop: global=$global"
 done
 
 echo "Outside loop, global=$global again :-("

Using command substitution, we avoid this elegantly:

 #!/bin/bash
 global=0
 
 echo "Outside loop, global=$global"
 
 while read i; do
     global=$i
     echo "Inside loop: global=$global"
 done < <( for n in 1 2 3; do echo $n; done )
 
 echo "Outside loop, global=$global still :-)"