Archive for the ‘sysadmin’ Category

Poor man’s VPN via ssh socks proxy

Sunday, December 13th, 2015

This was also posted on Redpill Linpro’s Sysadvent blog

It is late night. You have just arrived at your Grandparents, when the SMS beeper goes off. There is a problem with a SAN controller, and the on-call person know you fixed it the last time. Now, if you only had documented it.

You know you have to fix this yourself, but you have no VPN access. You don’t even have an Internet connection, except your 3G mobile phone, and you really need access to that admin web gui. There is an emergency ssh port available, but no other port is open. X-forwarding over 3G? Not an option. ssh port-forwarding and fix /etc/hosts. Doable perhaps? VNC over ssh? Awkward. Enter the ssh socks proxy!

Emergency web access

Simply run:

$ ssh -D 1080 login.example.com

Now, you have a local port 1080 that creates a SOCKS proxy to the server side. Firefox has support for that proxy.

Settings -> Advanced -> Network -> Configure how Firefox connects to the Internet -> Manual settings, Socks: localhost, Port: 1080

If you need to resolve addresses from the server side, add that to the config. In the URL field, type about:config , then search for key

network.proxy.socks_remote_dns

Set it to true. That is all. You are now surfing as if Firefox was running locally on the login server. Remember to reset your settings after you have finished your session, or Firefox will not work properly when you close your SOCKS proxy ssh shell.

Not just surfing

But wait, there’s more. With a local SOCKS proxy, you may also use other programs, and they don’t even have to support SOCKS themselves. Install tsocks, and set localhost as the socks proxy host:

$ sudo yum install tsocks || sudo apt-get install tsocks
$ echo "server = 127.0.0.1" | sudo tee /etc/tsocks.conf

tsocks is a little gem of a program. It hooks into other programs, and redirects network traffic to the local SOCKS proxy. Now, while the ssh SOCKS proxy is still running (the ssh -D1080 command), just use tsocks to run your favourite program through the proxy:

# Log into a server on a closed network behind the firewall
$ tsocks ssh server.behind.firewall.example.com
# Run a local psql shell against a remote server through the SOCKS proxy
$ tsocks psql -U pg_admin_user -W -h database.behind.firewall.example.com -W template1

or to run a whole session of commands through the socks proxy, start with “. tsocks on” (note the leading dot), and stop it with “. tsocks off”

$. tsocks on
$ command
$ command
$ command 
$. tsocks off

To run Firefox through the SOCKS proxy, but without changing its configuration:

$ tsocks firefox http://ripe.net   # Stop firefox first

To check tsocks status, run

$ tsocks show

If the LD_PRELOAD variable is empty, tsocks is disabled for this shell.

Note that all Internet traffic is not routed via tsocks. For example, ICMP is not.

Bash process substitution

Saturday, December 12th, 2015

Also posted on Redpill Linpro’s sysadvent blog

In bash, we often use redirects (that is < and > ) to get output from a command to a file, or input from a file to a command. But sometimes, commands takes two or more files as input. Then our ordinary scheme does not work anymore.

Let’s say you want to diff(1) the output of two commands. For example, compare the contents of two directories. You may run the two commands, and redirect the output to files, then diff the files, and finally remove the files. Awkward.

 $ ls dir1 | sort > file1
 $ ls dir2 | sort > file2
 $ diff -u file1 file2
 $ rm file1 file2

Since diff can take stdin as one input via the special filename ‘-‘, we might cut down to one file, but this is still awkward.

 $ ls dir1 | sort > file1
 $ ls dir2 | sort | diff -u file1 -
 $ rm file1

Bash has (of course) a better solution: Process Substition, that is, treat the output (or input) of commands as files. Enter the process substitution operators:

 >(command list) # Input
 <(command list) # Output

Now, let us solve our diff challenge with a simple oneliner:

 $ diff -u <( ls dir1 | sort)  <( ls dir2 | sort )

Neat, isn’t it? I use this all the time!

Bonus: Avoid subshell scripting

The following bash shell loop is a pitfall often missed, leading to subtle bugs that are hard to spot. Pipe to a while loop runs in a subshell, so global variables goes out of scope when they are changed inside the loop.

 #!/bin/bash
 global=0

echo "Outside loop, global=$global"

 for n in 1 2 3; do echo $n; done | \
 while read i; do
     global=$i
     echo "Inside loop: global=$global"
 done
 
 echo "Outside loop, global=$global again :-("

Using command substitution, we avoid this elegantly:

 #!/bin/bash
 global=0
 
 echo "Outside loop, global=$global"
 
 while read i; do
     global=$i
     echo "Inside loop: global=$global"
 done < <( for n in 1 2 3; do echo $n; done )
 
 echo "Outside loop, global=$global still :-)"

Varnish-4.1.0 released, packages for fedora and epel

Monday, October 12th, 2015

Varnish-4.1.0 was recently released, and as usual, I have patched and wrapped up packages for fedora and epel. As 4.1.0 is not api/abi compatible with varnish-4.0, packages for stable releases of epel and fedora are not updated. Varnish-4.1.x will be available in a stable Fedora at latest from f24, though the package recompiles fine on anything from el5 to f23 as well.

Prebuilt packages for epel5, epel6, and epel7 are available here: http://users.linpro.no/ingvar/varnish/4.1.0/.

If you are a fedora contributor, please test the f23 package. The package should install directly on el7 and all supported fedoras, including f23. Then report feedback and add karma points. With a little luck, varnish-4.1 will go into fedora 23 before it freezes.

Ingvar

Varnish Cache is a powerful and feature rich front side web cache. It is also very fast, and that is, fast as in powered by The Dark Side of the Force. On steroids. And it is Free Software.

Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at www.redpill-linpro.com.

jemalloc-4.0.x for fedora and epel

Thursday, August 20th, 2015

jemalloc, Jason Evans’ general-purpose scalable concurrent malloc implementation, was recently updated to version 4.0.0. I have wrapped packages for Fedora, and will update rawhide in a few days. If you would like to test the packages already, have a look at http://users.linpro.no/ingvar/jemalloc/4.0.0/.

Update: Jason recently released updates through 4.0.1 to 4.0.3. Packages for 4.0.3 are pushed to rawhide. Builds for epel are available at http://users.linpro.no/ingvar/jemalloc/4.0.3/.

There are a few fedora packages that rely on jemalloc. If you have a chance to help testing, please recompile and test the package against the updated version. You can leave comments here, or send me a mail.

$ sudo repoquery --whatrequires jemalloc |\
  sed 's,\(.*\)-.*-.*,\1,g;' | sort | uniq | tr '\n' ' ' | fold -s; echo

blender blenderplayer bro gridengine gridengine-execd gridengine-qmaster 
gridengine-qmon jemalloc-devel nfs-ganesha nfs-ganesha-ceph nfs-ganesha-gluster 
nfs-ganesha-proxy nfs-ganesha-utils nfs-ganesha-vfs nfs-ganesha-xfs redis 
varnish 

For those that would like to use jemalloc-4.0 on epel, I have built packages for epel 5, 6, and 7 as well. These will not be pushed to the official epel mirrors, as there are api and abi changes that make them binary incompatible with the existing packages in epel.

I have my happy day job at Redpill Linpro in Norway. Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at http://www.redpill-linpro.com, or follow us on social media:

hitch-1.0.0-beta for Fedora and EPEL

Friday, June 26th, 2015

The Varnish project has a new little free software baby arriving soon: Hitch, a scalable TLS proxy. It will also be made available with support by Varnish Software as part of their Varnish Plus product.

A bit of background:

Varnish is a high-performance HTTP accelerator, widely used over the Internet. To use varnish with https, it is often fronted by other general http/proxy servers like nginx or apache, though a more specific proxy-only high-performance tool would be preferable. So they looked at stud.

hitch is a fork of stud. The fork is maintained by the Varnish development team, as stud seems abandoned by its creators, after the project was taken over by Google, with no new commits after 2012.

I wrapped hitch for fedora, epel6 and epel7, and submitted them for Fedora and EPEL. Please test the latest builds and add feedback: https://admin.fedoraproject.org/updates/search/hitch . The default config is for a single instance of hitch.

The package has been reviewed and was recently accepted into Fedora and EPEL (bz #1235305). Update august 2015: Packages are pushed for testing. They will trickle down to stable eventually.

Note that there also exists as a fedora package of the (old) version of stud. If you use stud on fedora and want to test hitch, the two packages may coexist, and should be able to install in parallel.

To test hitch in front of varnish, in front of apache, you may do something like this (tested on el7):

  • Install varnish, httpd and hitch
      sudo yum install httpd varnish
      sudo yum --enablerepo=epel-testing install hitch || sudo yum --enablerepo=updates-testing install hitch
    
  • Start apache
      sudo systemctl start httpd.service
    
  • Edit the varnish config to point to the local httpd, that is, change the default backend definition in /etc/varnish/default.vcl , like this:
      backend default {
        .host = "127.0.0.1";
        .port = "80";
      }
    
  • Start varnish
      sudo systemctl start varnish.service
    
  • Add an ssl certificate to the hitch config. For a dummy certificate,
    the example.com certificate from the hitch source may be used:

      sudo wget -O /etc/pki/tls/private/default.example.com.pem http://users.linpro.no/ingvar/varnish/hitch/default.example.com.pem
    
  • Edit /etc/hitch/hitch.conf. Change the pem-file option to use that cert
      pem-file = "/etc/pki/tls/private/default.example.com.pem"
    
  • Start hitch
      sudo systemctl start hitch.service
    
  • Open your local firewall if necessary, by something like this:
      sudo firewall-cmd --zone=public --add-port=8443/tcp
    
  • Point your web browser to https://localhost:8443/ . You should be greeted with a warning about a non-official certificate. Past that, you will get the apache frontpage through varnish and hitch.

    Enjoy, and let me hear about any interesting test results.

    Ingvar

    Varnish Cache is powerful and feature rich front side web cache. It is also very fast, that is, Fast as in on steroids, and powered by The Dark Side of the Force.

    Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at www.redpill-linpro.com.

  • varnish-4.0.3 for Fedora and EPEL

    Thursday, March 5th, 2015

    varnish-4.0.3 was released recently. I have wrapped packages for Fedora and EPEL, and requested updates for epel7, f21 and f22. They will trickle down as stable updates within some days. I have also built packages for el6, and after som small patching, even for el5. These builds are based on the Fedora package, but should be only cosmetically different from the el6 and el7 packages available from http://varnish-cache.org/.

    Also note that Red Hat finally caught up, and imported the necessary selinux-policy changes for Varnish from fedora into el7. With selinux-policy-3.13.1-23.el7, Varnish starts fine in enforcing mode. See RHBA-2015-0458.

    My builds for el5 and el6 are available here: http://users.linpro.no/ingvar/varnish/4.0.3/. Note that they need other packages from EPEL to work.

    Update 1: I also provide an selinux module for those running varnish-4.0 on el6. It should work for all versions of varnish-4.0, including mine and the ones from varnish-cache.org.

    Update 2: Updated builds with a patch for bugzilla ticket 1200034 are pushed for testing in f21, f22 and epel7. el5 and el6 builds are available on link above.

    Enjoy.

    Ingvar

    Varnish Cache is powerful and feature rich front side web cache. It is also very fast, that is, Fast as in on steroids, and powered by The Dark Side of the Force.

    Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at redpill-linpro.com.

    rpm packages of vmod-ipcast

    Thursday, January 8th, 2015

    Still on varnish-3.0? Missing the ability to filter X-Forwarded-For through ACLs? Use vmod ipcast by Lasse Karstensen.

    I cleaned up and rolled an rpm package of vmod-ipcast-1.2 for varnish-3.0.6 on el6. It’s available here: http://users.linpro.no/ingvar/varnish/vmod-ipcast/.

    Note that the usage has changed a bit since the last version. You are now longer permitted to change client.ip (and that’s probably a good thing). Now it’s called like this, returning an IP address object:

    ipcast.ip("string","fallback_ip");

    If the string does not resemble an IP address, the fallback ip is returned. Note that if the fallback ip is an unvalid address, varnishd will crash!

    So, if you want to filter X-Forwarded-For through an ACL, you would something like this:

    import ipcast;
    sub vcl_recv {
       # Add some code to sanitize X-Forwarded-For above here, so it resembles one single IP address
       if ( ipcast.ip(req.http.X-Forwarded-For, "198.51.100.255") ~ someacl ) {
         # Do something special
       }
    }

    And that’s all for today.

    Varnish Cache is powerful and feature rich front side web cache. It is also very fast, that is, Fast as in on steroids, and powered by The Dark Side of the Force.

    Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at redpill-linpro.com.

    What is slowing down my ssh login process

    Monday, December 22nd, 2014

    On one box, I had this strange problem. Every login could take 40-60 seconds, but when first in, everything worked as a charm. As I use ssh for login, I checked the obvious; that reverse DNS lookup did not time out (sshd_config: UseDNS no), and that unnecessary gssapi was not used, but to no avail. So I fetched out old uncle strace from the drawer, and was to run sshd in debug mode, on the console. Then I realized that login on the console took at least as long as via ssh.

    So, the problem had to be somewhere else, probably som pam module. strace to the rescue

    # strace -e file -ff /usr/sbin/sshd -D -e -ddd -p 2122
    

    and logged in via ssh on port 2122

    There it was. An old /var/log/btmp had grown and grown and grown, and login, via pam_lastlog.so (in fedora called in from session), scans it to check for previous logins, using a lot of cpu, io and time in the process.

    But why had the file grown so large? Because the btmp log saves failed logins, and this box (by design) had an open ssh to the world, and was often hit by scanners. But also because of missing log rotation. /etc/logrotate.conf on fedora actually rotates /var/log/btmp once a month, but to save space, someone had gzipped the last rotation (again, because of size). And by some strange reasoning (bug?), logrotate on fedora won’t rotate /var/log/btmp at all, if there exists some /var/log/btmp-20140606.gz (unless compress is switched on, which it by default, is not).

    Today’s sysadmin tip: ipsort – sort text by ip address

    Tuesday, March 25th, 2014

    Quite often, I have a list of ip addresses or networks available as an output from a list or a script, and would like to sort them based on the address. The usual unix sort -n does numerical sort, so the list often becomes almost correct

    $ (echo 192.168.1.1; echo 192.168.10.2; echo 192.168.2.10) | sort -n 
    192.168.10.2
    192.168.1.1
    192.168.2.10
    

    But grep has a few tricks up its sleeves. Numerical sort by field:

    $ alias ipsort="sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4"
    
    $ (echo 192.168.1.1; echo 192.168.10.2; echo 192.168.2.10) | ipsort
    192.168.1.1
    192.168.2.10
    192.168.10.2
    

    Nice, isn’t it?

    Today’s sysadmin tip: Latest rpm changelog entry

    Thursday, March 13th, 2014

    Nothing special today, just a small script that shows (only) the newest changelog entries from the latest installed version of a package, optionally only showing security entries. Especially useful for multiversioned packages, like the kernel.

    latest-changelog script may be downloaded here.

    Example run:

    latest-changelog: Shows the most recent changelog part of the latest installed package given
    
    Usage: latest-changelog [-s] package | [-h]
    Options:
      -s | --security : Filter out security information
      -h | --help     : This message
    
    Example: latest-changelog -s kernel
    
    [ingvar@thijs ~]$ latest-changelog -s kernel
    - [exec] ptrace: fix get_dumpable() incorrect tests (Petr Oros) [1039486 1039487] {CVE-2013-2929}
    - [net] ipv6: fix leaking uninitialized port number of offender sockaddr (Florian Westphal) [1035882 1035883] {CVE-2013-6405}
    - [net] inet: fix addr_len/msg->msg_namelen assignment in recv_error functions (Florian Westphal) [1035882 1035883] {CVE-2013-6405}
    - [net] inet: prevent leakage of uninitialized memory to user in recv syscalls (Florian Westphal) [1035882 1035883] {CVE-2013-6405}
    - [net] ipvs: Add boundary check on ioctl arguments (Denys Vlasenko) [1030817 1030818] {CVE-2013-4588}