Archive for the ‘sysadmin’ Category

varnish-4.0.3 for Fedora and EPEL

Thursday, March 5th, 2015

varnish-4.0.3 was released recently. I have wrapped packages for Fedora and EPEL, and requested updates for epel7, f21 and f22. They will trickle down as stable updates within some days. I have also built packages for el6, and after som small patching, even for el5. These builds are based on the Fedora package, but should be only cosmetically different from the el6 and el7 packages available from http://varnish-cache.org/.

Also note that Red Hat finally caught up, and imported the necessary selinux-policy changes for Varnish from fedora into el7. With selinux-policy-3.13.1-23.el7, Varnish starts fine in enforcing mode. See RHBA-2015-0458.

My builds for el5 and el6 are available here: http://users.linpro.no/ingvar/varnish/4.0.3/. Note that they need other packages from EPEL to work.

Update 1: I also provide an selinux module for those running varnish-4.0 on el6. It should work for all versions of varnish-4.0, including mine and the ones from varnish-cache.org.

Update 2: Updated builds with a patch for bugzilla ticket 1200034 are pushed for testing in f21, f22 and epel7. el5 and el6 builds are available on link above.

Enjoy.

Ingvar

Varnish Cache is powerful and feature rich front side web cache. It is also very fast, that is, Fast as in on steroids, and powered by The Dark Side of the Force.

Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at redpill-linpro.com.

rpm packages of vmod-ipcast

Thursday, January 8th, 2015

Still on varnish-3.0? Missing the ability to filter X-Forwarded-For through ACLs? Use vmod ipcast by Lasse Karstensen.

I cleaned up and rolled an rpm package of vmod-ipcast-1.2 for varnish-3.0.6 on el6. It’s available here: http://users.linpro.no/ingvar/varnish/vmod-ipcast/.

Note that the usage has changed a bit since the last version. You are now longer permitted to change client.ip (and that’s probably a good thing). Now it’s called like this, returning an IP address object:

ipcast.ip("string","fallback_ip");

If the string does not resemble an IP address, the fallback ip is returned. Note that if the fallback ip is an unvalid address, varnishd will crash!

So, if you want to filter X-Forwarded-For through an ACL, you would something like this:

import ipcast;
sub vcl_recv {
   # Add some code to sanitize X-Forwarded-For above here, so it resembles one single IP address
   if ( ipcast.ip(req.http.X-Forwarded-For, "198.51.100.255") ~ someacl ) {
     # Do something special
   }
}

And that’s all for today.

Varnish Cache is powerful and feature rich front side web cache. It is also very fast, that is, Fast as in on steroids, and powered by The Dark Side of the Force.

Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at redpill-linpro.com.

What is slowing down my ssh login process

Monday, December 22nd, 2014

On one box, I had this strange problem. Every login could take 40-60 seconds, but when first in, everything worked as a charm. As I use ssh for login, I checked the obvious; that reverse DNS lookup did not time out (sshd_config: UseDNS no), and that unnecessary gssapi was not used, but to no avail. So I fetched out old uncle strace from the drawer, and was to run sshd in debug mode, on the console. Then I realized that login on the console took at least as long as via ssh.

So, the problem had to be somewhere else, probably som pam module. strace to the rescue

# strace -e file -ff /usr/sbin/sshd -D -e -ddd -p 2122

and logged in via ssh on port 2122

There it was. An old /var/log/btmp had grown and grown and grown, and login, via pam_lastlog.so (in fedora called in from session), scans it to check for previous logins, using a lot of cpu, io and time in the process.

But why had the file grown so large? Because the btmp log saves failed logins, and this box (by design) had an open ssh to the world, and was often hit by scanners. But also because of missing log rotation. /etc/logrotate.conf on fedora actually rotates /var/log/btmp once a month, but to save space, someone had gzipped the last rotation (again, because of size). And by some strange reasoning (bug?), logrotate on fedora won’t rotate /var/log/btmp at all, if there exists some /var/log/btmp-20140606.gz (unless compress is switched on, which it by default, is not).

Today’s sysadmin tip: ipsort – sort text by ip address

Tuesday, March 25th, 2014

Quite often, I have a list of ip addresses or networks available as an output from a list or a script, and would like to sort them based on the address. The usual unix sort -n does numerical sort, so the list often becomes almost correct

$ (echo 192.168.1.1; echo 192.168.10.2; echo 192.168.2.10) | sort -n 
192.168.10.2
192.168.1.1
192.168.2.10

But grep has a few tricks up its sleeves. Numerical sort by field:

$ alias ipsort="sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4"

$ (echo 192.168.1.1; echo 192.168.10.2; echo 192.168.2.10) | ipsort
192.168.1.1
192.168.2.10
192.168.10.2

Nice, isn’t it?

Today’s sysadmin tip: Latest rpm changelog entry

Thursday, March 13th, 2014

Nothing special today, just a small script that shows (only) the newest changelog entries from the latest installed version of a package, optionally only showing security entries. Especially useful for multiversioned packages, like the kernel.

latest-changelog script may be downloaded here.

Example run:

latest-changelog: Shows the most recent changelog part of the latest installed package given

Usage: latest-changelog [-s] package | [-h]
Options:
  -s | --security : Filter out security information
  -h | --help     : This message

Example: latest-changelog -s kernel

[ingvar@thijs ~]$ latest-changelog -s kernel
- [exec] ptrace: fix get_dumpable() incorrect tests (Petr Oros) [1039486 1039487] {CVE-2013-2929}
- [net] ipv6: fix leaking uninitialized port number of offender sockaddr (Florian Westphal) [1035882 1035883] {CVE-2013-6405}
- [net] inet: fix addr_len/msg->msg_namelen assignment in recv_error functions (Florian Westphal) [1035882 1035883] {CVE-2013-6405}
- [net] inet: prevent leakage of uninitialized memory to user in recv syscalls (Florian Westphal) [1035882 1035883] {CVE-2013-6405}
- [net] ipvs: Add boundary check on ioctl arguments (Denys Vlasenko) [1030817 1030818] {CVE-2013-4588}

Today’s sysadmin tip: Finding what binaries to restart revisited

Monday, January 20th, 2014

Almost exactly two years ago, I posted a perl script to find what binaries to restart for Red Hat based systems. I digs a bit deeper than the excellent needs-restarting script that is provided by Red Hat, by running ldd on the running process binaries, and recursively checking all underlying libraries. I did an extra variant for Debian and derivates today.

Why is this necessary? Because processes may map libraries without opening them. If the underlying library is updated, needs-restarting (or checkrestart on Debian, Ubuntu and derivates) won’t list the process as need to be restarted. But the process may crash or behave strangely when it some time in the future opens a mapped library, and that library has been changed by an update.

And yes, this is a real problem, experienced on production systems.

Red Hat variant
Debian/Ubuntu variant

Today’s sysadmin tip: mpt-raid on SunFire: What drive to change?

Saturday, June 29th, 2013

One of the disks in one of our old RHEL5 SunFire x4150 boxes wouldn’t spin up after a controlled reboot. Oracle Support showed a friendly face, and sendt a new disk after a few hours. Going down to the data center, and finding the right box, both disk lamps blinked green, one of them blinked a bit more than the other, but was that the broken one or the other? Pulling the wrong disk would bring a production system down.

How do I find which disk is the broken one?

The iLO web interfaces shows … nothing interesting about the broken disk at all. Now, these SunFire work horses are usually equipped with LSI SAS1068E Fusion-MPT entry-level raid cards, using the mptscsi driver, so we can poll status with mpt-status or lsiutil. mpt-status says that the broken disk is “phy 1 scsi_id 2″. lsiutil says the broken disk is “Bus 0 Target 2″. The Sun/Oracle docs showing the disk drawers enumerate the disks, but does not indicate SCSI IDs. What to do, what to do? The clock is ticking away, and at home, the dinner is ready.

Finally, the mother of all disk status tools, S.M.A.R.T. to the rescue: The mptscsi driver adds generic scsi devices to all physical devices, as well as the logical raid device. So we can use smartmontools to poll status of each physical device. On a typical system disk with a raid1 mirror, sg0 is the first physical disk, sg1 is the second, and sg2 is the logical lun provided by the mirror. What is so magical with smartmontools? It provides the actual serial number of the disk. And that is visible through the disk drawer front panel.

smartctl -a /dev/sg0
smartctl -a /dev/sg1

The broken disk should report (or fail to report) its status, and may be located by its serial number. Now change the disk and get home before the dinner gets cold.

Btw, smartctl reports other scsi ids on the physical disks than mpt-status and lsiutil did. Go figure.

Updated vagrant packages for fedora

Friday, March 22nd, 2013

Vagrant-1.1.2 was released upstream a few days ago, so I have updated my fedora packages.

A yum repo for Fedora 18 with rubygem-vagrant, and all deps and builddeps is available at http://users.linpro.no/ingvar/vagrant/

Missing deps in Fedora:

  • rubygem-childprocess >=0.3.7 < 0.4.0 (0.3.6 in rawhide)
    Fix: Update package to 0.3.7

  • rubygem-json >= 1.6.6 < 1.7.0 (1.6.8 in f18, 1.7.5 in rawhide)
    Fix: f18 works, but rawhide will be broken

  • rubygem-log4r >= 1.1.9 < 2.0.0
    Fix: Build new package, bz #905240

  • rubygem-net-ssh >= 2.6.6 (2.2.1 in f18)
    Fix: rebuild f18 package from rawhide

  • rubygem-net-scp >= 1.1.0 (1.0.4 in rawhide)
    Fix: Update package to 1.1.0

According to the gemspec, it needs rspec-* ~> 2.11.0, but it seems to build fine with 2.8.0 from f18

rubygem-vagrant was submitted for package review in bz #905396.

Vagrant packages for fedora revisited

Tuesday, January 29th, 2013

I have updated my vagrant packages for fedora with vagrant-1.0.6, and adding a yum repo with packages for fedora 18. The packages missing from fedora are posted for package review.

The yum repo has prebuilt packages for f17 and f18, and is available here: http://users.linpro.no/ingvar/vagrant/

Snipped from my posting to the fedora devel mailing list:

Vagrant offers scripted provisioning and deployment of virtual instances, removing the infamous “but it works om my laptop” obstacle. Vagrant is well-known and much used and praised in the devops community. Its home page is http://vagrantup.com/

Though VirtualBox is the current supported target, future versions of vagrant may support other hypervizors as well, including kvm. Being in itself free software under the MIT license, I think vagrant could be included in fedora.

While an upstream rpm exists (putting all dependent packages in /opt) a native fedora package of vagrant was missing. So I wrapped one up.

Review request: bz #905396

It depends on the following packages missing from fedora 18:

rubygem-log4r >= 1.1.9, < 2.0.0
Fix: Build new package
Package review request: bz #905240

rubygem-childprocess >=0.3.1 < 0.4.0 (0.3.6 in rawhide)
Fix: Grab 0.3.6 package from rawhide

rubygem-json >= 1.5.1, < 1.6.0 (1.6.5 in f18, 1.9.1 in rawhide)
Fix: Build rubygem-json15, roughly based on current package.
Package review request: #bz 905389

rubygem-net-ssh >= 2.2.2, < 2.3.0 (2.2.1 in rawhide)
Fix: Build 2.2.2 package based on current package.
Update request: bz #905393

vagrant packages for fedora

Friday, November 16th, 2012

Vagrant offers scripted provisioning and deployment of virtual instances, removing the well-known “but it works om my laptop” obstacle. Vagrant is well-known and much used and praised in the devops community. Its home page is http://vagrantup.com/

While VirtualBox is the current supported target, future versions of vagrant may support other hypervizors as well, including kvm. Being in itself free software under the MIT license, I think vagrant could be included in fedora.

While an upstream rpm exists (putting all dependent packages in /opt) a native fedora package of vagrant was missing. So I wrapped one up. It depends on the following packages missing from fedora 18:

  • rubygem-log4r >= 1.1.9 < 2.0.0
    Fix: Build new package
  • rubygem-childprocess >=0.3.1 < 0.4.0 (0.2.0 in rawhide)
    Fix: Build 0.3.6 package based on current package
  • rubygem-json >= 1.5.1, < 1.6.0
    Fix: Build rubygem-json15, roughly based on current package
  • rubygem-net-ssh >= 2.2.2 < 2.3.0 (2.2.1 in rawhide)
    Fix: Build 2.2.2 package based on current package
  • Discussion on the packages:

    vagrant needs the log4r gem. It is not included in fedora, so I wrapped that as well. It’s quite trivial, but its license is a bit unclear. LGPLv3 or, uhm, something else?

    vagrant needs the json gem, version 1.5.x. Fedora has 1.6.x, so I built a rubygem-json15 based on the current version. It fails some of the tests, in particular those checking utf8 conversion, so I rudely commented out the check part. Note that ruby has no problems coping with several versions of the same gem installed in parallell.

    The net-ssh and childprocess gems are needed with newer versions than those in f17. I built updated versions based on the ones in rawhide.

    Finaly, to make this work on f17, rubygem-i18n and rubygem-erubis were recompiled from f18.

    f17 packages, source and specfiles can be downloaded here: http://users.linpro.no/ingvar/vagrant/

    Based on the feedback on this post, I’ll consider asking for a proper fedora packaging review.