Archive for January, 2010

The usage of varnish revisited

Thursday, January 21st, 2010

Varnish is a high-performance HTTP accelerator. Working with Varnish is part of my day job. Among other things, I maintain the packages for Fedora and EPEL.

This is more or less a repost, with updated numbers.

Some months have passed, and it is time to run my poking scripts again, looking for sites that run Varnish. There is no deep magic here. I just parse the available top lists that I know of, and peek at the HTML headers of the sites that are listed. If there are subsites linked from the front page of the site, I scan them too. This means that twitter.com shows up, though Twitter only runs Varnish on its search site. Subsites with a Varnish match are shown in parenthesis in the results.

For the Nordic countries, I have found quite good lists, that is, upload result lists from the probably most visited media sites in the respective countries. Remember of course, that these are generally pay-to-be-included lists, and there may exist sites with far more hits than the ones listed.

For a global overview, I have used Alexa.

Now for the results. Varnish is sponsored by large Norwegian sites, so it is no big surprise that there are a lot of hits in Norway. Of the TNS Gallup top list, Varnish runs at 36 of the top 100 sites. That’s 3 up since my last probe.

For Denmark, I use FDIM‘s list. From May, we are up from 3 to 11 sites in the top 100. For Finland, I use TNS’ numbers again. No changes there. For Sweden, I use the KIA Index list. I now probe the 200 top sites, so there are several more varnish sites on the list. In the top 100, we are up from 8 to 9.

For the Alexa’s World top 500 list, I have tweaked my filters a bit, and the list is up from 7 to 17 sites since my last probe in May. World Domination, here we come!

The whole World: Alexa's global top 500 list
Place  12 Varnish running on twitter.com (http://integratedsearch.twitter.com/search.html)
Place  47 Varnish running on photobucket.com (http://blog.photobucket.com/)
Place  90 Varnish running on mixi.jp (http://img.mixi.jp/static/css/basic/logout_quirks) (and others)
Place 101 Varnish running on weather.com (http://www.weather.com/) (weather.com)
Place 107 Varnish running on globo.com (globo.com) (http://www.globo.com/)
Place 111 Varnish running on ifeng.com (http://big5.ifeng.com/gate/big5/www)
Place 138 Varnish running on answers.com (http://www.answers.com/) (answers.com)
Place 170 Varnish running on orange.fr (http://r.orange.fr/r/Eorangepublicite)
Place 179 Varnish running on hulu.com (hulu.com)
Place 199 Varnish running on wikia.com (wikia.com)
Place 213 Varnish running on xinhuanet.com (http://big5.xinhuanet.com/gate/big5/www)
Place 290 Varnish running on people.com.cn (http://bbs1.people.com.cn/boardList) (and others)
Place 344 Varnish running on tuenti.com (http://estaticos1.tuenti.com/layout/web2/images/favicon)
Place 428 Varnish running on chinanews.com.cn (http://www.chinanews.com.cn/test/index_back_d.html) (and others)
Place 433 Varnish running on mercadolivre.com.br (http://veiculos.mercadolivre.com.br/) (and others)
Place 447 Varnish running on mercadolibre.com.mx (http://tendencias.mercadolibre.com.mx/) (and others)
Place 456 Varnish running on break.com (break.com)

In my last probe, I poked sites all over Europe. With a few exceptions, that was a bit less interesting. Global .com and .net sites tend to cover most of the top 100 entries, as I had only toolbar lists, and it’s not that spectacular that for example people in Serbia and Monte Negro are browsing Twitter, like the rest of the World does. So I have skipped other countries. If you know of any good top list for you country that is not toolbar based, please let me know.

All the gory details are available here.

Other more or less worth mentioned sites that is reported to use Varnish but does not show up in my lists, may be Slashdot, WAT TV, The Pirate Bay, JDownloader, e.Republik, WOWwiki, Globo.com, PCWelt.de, BlackPlanet, funnyordie.com, n-tv.de and 20minutos.es to name a few.

Do you know of other famous sites running Varnish? Use the comments.

Syncing your iPhone media database os 3.x from Fedora 12

Tuesday, January 5th, 2010

So, Apple’s attempt to lock iPhone/iPod users to iTunes failed. Again. A solution for syncing from free Unix like platforms has existed for a few months now. Apple, please. Continuing this cat-and-mouse game is stupid. Stop it. Accessing the phones from other platforms than iTunes is fair usage.

This is how I got syncing my iPhone media database os 3.x from Fedora 12 to work. This is based on marcan’s blog. All kudos to marcan, the libgpod hackers and others who helped to hack the ipodhash. Please read his blog entry. The following concludes step 1-5 on a fedora 12 system.

Update: Got rhythmbox working on one box in addition to gtkpod.
Update: Rebuilt packages for f12, backported from f13. Latest packages here. Exchange “libiphone” with “libimobiledevice” in the following. Since all packages are more or less clean backports from f13, this will probably work out of the box on f13 :-)

In the following, I use sudo to gain root access.

If you have not got libusb installed yet, just install it. You also want gtkpod to do the actual ipod transfer.

sudo yum install libusb gtkpod

Get new versions of libiphone and dependencies from rawhide

sudo yum --enablerepo rawhide install usbmuxd-devel libplist-devel libiphone-devel ifuse

Set up usbmuxd. For security reasons, usbmuxd should run as its own user. (This is no longer necessary in newer packages. The rpm postscript adds the user if it’s not added already.)

sudo useradd -r -c "usbmuxd daemon user" -d /media -s /bin/false usbmuxd

Edit /lib/udev/rules.d/85-usbmuxd.rules as root, and add OWNER=”usbmuxd”.

ACTION=="add", SUBSYSTEM=="usb", OWNER="usbmuxd", ...

You can also add “-v -v” to the RUN string, to get more syslog info. run sudo tail -f /var/log/messages in a separate window to see for example how plugging and unplugging happen.

Force a udev rules reload (should happen automatically, actually).

sudo udevadm control --reload-rules

Edit /etc/fuse.conf as root, and add a line

user_allow_other

Now check that you can mount and umount your ipod via ifuse. Inside, you should see iTunes database files. You will need to do this to get gtkpod to access your phone. Plug the phone, and do

sudo mkdir /media/iphone
sudo chown $(whoami) /media/iphone
ifuse /media/iphone/
ls /media/iphone/iTunes_Control/iTunes/
fusermount -u /media/iphone

Download and install a preview version of libgpod from http://users.linpro.no/ingvar/libgpod.

Now, you need some setup on the phone

ifuse /media/iphone
mkdir /media/iphone/iTunes_Control/Device

Read the uuid of your phone (some 40 characters string), then use ipod-read-sysinfo-extended with that value.

sudo lsusb -v | grep -i iSerial
ipod-read-sysinfo-extended your_uuid_string /media/iphone
ls -sh /media/iphone/iTunes_Control/Device/SysInfoExtended
fusermount -u /media/iphone

/media/iphone/iTunes_Control/Device/SysInfoExtended should now be an 20k+ xml file.

If you have come this far without errors, you can start gtkpod, plug and mount (ifuse) your phone, and sync. Remember to unmount (fusermount -u) the phone before unplugging it.

As my iPhone had not yet been synced with iTunes (and probably never will). I got a HashInfo file from http://ihash.marcansoft.com/, and put it into iTunes_control/Device. Then I cleared out the content of the database (removed Music/*, iTunesCDB, iTunesDB, all .ext files and the .itlp directory). Finally I let gtkpod reinit the database, and hey presto, syncs like a dream. The usbmuxd sync is also a lot faster than the sshfs method I used for the iPhone os v2.x.

As expected, amarok-1.4 is still broken, and can’t use the changed libgpod, and since it has no upstream support, this will probably never be fixed. I’ll miss amarok-1.4, but I can live with gtkpod. Perhaps some day even amarok2 will become usable.

Installing gvfs-afc from rawhide, I got rhythmbox syncing with my i686 box at home, but not with my x86_64 box at work. I haven’t been able to track down why*. With rhythmbox, you don’t have to use ifuse. It will happily pick up the phone via gvfs when it’s plugged. Press Cancel on the camera stuff, and don’t use ifuse and gvfs-afc at the same time. Rhythmbox can’t transfer other content than music, so if you want to transfer video or other stuff, you need to use gtkpod/ifuse.

I’d like feedback from others getting rhythmbox working on x86_64.

Great thanks to marcan and teuf for excellent help on irc.

*)
Update: This is resolved. gvfs does not support home on NFS. It works with no problems with an account with home on local disk on the x86_64 box.

Terry Pratchett: Unseen Academicals

Sunday, January 3rd, 2010

Ankh Morporks eneherskende tyrann Vetinari bestemmer at det eldgamle spillet fotball har kommet for å bli på den magiske Skiveverdenen, men at det er nødvendig at den organiseres litt mer enn dagens rene gatekamper. Oppgaven med å få orden på sakene havner hos rektor Ridcully ved Unseen University, som også må lage sitt eget universitetslag. Som keeper finner vi selvsagt bibliotikaren (en orangutang), og selv om det er forbud mot å bruke magi underveis blir det mange magiske øyeblikk.

Som vanlig på UU er det mye slapstick og moro. Klassiske fotballroller blir parodiert, ikke minst reporteren fra sidelinja. Et bittelite kjærlighetsbilde fra West Side Story kan anes. Og som moralsk sidetema: Alle bør få en sjanse; et spark til Tolkiens ensidige fordømmelse av orkene, som aldri vises noen nåde i hans verden.

Pratchett anbefales som vanlig alltid, selv om dette ikke er hans aller beste bok.