Archive for the ‘sysadmin’ Category

Five valuable considerations while moving services to the cloud

Friday, August 17th, 2018

Also posted at Redpill Linpro’s Operations and DevOps blog.

Going cloud is the new black, and has been for a few years already. Customers ask us: What is your Cloud Strategy? Can you help us moving to the cloud? Are your services Cloud Compliant? (Is that even a valid term?)

Burning the bridges and moving everything to the cloud may sound compelling, and public and private cloud services may be quite rewarding. No hardware responsibility. Pay for what you actually use, not what you may use. Scale up. Scale down. Scale out. Infinite storage. Multi data center. Multi location. High availability. Global location based load balancing. Functions as a service. Databases as a service. Anything as a service! Why wait?

All players in the IT field should consider cloud technologies, but while planning the setup of a new stack, there are issues to take into consideration. Not all services suits any cloud setup. Here are a few real-life scenarios:

1. Apps in the cloud, cache locally

A media house had reimplemented their production stack at a public cloud provider. The apps worked great, the developers were happy, the performance was satisfactory, and the users content. But the costs went up, as the cloud provider’s network traffic toll was quite high. Using a CDN could be a solution, but was considered too costly and unnecessary, as most users were local to a few central locations.

Keeping the stack in the cloud, and adding local web caches to our data centers ended up being a good solution, keeping high traffic volumes on low cost lines, while sending backend traffic to the cloud.

For high volume sites, consider caching in local data centers

2. Apps locally, cache in the cloud

Another media house had a classic in-house publication system, but had readers around the globe. The volume was quite low, but with high quality content to paying customers worldwide. Users in South-East Asia or the West Coast of USA got high latency and slow content loading. Actually building a scaled-down CDN service, we put cache servers in public cloud provider locations close to the users, and got happy readers. Using the Varnish Plus product, local users got cached content even though protected by a paywall.

For low volume sites, consider local caching in the cloud

3. Legal storage

A media content provider was moving their services and content library to the cloud. They considered using their existing platform for building a library product, delivering storage and search for images and video data, directed at public service usage, and asked for advice. Their platform used Amazon’s AWS S3 for storage and AWS Glacier for backup. Then it occurred that storing data for public services abroad might have legal consequences, and we had to search legal advice on document storage. A proposed solution was to use Ceph based S3 compatible storage services in our data centers within Norway. With this solution, network traffic expenses became a factor.

Storing data abroad may have consequences

4. Anything as a service, cost by call

Skipping the overhead of creating and maintaining virtual machines is tempting. Public cloud providers offer Database as a Service variants compatible with well-known databases, including most SQL and noSQL variants. Include Functions as a Service, and you may be able to build a complete serverless solution including data backend and APIs.

A customer built a system using cloud provided services for database and message queue. It worked flawlessly for development and test, but when adding production traffic to the solution, cost became a large issue, as the services were tolled by request. Biting the bullet, they admitted the work of maintaining virtual machines for some of the services, paying for cpu, memory, network, and man hours.

When using anything as a service, consider traffic driven costs against the overhead of server management.

Also, «serverless» computing is of course a lie. The servers are there, the interface just hides the database setup. In a test using one public cloud provider’s MySQL variant, a multi database slave instance setup was shown to be quite non-resilient against sudden death, with downtime for the service while the slaves were resynced.

There is no such thing as «serverless» computing, just another level of abstraction in front of another computer

5. Trust the cloud provider, the cloud provider is your friend

A well-known story tells how a complete site was taken down by a public cloud provider’s robots looking for suspicious activity. With customer chat down, and no on-call service available, the developers were forced to handle the incident by waking up the CFO, and send credit card information manually to the provider. Read the full story at https://bit.ly/2yWQBnD

Last year, a major part of Amazon’s S3 storage system went down, and was unavailable for hours, making trouble for thousands of sites. Read the details at https://amzn.to/2melOup

Nobody is perfect. Not even public cloud providers. Also small fish are … small, so who are you gonna call?

Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at www.redpill-linpro.com.

Packages of varnish-6.0 with matching vmods, for el6 and el7

Thursday, August 9th, 2018

Some time ago, the Varnish Cache project released a new upstream version 6.0 of Varnish Cache. I updated the fedora rawhide package a few weeks ago. I have also built a copr repo with varnish packages for el6 and el7 based on the fedora package. A selection of matching vmods is also included.

Packages are available at https://copr.fedorainfracloud.org/coprs/ingvar/varnish60/

The following vmods are available:

Included in varnish-modules:
vmod-bodyaccess
vmod-cookie
vmod-header
vmod-saintmode
vmod-tcp
vmod-var
vmod-vsthrottle
vmod-xkey

Packaged separately:
vmod-curl
vmod-digest
vmod-geoip
vmod-memcached
vmod-querystring
vmod-uuid

Please test and report bugs. If there is enough interest, I may consider pushing these to fedora as well.

12 days of Varnish

Tuesday, December 19th, 2017

While Varnish is most famous for its speedy caching capabilities, it is also a general swiss army knife of web serving. In the spirit of Christmas, here’s Twelve Days of Varnish Cache, or at least, twelve Varnish use cases. Read the rest of this post on Redpill Linpro’s Sysadvent calendar.

Dynamic DNS helper scripts

Friday, December 1st, 2017

While dynamic DNS is a wonderful tool for automation and orchestration, tools for easy cleaning up and logging changes are needed. This post describes a couple of scripts that may help.

Read the rest of this post on Redpill Linpro SysAdvent Calendar.

copr packages of varnish-5.2, varnish-modules and miscellaneous vmods for el6 and el7

Friday, October 27th, 2017

Some weeks ago, the Varnish Cache project released a new upstream version 5.2 of varnish cache. I have built a copr repo with varnish packages for el6 and el7 based on the fedora package, and a selection of matching vmods.

The following vmods are available:

Included in varnish-modules:
vmod-cookie
vmod-header
vmod-saintmode
vmod-softpurge
vmod-tcp
vmod-var
vmod-vsthrottle
vmod-xkey

Packaged separately:
vmod-geoip
vmod-basicauth
vmod-curl
vmod-digest
vmod-memcached
vmod-querystring
vmod-rfc6052
vmod-uuid

Please test and report bugs. If there are enough interest, I may consider pushing these to fedora as well. Packages are available at https://copr.fedorainfracloud.org/coprs/ingvar/varnish52/

OCSP: What, why, how?

Thursday, January 12th, 2017

While debugging a problem with OCSP, I had to sit down and understand what it really does and why. So What is OCSP, and why do we use it?

Read the rest of this entry

Deduplication of old filesystems

Sunday, December 18th, 2016

Modern filesystems, and even storage systems, might have built-in deduplication, but common filesystems still do not. So checking for redundant data and do deduplication when possible might save disk space.

Once up on a a time, there was a system, were we had this 6TB spool of binary files on an production ext4 filesystem, and the volume was running out of disk space. The owner of the data thought it likely that there were duplicates in the vast ammount of files, and wanted to check this up. We checked using fdupes, and yes, there were a lot of duplicates.

Read the rest of the post at Redpill Linpro’s sysadvent blog

Bash: Random numbers for fun and profit

Tuesday, December 13th, 2016

bash has many things that just works automagically. Did you know it has a built-in pseudorandom number generator? Let’s play some games! Read rest of the post here!

varnish-5.0, varnish-modules-0.9.2 and hitch-1.4.1, packages for Fedora and EPEL

Thursday, October 20th, 2016

The Varnish Cache project recently released varnish-5.0, and Varnish Software released hitch-1.4.1. I have wrapped packages for Fedora and EPEL.

varnish-5.0 has configuration changes, so the updated package has been pushed to rawhide, but will not replace the ones currently in EPEL nor in Fedora stable. Those who need varnish-5.0 for EPEL may use my COPR repos at https://copr.fedorainfracloud.org/coprs/ingvar/varnish50/. They include the varnish-5.0 and matching varnish-modules packages, and are compatible with EPEL 5, 6, and 7.

hitch-1.4.1 is configure file compatible with earlier releases, so packages for Fedora and EPEL are available in their respective repos, or will be once they trickle down to stable.

As always, feedback is warmly welcome. Please report via Red Hat’s Bugzilla or, while the packages are cooking in testing, Fedora’s Package Update System.

Varnish Cache is a powerful and feature rich front side web cache. It is also very fast, and that is, fast as in powered by The Dark Side of the Force. On steroids. And it is Free Software.

Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at www.redpill-linpro.com.

IPV6: clatd, a component of 464XLAT, for Fedora and EPEL

Friday, September 2nd, 2016

The World is running out of IPv4 addresses, but luckily, we have IPv6 here now, and running the whole data center on IPv6 only is not just happening, it’s becoming the standard. But what if you have an app, a daemon, or a container that actually needs IPv4 connectivity? Then you may use 464XLAT to provide an IPv4 tunnel through your IPv6 only infrastructure. clatd is one component in 464XLAT.

clatd is a CLAT / SIIT-DC Edge Relay implementation for Linux. From the github wash label:

clatd implements the CLAT component of the 464XLAT network architecture specified in RFC 6877. It allows an IPv6-only host to have IPv4 connectivity that is translated to IPv6 before being routed to an upstream PLAT (which is typically a Stateful NAT64 operated by the ISP) and there translated back to IPv4 before being routed to the IPv4 internet. This is especially useful when local applications on the host requires actual IPv4 connectivity or cannot make use of DNS64 (…) clatd may also be used to implement an SIIT-DC Edge Relay as described in RFC 7756.

Note that clatd relies on Tayga for the actual translation of packets between IPv4 and IPv6.

Yesterday, I pushed clatd for fedora testing and epel testing. Please test and report feedback by bugzilla.

For more information on clatd, see the documentation included in the package, or the clatd github home. For more info on Tayga, visit http://www.litech.org/tayga/.

For general information about the process of transisioning to the britght future of IPv6, consider https://en.wikipedia.org/wiki/IPv6_transition_mechanism

Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at www.redpill-linpro.com.