Archive for the ‘sysadmin’ Category

Free Software and Open Source: Get involved

Thursday, February 18th, 2021

Contributing to Free Software using Open Source methodics may look like intimidating deep expert work. But it doesn’t have to be that. Most Free Software communities are friendly to newcomers, and welcome all kind of contributions.

Reporting bugs

Hitting a bug is an opportunity, not a nasty problem. When you hit a bug, it should be reported, and with a bit of luck, it may even be fixed. Reporting the bug in an open forum also makes other users find the bug, give attention to it, and they may in turn be able to help out working around or fixing it. Reporting bugs is the most basic, but still of the most valuable contributions you may do. Finding bugs are finding real problems. Reporting bugs are helping fixing them, for you, and for other users. You may not complain to your coworker on a bug unless it is reported upstream.

While reporting bugs, remember to collect as much information as possible on the issue, including logs, runtime envionment, hardware, operating system version, etc. While collecting this information, make sure you don’t send any traceable private information that may be used by rouge parties, like ip adresses, hostnames, passwords, customer details, database names, etc.

Bugs in operating system packages

Bugs in components delivered by a Linux distribution (Ubuntu, Debian, Fedora, Red Hat, SuSE, etc), should be reported through their bug reporting interface. Remember to search for the bug before posting yet another duplicate bug. Perhaps a workaround already exists.

So the next time something strange happens to your haproxy, nginx, varnish, or your firefox browser crashes or has unexpected behaviour, collect data from your logs, and open a bug report.

  • Red Hat / EPEL / Fedora users should report bugs through https://bugzilla.redhat.com/
  • Similarly, OpenSuSE users may search for and report bugs at https://bugzilla.opensuse.org
  • Ubuntu users may have luck looking at https://help.ubuntu.com/community/ReportingBugs
  • As Ubuntu’s upstream is Debian, you may search for bugs, fixes and workarounds using their tools at https://www.debian.org/Bugs/Reporting

    These tools have detailed guidelines on the details on how to search, report, and follow up the bugs.

    For an example of an end user bug report with an impressive follow up from a dedicated package maintainer, have a look at https://bugzilla.redhat.com/show_bug.cgi?id=1914917

    Reporting upstream bugs

    Using software directly from the upstream project is growing more usual, specially as container technology has matured, enabling developers to use software components without interfering with the underlying operating system. Reporting and follow up bugs becomes even more important, as such components may not be filtered and quality assured by operating system security teams.

    Find your component’s upstream home page or project development page, usually on Github, Savannah, Gitlab, or similar code repo service. These services have specialised issue trackers made for reporting and following up bugs and other issues. Some projects only has good old mailing lists. They may require you to subscribe to the list before you are allowed to report anything.

    Following up the report, you may be asked for test cases and debugging. You will learn a lot in the process. Do not be shy to ask for help, or admitting that you don’t understand or need guidance. Everybody started somewhere. Even you may learn to use the GNU debugger (gdb) in time.

    Non code commits

    Similarly to reporting bugs, non code commits may be low-hanging fruit to you, but may be crucial to a project’s success. If you can write technical documentation, howtos, or do translations to your native language, such contributions to Free Software are extremely welcome. Even trivial stuff like fixing typos in a translated piece of software should be reported. No fix is too small. I once did a single word commit to GPG: A single word typo fix in their Norwegian translation. Also, write blog posts. Don’t have a blog yet? Get one. Free blog platforms are thirteen to a dozen.

    Use source code tools

    Admit it: You already use git in your day job. Using it for documentation or translation should be trivial. If you have not done so already, learn how to clone a project on github (just google it), grep through the source for what you like to fix or add, make a branch with your contribution, and ask for a pull request (again, just google it). If you changes are not merged at once, be patient, ask for the maintainer’s advice, and listen to their guidelines. Be proud of your contribution, but humble in your request.

    Feature requests

    Usage of a piece of software is not given from the start. Perhaps you have ideas to how a piece of code may be used in some other way, or there is some piece missing that is obvious to you, though not reported in the project’s future roadmap. Don’t be shy to ask. Report a feature request. Usually this is done the same way as reporting a bug. The worst you can get is that they are not interested, or a request for you to produce the missing code. Which you may do.

    Join a project

    If your work require it, and/or your interests and free time to spend allows for it, join a Free Software project.

    Distribution work

    Upstream distributions like Fedora, Debian, and OpenSuse (not to mention Arch and Gentoo) are always looking for volunteers, and have sub projects for packagers, documentation, translation, and even marketing. As long time players in the field, they have great documentation for getting started. Remember to be patient, ask for advice, follow guidelines. Be proud of your contributions, but humble in your requests.

    Upstream projects

    If you want to join a project, show your interest. Join the project’s social and technical forums. Subscribe to their development email lists. Join their IRC channels. Lurk for a while, absorbing the project’s social codes. Some projects are technoraties, and may seem hostile to newbie suggestions without code to back them up. Others are welcoming and supportive. Do some small work showing what you are capable of. Fix things in their wiki documentation. Create pull requests for simple fixes. Join in their discussion. Grow your fame. Stay humble. Listen the long time players.

    Release your own

    Made a cool script at work? A build recipe for some special case? An Ansible playbook automating som often-visited task? A puppet module? Ask your manager for permission to release it as Free Software. Put GPLv3 or some other OSS license on it, and put it on Github. Make a blog post about it. Tell about it in social media. Congratulations, you are now an open source project maintainer. Also, Google will find it, and so will other users.

  • Hording AD groups through wbinfo

    Tuesday, November 24th, 2020

    In a samba setup where users and groups are fetched from Active Directory to be used in a unix/linux environment, AD may prohibit the samba winbind tools like wbinfo to recurse into its group structure. You may get groups and users and their corresponding gids and uids, but you may not get the members of a group.

    It is usually possible to do the opposite, that is, probing a user object and get the groups that user is member of. Here is a little script that collects all users, probing AD for the groups of each and every user, and sorting and putting it together. In perl of course.

    https://github.com/ingvarha/groupmembers

    Packages of varnish-6.0.5 with matching vmods for el6 and el7, and a fedora modularity stream

    Thursday, January 2nd, 2020

    Some time back in 2019, Varnish Software and the Varnish Cache project released a new LTS upstream version 6.0.5 of Varnish Cache. I updated the fedora 29 package, and added a modularity stream varnish:6.0 for fedora 31. I have also built el6 and el7 packages for the varnish60 copr repo, based on the fedora package. A snapshot of matching varnish-modules, and a selection of other misc vmods are also available.

    Packages may be fetched from https://copr.fedorainfracloud.org/coprs/ingvar/varnish60/.

    vmods included in varnish-modules:
    vmod-bodyaccess
    vmod-cookie
    vmod-header
    vmod-saintmode
    vmod-tcp
    vmod-var
    vmod-vsthrottle
    vmod-xkey

    vmods packaged separately:
    vmod-blobsynth
    vmod-rfc6052
    vmod-querystring
    vmod-blobdigest
    vmod-memcached
    vmod-digest
    vmod-geoip
    vmod-basicauth
    vmod-curl
    vmod-uuid

    Packages of varnish-6.2.0 with matching vmods, for el6 and el7

    Thursday, March 21st, 2019

    The Varnish Cache project recently released a new upstream version 6.2 of Varnish Cache. I updated the fedora rawhide package yesterday. I have also built a copr repo with varnish packages for el6 and el7 based on the fedora package. A snapshot of matching varnish-modules (based on Nils Goroll’s branch) is also available.

    Packages are available at https://copr.fedorainfracloud.org/coprs/ingvar/varnish62/.

    vmods included in varnish-modules:
    vmod-bodyaccess
    vmod-cookie
    vmod-header
    vmod-saintmode
    vmod-tcp
    vmod-var
    vmod-vsthrottle
    vmod-xkey

    Updated packages of varnish-4.1.11 with matching vmods, for el6 and el7

    Friday, March 1st, 2019

    Recently, the Varnish Cache project released an updated upstream version 4.1.11 of Varnish Cache. This is a maintenance and stability release of varnish 4.1, which you may consider as the former “LTS” branch of varnish. I have updated my varnish 4.1 copr repo with packages for el6 and el7. A selection of matching vmods is also included in the copr repo.

    Packages are available at https://copr.fedorainfracloud.org/coprs/ingvar/varnish41/

    The following vmods are available:

    Included in varnish-modules:
    vmod_bodyaccess
    vmod_cookie
    vmod_header
    vmod_saintmode
    vmod_softpurge
    vmod_tcp
    vmod_var
    vmod_vsthrottle
    vmod_xkey

    Packaged separately:
    vmod-curl
    vmod-digest
    vmod-geoip
    vmod-memcached
    vmod-rfc6052
    vmod-rtstatus
    vmod-uuid
    vmod-vslp

    And varnish-agent is also thrown in.

    Please test and report bugs. If there is enough interest, I may consider pushing these to fedora as well.

    Varnish Cache is a powerful and feature rich front side web cache. It is also very fast, and that is, fast as in powered by The Dark Side of the Force. On steroids. And it is Free Software.

    Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, data center, and cloud, contact us at www.redpill-linpro.com.

    Updated packages of varnish-6.0.2 matching vmods, for el6 and el7

    Wednesday, November 28th, 2018

    Recently, the Varnish Cache project released an updated upstream version 6.0.2 of Varnish Cache. This is a maintenance and stability release of varnish 6.0, which you may consider as the current “LTS” branch of varnish. I have updated the fedora rawhide package, and also updated the varnish 6.0 copr repo with packages for el6 and el7 based on the fedora package. A selection of matching vmods is also included in the copr repo.

    Packages are available at https://copr.fedorainfracloud.org/coprs/ingvar/varnish60/

    The following vmods are available:

    Included in varnish-modules:
    vmod-bodyaccess
    vmod-cookie
    vmod-header
    vmod-saintmode
    vmod-tcp
    vmod-var
    vmod-vsthrottle
    vmod-xkey

    Packaged separately:
    vmod-curl
    vmod-digest
    vmod-geoip
    vmod-memcached
    vmod-querystring
    vmod-uuid

    Please test and report bugs. If there is enough interest, I may consider pushing these to fedora as well.

    Varnish Cache is a powerful and feature rich front side web cache. It is also very fast, and that is, fast as in powered by The Dark Side of the Force. On steroids. And it is Free Software.

    Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, data center, and cloud, contact us at www.redpill-linpro.com.

    Updated packages of varnish-6.0.1 with matching vmods, for el6 and el7

    Thursday, October 11th, 2018

    Recently, the Varnish Cache project released an updated upstream version 6.0.1 of Varnish Cache. This is a maintenance and stability release of varnish 6.0. I have updated the fedora rawhide package, and also updated the varnish 6.0 copr repo with packages for el6 and el7 based on the fedora package. A selection of matching vmods is also included in the copr repo.

    Packages are available at https://copr.fedorainfracloud.org/coprs/ingvar/varnish60/

    The following vmods are available:

    Included in varnish-modules:
    vmod-bodyaccess
    vmod-cookie
    vmod-header
    vmod-saintmode
    vmod-tcp
    vmod-var
    vmod-vsthrottle
    vmod-xkey

    Packaged separately:
    vmod-curl
    vmod-digest
    vmod-geoip
    vmod-memcached
    vmod-querystring
    vmod-uuid

    Please test and report bugs. If there is enough interest, I may consider pushing these to fedora as well.

    Varnish Cache is a powerful and feature rich front side web cache. It is also very fast, and that is, fast as in powered by The Dark Side of the Force. On steroids. And it is Free Software.

    Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, data center, and cloud, contact us at www.redpill-linpro.com.

    Five valuable considerations while moving services to the cloud

    Friday, August 17th, 2018

    Also posted at Redpill Linpro’s Operations and DevOps blog.

    Going cloud is the new black, and has been for a few years already. Customers ask us: What is your Cloud Strategy? Can you help us moving to the cloud? Are your services Cloud Compliant? (Is that even a valid term?)

    Burning the bridges and moving everything to the cloud may sound compelling, and public and private cloud services may be quite rewarding. No hardware responsibility. Pay for what you actually use, not what you may use. Scale up. Scale down. Scale out. Infinite storage. Multi data center. Multi location. High availability. Global location based load balancing. Functions as a service. Databases as a service. Anything as a service! Why wait?

    All players in the IT field should consider cloud technologies, but while planning the setup of a new stack, there are issues to take into consideration. Not all services suits any cloud setup. Here are a few real-life scenarios:

    1. Apps in the cloud, cache locally

    A media house had reimplemented their production stack at a public cloud provider. The apps worked great, the developers were happy, the performance was satisfactory, and the users content. But the costs went up, as the cloud provider’s network traffic toll was quite high. Using a CDN could be a solution, but was considered too costly and unnecessary, as most users were local to a few central locations.

    Keeping the stack in the cloud, and adding local web caches to our data centers ended up being a good solution, keeping high traffic volumes on low cost lines, while sending backend traffic to the cloud.

    For high volume sites, consider caching in local data centers

    2. Apps locally, cache in the cloud

    Another media house had a classic in-house publication system, but had readers around the globe. The volume was quite low, but with high quality content to paying customers worldwide. Users in South-East Asia or the West Coast of USA got high latency and slow content loading. Actually building a scaled-down CDN service, we put cache servers in public cloud provider locations close to the users, and got happy readers. Using the Varnish Plus product, local users got cached content even though protected by a paywall.

    For low volume sites, consider local caching in the cloud

    3. Legal storage

    A media content provider was moving their services and content library to the cloud. They considered using their existing platform for building a library product, delivering storage and search for images and video data, directed at public service usage, and asked for advice. Their platform used Amazon’s AWS S3 for storage and AWS Glacier for backup. Then it occurred that storing data for public services abroad might have legal consequences, and we had to search legal advice on document storage. A proposed solution was to use Ceph based S3 compatible storage services in our data centers within Norway. With this solution, network traffic expenses became a factor.

    Storing data abroad may have consequences

    4. Anything as a service, cost by call

    Skipping the overhead of creating and maintaining virtual machines is tempting. Public cloud providers offer Database as a Service variants compatible with well-known databases, including most SQL and noSQL variants. Include Functions as a Service, and you may be able to build a complete serverless solution including data backend and APIs.

    A customer built a system using cloud provided services for database and message queue. It worked flawlessly for development and test, but when adding production traffic to the solution, cost became a large issue, as the services were tolled by request. Biting the bullet, they admitted the work of maintaining virtual machines for some of the services, paying for cpu, memory, network, and man hours.

    When using anything as a service, consider traffic driven costs against the overhead of server management.

    Also, «serverless» computing is of course a lie. The servers are there, the interface just hides the database setup. In a test using one public cloud provider’s MySQL variant, a multi database slave instance setup was shown to be quite non-resilient against sudden death, with downtime for the service while the slaves were resynced.

    There is no such thing as «serverless» computing, just another level of abstraction in front of another computer

    5. Trust the cloud provider, the cloud provider is your friend

    A well-known story tells how a complete site was taken down by a public cloud provider’s robots looking for suspicious activity. With customer chat down, and no on-call service available, the developers were forced to handle the incident by waking up the CFO, and send credit card information manually to the provider. Read the full story at https://bit.ly/2yWQBnD

    Last year, a major part of Amazon’s S3 storage system went down, and was unavailable for hours, making trouble for thousands of sites. Read the details at https://amzn.to/2melOup

    Nobody is perfect. Not even public cloud providers. Also small fish are … small, so who are you gonna call?

    Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at www.redpill-linpro.com.

    Packages of varnish-6.0 with matching vmods, for el6 and el7

    Thursday, August 9th, 2018

    Some time ago, the Varnish Cache project released a new upstream version 6.0 of Varnish Cache. I updated the fedora rawhide package a few weeks ago. I have also built a copr repo with varnish packages for el6 and el7 based on the fedora package. A selection of matching vmods is also included.

    Packages are available at https://copr.fedorainfracloud.org/coprs/ingvar/varnish60/

    The following vmods are available:

    Included in varnish-modules:
    vmod-bodyaccess
    vmod-cookie
    vmod-header
    vmod-saintmode
    vmod-tcp
    vmod-var
    vmod-vsthrottle
    vmod-xkey

    Packaged separately:
    vmod-curl
    vmod-digest
    vmod-geoip
    vmod-memcached
    vmod-querystring
    vmod-uuid

    Please test and report bugs. If there is enough interest, I may consider pushing these to fedora as well.

    12 days of Varnish

    Tuesday, December 19th, 2017

    While Varnish is most famous for its speedy caching capabilities, it is also a general swiss army knife of web serving. In the spirit of Christmas, here’s Twelve Days of Varnish Cache, or at least, twelve Varnish use cases. Read the rest of this post on Redpill Linpro’s Sysadvent calendar.