Software is Eating the Ops World

One thing I've thought a lot about is how the role of the system administrator is changing. This reflection was prompted by a couple of things: one, I'm a co-chair for talks at one of the longest running system administration conferences, so I should probably think about this kind of thing seriously when planning what talks we'll accept, etc. The other thing, though, is that I've read what some peers have had to say about the tone of the Google Site Reliability Engineering (SRE) book. My own interpretation is that the book thinks of traditional system administrators as "button pushers" who solely operate something that someone else gave them; similar to what you see in many large organization IT departments. There's a heavy emphasis on Engineering™, which isn't present in large organization IT departments. I haven't really dug in to the book -- so I'm going to leave those thoughts here and circle back in a few.

The idea that administrators don't program reminds me of my first interview for a system administration internship back in May 2007. This is funny to me because one thing I remember clear as day was stating that I was not a programmer. Code was not something that I did. I just made computers work well and do useful things for people. Other people wrote code, I just lived in their world.

Yet, in the last week I found and patched a bug1 in Debian apt (a minor one at that, but nonetheless something installed on many Linux machines across the world) and added functionality to the software we're using to review submissions for LISA '162. This does not correlate with the 2007 me, who decided to be so resolute in what he could do and could (would?) not do. What gives?

I was one of those stereotypical kids who was good at computers. In short, I ended up making my town's website at a young age, could fix computers and their software issues, played with Linux at a young age, life was good. What was more difficult to wrap my head around was learning how to actually program. I was given a copy of Visual Basic growing up; I wasn't quite sure what to do with it. Our basic CS classes in high school had us reading in values and printing out algorithmically generated Christmas trees. I was not exceptional at math, a skill that people often referred to as extremely important to being able to write programs.

Here's the thing, though: I was able to run my high school's email server, and make sure backups worked for a small elementary school in the school district, and I was later asked to help my town buy and maintain computers at town hall and basically be the IT manager there. In considering my future, system administration didn't seem like it got the same respect that developers did, but I felt decent at it. I felt decent at making systems reliable and avoiding data loss and working around the issues that developers seemed to throw at their users (both end users and administrators). Additionally, people seemed to find value in these skills, and even paid for this. I was a lucky high schooler, and undergrad student, no doubt. Speaking of undergrad, I sucked it up and did what I could to pass C++ and Java (the only coding requirements for my major) but wasn't particularly stellar at either.

I remember being interested in automation -- it was important to figure out out Windows Deployment Services because there was no way I was going to install Windows by hand on 20+ machines. Reducing patterns into shell scripts was a lot more palatable than going around doing things by hand. During my internship, I wrote a provisioning system in Perl because one of my first assignments was to get Red Hat Enterprise Linux 4 onto 14 machines. There was parsing to be done to move the infrastructure from NIS+ to LDAP. User accounts were previously done by email - we moved to a web-based system after I made one.

The entire time I was working on these things it never felt like programming-with-a-capital-P. I was writing in Perl, PHP, BASH and TCSH -- not the compiled languages I had struggled with earlier. But I wasn't a developer, I wasn't mathing (is that a word?) and so it never felt real. No complicated algorithmic sorts here - just some SQL and outputting HTML and moving text around when I had to.

I think that I would have been a lot better off if I started with a different mindset: if you made a computer do something, congrats! you did a code. I'm not going to say this is the same as building a complex distributed system for production use, for sure (maybe someday!). I do think I would have been more comfortable with such a thing earlier (rather than now) if I had a few wins early on. Alas.

Today I mostly write in Ruby, and have reached the point where I can jump to another language and flail around enough to get something done. I "get" types now, thanks to toying around with writing something in Go. A few weeks ago I wrote an internal signup form in node.js. I still experience a certain amount of angst when I start writing something, but once I get going it's okay.

I wrote this because I don't think I'm the only person in the world who has learned how to code as a system administrator or operations engineer or whatever it's called today. As software continues to eat the world, I don't think the operations side of the house is going to get a free pass on continuing to maintain the status quo just because these issues are hard to write logic around. Indeed, some places are figuring out how to remove the human out of the loop as we speak. I see this change in the interview stories that people share in various ops-heavy communities (hangops, etc.), and in the kind of work that people at the middle or top of the field (that I look up to) are doing.

Which brings me back to the SRE book. I can understand the frustration with the way in which they write of (write off?) system administration, especially for those who have cut their teeth and evolved over time. But even 10 years ago, I didn't have the impression that system administration involved actual engineering so much as just MacGuyver-ing around the decisions of folks who may or may not have run a system in production. Perhaps I was too naive. As the pendulum swings towards larger organizations being more comfortable running their own systems, more organizations are going to look at processes and techniques that have worked for others and try to apply the ones that make sense to their organization. Obviously, not everyone is a large organization that makes everything from scratch, but there are broader lessons to be learned here.

Administrators (myself included!) have complained about the disconnect between people who write software and us for decades. Now we finally have a chance to sit at the table in some organizations. My own hangups about code ability aside: this is actually exciting! We get to care about the things that are important to us! This shift carries with it some baggage3, but I suppose sitting at the table ain't free, either.

I'm looking forward to a future of working with (and contributing to) systems that care about ops the way I always have, regardless of what it's called. That's pretty cool in my book.

Thanks to Julie Hansbrough, Jarrod Pooler, Karen Rivera, and Carolyn Rowland for their thoughts and corrections.

  1. Shameless feel goodery

  2. Jason Dixon's work on Judy has been a lifesaver as a conference organizer. Thanks Jason! 

  3. tef's writing on whiteboard interviews comes to mind when thinking of baggage. 

Masters Project Defense

On 5 December 2014, I will present/defend my masters project, Demand-Provisioned Linux Containers for Private Network Access at Rochester Institute of Technology, room 70-2115 at 2:30 PM EST.

At work, I have created a neat system for non-persistant bastion or "jump" hosts, that allow you to connect one network to another. I toss in some auditing, multifactor auth, and containers give me (a little) isolation. It's a neat proof of concept, and I'm excited about presenting it. This is a public event, so, I suppose if you are in the area and interested in Docker, come on down.

I am especially excited for the snake fight portion of my project defense. Hopefully my assigned snake will be small. Also: hopefully it does not snow and cancel my flight... that would be the worst.

People and Ops

I'm a master's student at Rochester Institute of Technology. Someday, it's likely that I'll graduate. I'm excited to write a post about that someday, especially because I think the people who may read this site would find the project aspect particularly interesting.

Anyways, back in 2011 I wrote a personal statement as part of my application package. Squeezing out the mushy details of my past projects and employers, leaves this:

At the end of the day, it is my belief that system administration is about bringing people together.

System Administrators are the maintainers of spaces that exist only physically by way of printed circuit boards, integrated circuits and interconnected copper and glass. The gap between states, nations, continents, cultures, and even space are now almost purely a matter of one's interest in bridging the distance between two points. The responsibility of maintaining this virtual space is not one to be taken lightly. People depend on the networks we create and maintain for a range of seemingly petty and very important (life or death) reasons. It is no longer practical or reasonable to be okay with "good enough" system administration.

[...] While technologically impressive, all these networks and architectures exist for a bigger reason.

I'm not quite sure what the point of posting this is, other than to remember the human aspect of what we do.

sudo serve && protect

Not too long ago, this guy held a logo contest to make an ops-themed logo with a union flare to it. Someone else came along and threw it up on The logo is CC-BY-4.0 which is pretty neat, so I took some time to make a few more locals.

Area code...

Not perfect, but fun enough if you happen to be in one of those area codes. These derivatives are licensed CC-BY-4.0 as well.

How'd you do that?

I made these using iDraw for Mac OS X. It seems to be a decent vector image editing tool for the mac, and only $25.

Docker and CentOS4

As much as we would want a world in which all applications were updated regularly and licensed sanely, that is not the world we live in. Some applications cost several hundreds of thousands of dollars (per seat!) and their users expect to be able to use them into the future, even after the vendor has moved on.

One solution to let these applications run and not have to keep older OS machines on your network is Docker. Docker does have a variety of base images available through their site, however typically they are newer OSes.

If you find yourself in the sad position of making a CentOS4 base image, here are some helpful pointers:

  1. CentOS4 used up2date to pull down system updates and new RPMS. However, it also supports Yum, and the CentOS4 repos do have a repodata folder for yum use. Grab the appropriate repodata folder, and put it in your /etc/yum.repos.d folder. Make sure to add a line for enabled=0.
  2. You will find the script here to be of particular value. I cut out the usage(), yum_config, and getops stuff.
  3. Add in a test to make sure $target exists. What if you accidentally kill off /tmp and then run the script and you forget to test for if $target exists? This exercise is left to the reader (but see #7)
  4. You will need to modify the first yum (...) install line to include --enablerepo=centos4 --disablerepo=*.
  5. Need extra packages? Make sure to copy in your repo definition from /etc/yum.repos.d to $target/etc/yum.repos.d as the the groupinstall will provide a repo that may not be what you want it to be.
  6. You will likely need to add in some important .i386/.i686 RPMs, especially if you are running the build script on a RHEL6 machine. Some good ones to include include libgcc and glibc.
  7. Kill off the RPM repository.1 rm -Rf $target/var/lib/rpm -- the reason for this is that you are likely running this on a newer RHEL machine. Older versions of RPM will not be able to read the RPM database that you have created. If you really need the ability to do development and install RPMs in a docker repo to figure out what you need, after you have built a base image, run rpm --initdb and re-run your groupinstall command. Then, tar the RPM library and put it into the script so that you may experiment without having to perform this step manually. I do suggest that once you are done with dev, you just kill off the RPM library to begin with -- no one should be installing RPMs inside your artisanal container, right?
  8. Since you know your OS name, it may be beneficial to just set name=centos up front.
  9. Bam. You have a CentOS4 image now!

There are arguments to be made for being able to use yum in a Dockerfile however, given that only one application i know of needs to be run like this, I find that building this once and keeping the scripts in revision control work well enough for me.

My Dockerfile is fairly simple - I have a FROM and MAINTAINER line, with an ADD and an ENTRYPOINT - the ADD simply drops in an init script that adds a local-to-container user with the appropriate uid/gid, then switches to that user and runs the application. X11 use is handled by proper management of the DISPLAY variable and/or by exposing /tmp/X11-Unix to the container.

Hopefully you find yourself not needing to support older apps. But, in the event you do, maybe this will help.

  1. katzj notes that he has a solution to this. See his tweet for more info.