The road to automation

Note: This post is the first of a series of, probably, half a dozen posts that will detail how my blog posts are automatically tested, rendered and published after I’ve written them.

If you are simply interested in the technical details of how I accomplished the project I am going to detail below, you can safely skip this post, since it consists almost entirely of related, but non-technical prosa about why I going to do it.

I left my systems administration job right around the time when a massive shift in how we are thinking about systems administration was happening. Automation was moving away from clunky shell scripts to things like Saltstack, Ansible or Puppet. You weren’t isolating customers in chroots that were held together by duct tape and blood sacrifices any longer, containers were becoming the standard rather than some nerdy hobby project. The entire culture changed, with servers stopping to be beloved pets, and turning into cattle instead.

I was lucky enough to learn from some incredibly talented people when I started my career, which means that I have a relatively reliable grasp of what I am doing when logging into one of my machines. And when stuff happens to break, most of the time I have at least some vague inkling about where I should look first.

The problem is: I have no idea how to do it “the cool way”. And because I never really stopped working on or with servers even when not doing it professionally that is almost entirely my fault. Nothing (except for an at-the-time undiagnose neurological disorder) was keeping me from learning about things like Docker and Kubernetes, Terraform and Ansible, S3 and Azure, Elastic, Telegraf and InfluxDB, and so on. It’s now easier than ever before to teach yourself how to make computers reliably do what you want.

To be transparent: I was quite hostile towards everything that had the label ‘DevOps’ attached to it at the time. I thought this was all a fad, that it was mostly marketing bullshit, that the people pushing the different topics were idiots who knew nothing about computers and so on.

And to add some uncomfortable honesty to this monologue: Now that I’m not barely 20 anymore, and with quite a few years of experience added to my CV, I can safely say that very little of my complaints were objective or even based on facts.

Don’t get me wrong, I still feel quite strongly about certain practices that have become standard in the industry. For example about the seemingly ever-growing reliance on third parties for core services of our personal and professional lives. Or the tendency to put abstraction layer on abstraction layer to simplify things, only to end up with incredibly complex systems that are, in some cases, impossible to debug once things go sour. Or everything that is produced and / or sold by Atlassian or Oracle. Admittedly, that last one might not be entirely objective.

But for the most part it was some, in hindsight, weird form of jealousy. I was, at the time, slowly but steadily becoming comfortable at referring to myself as a systems administrator. I successfully tackled all the topics that scared me, because I perceived them to be super complex. I had set up my own mailserver, after being scared of doing so for literal years by all the horror stories about open relays that I have been heard of or read about online.

I was finally “cool”. And now there was a plethora of new things to consider in order to be, in my twisted perception, “cool” again? There was a lot of stuff that I did not understand? There was a lot of stuff that would require me to admit that I don’t understand things, and force me to start new? There was a chance to learn great things from tremendously talented people, eventually furthering my understanding of computers?

Well, obviously not. All that was happening was nothing more than a bunch of young people trying to dumb down computers and the administration of computer networks because they were unable to learn things the hard way .. like I did (add another dozen exclamation marks here)!

And don’t even get me started on those people who were daring to use Linux-distributions that came pre-configured with a desktop environment, or people using (insert a scoffing tone here) Windows. Because Windows is obviously a bad operating system. Because .. yeah, well it’s shit (once again, add another dozen exclamation marks here, in order to compensate for the lack of arguments)!

Despite the fact that gatekeeping and elitism is never acceptable behaviour, it becomes especially embarrassing when you consider that I spent not even five years working in IT at the time, being barely in my early 20ies. I knew literally next to nothing. I had not contributed anything meaningful, nonetheless talking down on people who built things like entire new configuration management systems.

Yet in my mind I was part of a “better” “generation” of system administrators, rightfully having the moral high ground. It was a ridiculous mindset that lead to some horrible behaviour I displayed at times.

And while it’s uncomfortably, and I’m talking about being physically discomforted while writing these sentences, embarrassing to admit that this was the case I’m also happy that it makes me feel that way - because it’s an indicator that I have grown as a human being and that I’m most likely not an edgy teenager anymore. As slim as the chance is to have anyone of the people I was rude to reading this: I am genuinely sorry. I was an asshole.

When I talk to people about IT, especially when they are not working in IT themselves, I sometimes get asked how I became so good at computers. My answer is always the same and consists of two parts.

• I’m not that good at computers. I would consider myself moderately talented, proficient and knowledgeable at best. What I am good at is looking at issues and knowing what to search the Internet for.
• For the computer-related topics that I am actually good with the answer is surprisingly simple - I broke my machine so often that I eventually learned how to repair it myself, or avoid breaking it in the first place. By making the “Oh shit”-kind of mistakes that you make once and never, ever again. Because of the amount of internally wailing and missed sleep they caused you.

So, .. what better way to learn about exciting new things by getting my hands dirty and making horrible mistakes?

I set myself the following goals in order to have a clear definition of what I am planning on achieving, which should both help with keeping on track and holding myself accountable. Especially the latter is totally not something I have had issues with in the past. Not at all.

• Bring the website online

If you are reading this, then this step has been completed successfully. While I am certain that much of the issues I faced were because of my own errors and unwillingness to thoroughly read the documentation, I was annoyed by the regular hassle with paths and missing gems.

So while I liked Jekyll before, I decided to forego using it in favour of Hugo this time, opting for the theme PaperMod. And while the process of posting is currently an annoying combination of running commands manually, it’s a step into the right direction.

• Set up version control

As of right now I have the content of the site under version control, pushing changes to Github every time I work on it. But monopolies are a bad thing, and everyone using the same centralized hosting kind of negates the decentralized aspect of git.

So my plan is to keep using Github, but only as a mirror of my “main” repository. I have an instance of Gogs running on a machine that I use for a few things, but since the developers of Drone (which I plan on using) explicitly recommend switching to Gitea in their documentation I might switch over.

• Set up CI/CD

I mentioned it above, I’m planning on using Drone for that task. CI/CD is a topic that I have to do more research and learn a lot more about, but from what I have gathered so far it looks like my choices are limited and despite its shortcomings, Drone might be my best bet.

I don’t want to throw ton of memory at Jenkins, and most of the other products I found either restricted on-premise hosting to their enterprise customers, made it deliberately hard to run the software on your own hardware or were too complex and feature-rich for my needs.

• Integrate CI/CD into version control

Ideally, both my version control and my CI/CD-software are able to interact with each other. I’m not yet sure to which degree I want to achieve this, but for the beginning I want to be able to access repositories and manually run tests.

• Fully automate the deployment

That means that once I push a new post into the repository, a build on my CI/CD tool is triggered automatically and once that has succeeded, the site is redeployed to my server. I’m not sure if that’s feasible or would unnecessary complicate things, but I’m contemplating working with git-branches instead of Hugo-drafts.

The obvious answer to the question that some of you might ask is: Yes, absolutely. This setup is going to be massively overengineered. It would be perfectly fine to rent some webspace for pennies and directly publish this blog there (and it’s always a good idea to contribute to not-exactly-for-profit hosters like Uberspace).

But this endeavour isn’t about the most simple or easiest solution to a problem. It’s about learning, tinkering and having fun. Which is something that all of us should enjoy more of. :)