Evolution of System administration to TechOps/DevOps Culture
Wanna upgrade your self into next generation system admin?
We offer two options for you.
- Option1 : Signup For our Free Email crash course – “Free Email Crash Course – Getting Started with Puppet in 7 Days” –
- Option 2: Join our Weekend PAID Workshop “Automation with Puppet Workshop for unix and Linux professionals“. This workshop is carefully designed to give you the skills required to work in an existing puppet environment.
When I started out as a system administrator, 16 years ago, a large part of my office time was spent configuring server machines and making them ready for use. This is more or less the process and steps are almost same, like installing software, licensing it, configuring it, setting passwords, and so on.
In technical term, this whole process of installing and configuring the servers is is called configuration management, and it is most tedious task of sysadmin job.
System Administration in the Past
Let’s take a closer look of how an old school system admin used to spend his day from below example:
A day in the life of an Old school sysadmin
Let’s look at some of the tasks involved in preparing a web server, which is something sysadmins do pretty often. Let’s assume that we want to configure a website named unixadminschool.com
Assuming the machine has been physically put together, racked, cabled, and powered, and the operating system is installed, what do we have to do to make it usable as a server for unixadminschool.com
- Add some user accounts and passwords
- Configure security settings and privileges
- Install all the packages needed to run the application
- Customize the configuration files for each of these packages
- Create databases and database user accounts; load some initial data
- Configure the services that should be running
- Deploy the unixadminschool application
- Add some necessary files: uploaded cat pictures, for example
- Configure the machine for monitoring
That’s a lot of work. It may take a day or two if this is the first time you’re setting up the server. If you’re smart, you’ll write down everything you do, so next time you can simply run through the steps and copy and paste all the commands you need. Even so, the next time you build a unixadminschool.com server, it’ll still take you a couple of hours to do this.
If the live server goes down and you suddenly need to build a replacement, that’s a couple of hours of downtime, and with a pointy-haired boss yelling at you, it’s a bad couple of hours.
Wouldn’t it be nice if you could write a specification of how the server should be set up, and you could apply it to as many machines as you liked?
Our configuration management problem doesn’t end there:
There’s another problem. The next time you need to build an identical server, how do you do it?
Your painstaking notes will no longer be up to date with reality. While you were on vacation, the developers installed a couple of new Ruby gems that the application now depends on—I guess they forgot to tell you. Even if everybody keeps the build document up to date with changes, no one actually tests the modified build process, so there’s no way to know if it still works end-to-end.
Also, the latest version of MySQL in the Linux distribution has changed, and now it doesn’t support some of the configuration parameters you used before. So the differences start to accumulate.
By the time you’ve got four or five servers, they’re all a little different. Which is the authoritative one? Or are they all slightly wrong? The longer they’re around, the more they will drift apart.
Wouldn’t it be nice if the configuration on all your machines could be regularly checked and synchronized with a central, standard version?
The next big challenge of Configuration Management is , repeating changes across many servers in your environment:
After few weeks we want to rollout a new software theme for our unixadminschool.com servers. To roll out the new version to your five web servers, you need to install a couple of new package dependencies and change a configuration file. And you need to do this same process on each machine.
Humans just aren’t good at accurately repeating complex tasks over and over; that’s why we invented robots. It’s easy to make mistakes, leave things out, or be interrupted and lose track of what you’ve done.
Changes happen all the time, and it becomes increasingly difficult to keep things up to date and in sync as your infrastructure grows.
Wouldn’t it be nice if you only had to make changes in one place, and they rolled out to your whole network automatically?
And the other usual problems that we face with regular configuration management is :
Lack of proper documentation that reflects our current server configurations:
A new sysadmin joins your organization, and she needs to know where all the servers are, and what they do. Even if you keep scrupulous documentation, it can’t always be relied on. In real life, we’re too busy to stop every five minutes and document what we just did.
The only accurate documentation, in fact, is the servers themselves. You can look at a server to see how it’s configured, but that only applies while you still have the machine. If something goes wrong and you can’t access the machine, or the data on it, your only option is to reconstruct the lost configuration from scratch.
Wouldn’t it be nice if you had a configuration document which was guaranteed to be up to date?
Coping up with different platforms , version control and history
Ideally, all your machines would have the same hardware and the same operating system. If only things were that easy. What usually happens is that we have a mix of different types of machines and different operating systems and we have to know about all of them.
The command to create a new user account is slightly different for Red Hat Linux from the equivalent command for Ubuntu, for example. Solaris is a little different again. Each command is doing basically the same job, but has differences in syntax, arguments, and default values.
This means that any attempt to automate user management across your network has to take account of all these differences, and if you add another platform to the mix, then that further increases the complexity of the code required to handle it.
Wouldn’t it be nice if you could just say how things should be, and not worry about the details of how to make it happen?
Sometimes you start trying to fix a problem and instead make things worse. Or things were working yesterday, and you want to go back to the way things were then. Sorry, no do-overs.
When you’re making manual, ad hoc changes to systems, you can’t roll back to a point in time. It’s hard to undo a whole series of changes. You don’t have a way of keeping track of what you did and how things changed.
This is bad enough if there’s just one of you. When you’re working in a team, it gets even worse, with everybody making independent changes and getting in each other’s way.
Wouldn’t it be nice if you could go back in time, without much effort on rollback?
How do we solve all these configuration Management Problems:
Most of us have tried to solve these problems of configuration management in various ways. Some write shell scripts to automate builds and installs, some use makefiles to generate configurations, some use templates and disk images, and so on. Often these techniques are combined with version control, to solve the history problem. Systems like these can be quite effective, and even a little bit of automation is much better than none.
Reinventing the wheel
The disadvantage with this kind of home-brewed solution is that each sysadmin has to reinvent the wheel, often many times. The ways in which organizations solve the configuration management problem are usually proprietary and highly site-specific. So for every new place you work, you need to build a new configuration management system (CM system).
Because everyone has his own proprietary, unique system, the skills associated with it aren’t transferable. When you get a new job, all the time and effort you spent becoming a wizard on your organization’s CM system goes to waste; you have to learn a new one.
A waste of effort
Also, there’s a whole lot of duplicated effort. The world really doesn’t need more template engines, for example. Once a decent one exists, it would make sense for everybody to use it, and take advantage of ongoing improvements and updates.
Configuration management tools
This is roughly the situation we have now. Several different CM systems have been developed over the years, with new ones coming along all the time, but only a few have achieved significant market share.
The leading Configuration Management systems in the market are Puppet, Chef, and CFEngine.There really isn’t much to choose between these different systems. They all solve more or less the same problems, some people prefer the Puppet way of doing things; some people are more comfortable with Chef, and so on.
But essentially, these, and many other CM systems, are all great solutions to the CM problem, and it’s not very important which one you choose as long as you choose one.
System Administration , Now
Let’s discuss about one Example how System Administration using a configuration management tool , lets say puppet, changes the way that we used to work in the past:
System Administration using puppet
A new developer has joined the organization. She needs a user account on all the servers. The traditional approach will be as follows:
- Log in to server 1.
- Run the
useradd rachelcommand to create the new user. - Create Rachel’s home directory.
- Log in to server 2 and repeat these steps.
- Log in to server 3 and repeat these steps.
- Log in to server 4 and repeat these steps.
- Log in to server 5 and repeat these steps.
- The first three steps will be repeated for all the servers.
The Puppet way
Here’s what you might do to achieve the same result in a typical Puppet-powered infrastructure:
Add the following lines to your Puppet code:
user { 'rachel':
ensure => present,
}
Puppet runs automatically a few minutes later on all your machines and picks up the change you made. It checks the list of users on the machine, and if Rachel isn’t on the list, Puppet will take action. It detects what kind of operating system is present and knows what commands need to be run in that environment to add a user. After Puppet has completed its work, the list of users on the machine will match the ones in your Puppet code.
The key differences from the traditional, manual approach are as follows:
- You only had to specify the steps to create a new user once, instead of doing them every time for each new user
- You only had to add the user in one place, instead of on every machine in your infrastructure
- You didn’t have to worry about the OS-specific details of how to add users
Growing your network
It’s not hard to see that, if you have more than a couple of servers, the Puppet way scales much better than the traditional way. Years ago, perhaps many companies would have had only one or two servers. Nowadays it’s common for a single infrastructure to have hundreds and even thousands of servers.
By the time you’ve got to, say, five servers, the Puppet advantage is obvious. Not counting the initial investment in setting up Puppet, you’re getting things done five times faster. Your colleague doing things the traditional, hand-crafted way is still only on machine number 2 by the time you’re heading home.
Above ten servers the traditional approach becomes almost unmanageable. You spend most of your time simply doing repetitive tasks over and over just to keep up with changes. To look at it in another, more commercial way, your firm needs ten sysadmins to get as much work done as one person with Puppet.
Wanna upgrade your self into next generation system admin?
We offer two options for you.
- Option1 : Signup For our Free Email crash course – “Free Email Crash Course – Getting Started with Puppet in 7 Days” –
- Option 2: Join our Weekend PAID Workshop “Automation with Puppet Workshop for unix and Linux professionals“. This workshop is carefully designed to give you the skills required to work in an existing puppet environment.
It is good to know and learn about puppet too.
Ansible is also there…
Nice Article. How it help to developer in terms of balance the day to day life. Agile itself make life worse due to misuse by Scrum Masters especially in India.