To infinity… and beyond!

DevOps, IaC, AGILE, SCRUM, Containers, SysOps, CI/CD, microservices, serverless, FAAS

Yes, it’s a buzzwords shitstorm but all of these concepts summarize how software development and deployment evolves. Do i think that sysadmins time is over ? No.

But time as come, we, as sysadmins, have to adapt to this moving industry.

Accept it or die, but cloud computing is the norm now. Small IT companies with “good old LAMP” as the only business model will die soon.

Dont’ be scared, take a deep breathe and let’s dive into this !

DevOps is the new sexy

DevOps = “Dev” + “Ops”

DevOps, SecDevOps, DevOps Enginner are the most used terms in job requests. But, after all, what the fuck is that shit ?

In most cases, IT projets human resources can be split into two teams :

• Dev : This team write application code. Pushed by marketing teams, clients or users, the main purpose of the dev team is adding new feature to the software.

• Ops : This team have to maintain operational conditions and ensure that prod is up, up to date and stable.

By means, this two job positions are in conflict. On one side, ops team wants stable and predictable stuff, on the other side, dev team wants liberty to move quickly in order to add new features and meet commercial goals and objectives.

What could be done to satisfy both worlds ? DevOps

DevOps culture

First of all, no, DevOps is not a job title (even if everyone on LinkedIn thinks so). It’s a culture, a way of thinking and the more important point of this is : it tells how to organize software development teams.

The purpose is to put ops and devs to an agreement. Afterall, everyone works for the same goal : build the best product, work less and make more money.

DevOps presents a new way of thinking how teams collaborate. The main purpose of DevOps is to open discussions between Ops and Dev. Dev have to be aware of how the code is deployed and how the production systems are handled by the ops team. On the other hand, ops needs some dev skills in order to understand correcly what type of software is served in production.

To sum it up, DevOps could be presented using a loop of all these concepts and actions :

• Define a task, or feature, focused on needs from customers or clients
• Implement this feature or execute the defined task
• Code review
• Tests (unit tests, integration tests, staging tests)
• Functionnal testing on a preproduction environment
• Push to production
• Handle and analyse feedback from production

In order to archive all this tasks, teams have to communicate clearly on what should be done by each team and in some cases, there is an overlap between dev tasks and ops tasks but the final goal is the same : get to the best possible result.

Devops methods and concepts

Now that the culture is presented, let’s take a look at how this principles can be applied to the real world.

Contract

The top most important thing is what I call a contract. This is an agreement that create a link between dev and ops. A contract needs to be as descriptive as possible. On one hand developpers needs to tell to administrators all of the project details, what needs to be run and what are the dependencies or services required by the application. On the other hand, operators have to understand dev needs and do all the plumbing to deploy dev requests to production.

In most cases, this contract can be represented by containers and a docker-compose file. It’s declarative, easy to read, easy to understand and clear enough to know what needs to be run and what kind of plumbing is needed to make all services works together to create the whole application.

If this not clear enough, here is a generic example :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

version: '3'

services:
  web:
    image: galaxies:version
    volumes:
      - ./src:/dest
    depends_on:
     - db
    ports:
     - 8080
    label:
        - "frontend.rule=Host:galaxies.rick"

  db:
    image: postgres
    volumes:
      - ./mounts/db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: RICK
      POSTGRES_PASSWORD: C137
      POSTGRES_DB: galaxies

An ops, receiving this file, can extract a lot of information on how the application works.

• A web app that :
  • is accepting requests on port 8080
  • is responding using the url galaxies.rick
  • needs persitent storage
  • needs a a postgres database

Now let’s explain what should be done by ops to push this to production :

0. Ensure that galaxies.rick DNS points to production environment
1. Pull galaxies:version image
2. Ensure an access to a database (can be a container, a cluster or a standalone pg)
3. Inject database variables into production environment
4. Start the galaxies:version image
5. Update HTTP reverse proxy rules to redirect galaxies.rick to galaxies:version on port 8080

And voilà, now you have a clear line between what kind of stuff developers will push to production and how the operators will plug the project on the production environment.

Automation

Okay, okay ! But as an ops, I don't want to take care of all this stuff everytime dev needs to push a new version of the sotfware !

Me too, and this is why the second top most important thing is automation !

Take a look at all the tasks described earlier, do you really want to make all those changes by hand using vim on the production environment ?

Ten or even twenty years ago, the first automated things was machine boostrap and basic configuration using scripts. Modern applications requirements means more machines, more complexity. The easieast way to handle this new level is to delegate some of the tasks to computers using declarative structures.

This is why tools like Ansible are now popular and widely used. Today we want to describe a state and let tools do the stuff needed to get to this state. Why ? Because this is the simpliest way to normalize how things have to be done¹ and to get complex systems up and running. If there is a bug or a missing feature in one of this tool, there is a good chance that you will have to put your hands in the grease and code it.

This first step of automation gives ops opportunity to work on tools associated to concepts like Continuous Integration and Continuous Delivery.

Do not forget that current apps do more things than serving a simple website. By using small iterations, dev gain the ability to make small changes often in contrast to doing major updates. This way of develivering software enhance stability because most of the code base is not changed between deployments to production. If something goes wrong it will be a lot easier to bisect to the root causes of this bug or unexpected app response.

To give dev the power to deliver atomic changes to pre-production, staging and production, they need to deploy all the stuff by themselves in a predictable and reproducible and the answer is automation !

Metrics & alerting

With great power comes great responsibility !

This is why metrics matters. If devs can deploy stuff to prod, they also need to know if everything is working as expected and no, I will not give them root access to production !

Enabling low level metrics ensure to ops that production is up and running smoothly but this is not enough to apply DevOps principles. Devs also need visitiliy on how the app is handling requests (success / error rates) or some status about the queue system. Every team needs different sets of metrics, specific to their missions.

With metrics, comes alerting. If metrics are well defined, alterting can be routed to people who are aware of what can be done to resolve the problem. For example, if the app goes crazy, a member of the dev team will be in the right position to take action and fix the problem but if it’s a proxy memory leak, an ops will probably know what to do and to dig, to find and resolve the issue.

Performance monitoring is as mandatory as alerts. Teams wants to be sure that the newly pushed feature is working and that there is no performance drop somewhere else in the application.

Culture, sharing and empathy

Sharing is caring

Sharing is one of the key to apply DevOps concepts and culture. In this situation, sharing is not restricted to communication but also responsabilities

If dev teams share responsabilities with ops team, there is a all new field of possibilities to collaborate and simplify deployment and maintenance. For ops team, a better communication and shared responsabilities with dev ensure that they have access to informations regarding businesses goals, productions requirements.

If devs and ops are both responsible of failure, or sucess, of the product, there is less possibility to fall in a blame counter-blame situation. With more communication and a strong trust chain between members of ops and dev teams, everyone gets autonomy and a voice in dev or deployment process.

Last but not least, a lots of empathy is needed. Failure is a success ! It’s one of the best way to learn new things but this is only possible in a safe and tolerant environment. The war between dev and ops is over. Listening and talking to eveyrone will probably help every member of the teams.

Limits

Of course, DevOps is not perfect. When misunderstood, this can be quite catastrophic. Reducing DevOps to a buzzword is as silly as taking everything as Truth in the holy Bible. DevOps is a mentality, you will have to create your own DevOps culture.

A lots of managers and leaders thinks that DevOps means, fire half of the devs , half of the admin, mixup, and voilà. Is this sharing ? No ! Doing DevOps means more distributed responsability and less stress for everyone.

Another wrong idea is to think that more deployments means more features. Integrating deployments automations and testing is only used to enhance robusteness of the all system. The idea is not to make more feature, but more small changes in order to increase poduction stability.

A new open field

After all that, there is clearly an emerging need of programming skills common to all the principles and methods presented.

Why Industry needs (smart)ops ?

A smartops is someone who clearly understand that the IT industry is changing. Everything is moving to the cloud, more and more services are externalized and everything becomes more and more automated. All this stuff creates a violent shift between two sets of methods.

• an old one

  • launch command in a terminal using ssh
  • bash scripts to setup things
  • edit file directly on production using vim

• a new one

  • pipelines
  • automation
  • services interactions between HTTP services

No I’m not saying that ssh is dead. I’m saying that methods evolves.

As more automation means less human action, there is clearly a move to descriptive infrastructure deployments and internal services doing all the plumbing stuff needed to get a stable and viable production.

In order to achieve all this new challenges, industry needs to delegate tasks to smart programs write using code. This new services and automation programs have to be written by ops, beceause they are the ones who trully knows how to run production systems at scale. But, sorry to say that, perl and bash scripts can do that kind of jobs. More automation of everything also means automation of the most complex tasks in the stack and this is where scripting langages are not enough.

For the ones who thinks I’m wrong, maybe. But here is my opinion based on a lots of bash and perl script experiments. When things needs at least parallelism, http requests and response manipulations, strong error handling or ability to push stuff inside a monitoring stack, golang will be my choice and I deeply think it should be yours too because this is the main purpose of this kind of new languages, created specificaly to answer dev and ops problems.

Moving from scripting to programming will also help smartops understand how software they put in production works. By knowing how to construct a software, ops will gain the ability to help devs, integrating every one in a DevOps culture.

New profiles, new horizons

Yes, ops needs dev skills in order to get a role in teams resolving new challenges that comes with modern infrastructures and cloud infrastructures.

This changing ecosystem also gives evolution ability to ops and dev. With efforts, everyone can, at least, take a look at how roles and interconnection between ops and dev works. To be clear, i’m saying that dev also needs ops skills ! But i keep that for another article, stay tuned.

If old ops don’t want to make the effort, that’s not a problem because new kind of people get what is happening. Believe it or not but it’s real. The smartops community is inclusive. Even if this is not perfect yet, the golang and k8s is clearly LGBT and women friendly !

I want to thanks all the LBGT||women gophers||ops² I follow because they are the roots of this wonderful and refreshing community³. The best thing I can do is to invite you to follow this people ! Here is the list :

• Jessie Frazelle
• Ashley McNamara
• Ellen Korbes
• Kris Nova
• Jaana B. Dogan
• Francesc Campoy
• Aditya Mukerjee