Thinking about DevOps goals

I have been working on a number of data products and website products for the best part of 2.5 years. Indeed, the underlying goal was a data platform for the property market. Whilst this platform is very much a possibility now, the technology and business model are now ready for implementation.

Please take a look at my IRPA solution. It is currently on the Beta section of my Info Rhino website, as I haven't thought through the best way to brand it and am looking for ideas.

https://www.inforhino.co.uk/beta/automation-and-devops

What has this got to do with DevOps?

Any paradigm in technology always sounds trite; Big Data, Business Intelligence, Serverless, Agile - DevOps. They have an almost infinite number of interpretations. Often, they are a comfort blanket - by saying we are DevOps orientated, suddenly we are doing DevOps. We can buy some software and claim we are DevOps specialists. We often hear people say "I'm agile" and then cringe.

The specific challenges I faced with my web data platform

My data platform does amongst other things; web scraping, data transformation, data publication, reporting, data visualisation, batch processing, automated mailing. This has been done alone, aside from a small amount of outsourcing. Normally, I work on large enterprise data contracts in finance or retail. There were teams for Database Administration, Testing, Business Analysis, Deployment, Server Management, Batch Automation, and last but not least - development. My current goal was to create products that would work without most of this process and overhead.

My challenges, whilst not a team/cultural challenge, were the same - just with less people.

• Logical Units of work needing a workflow.
• Large data processing.
• Changes to configuration and source data.
• New releases of software needing quick deployment
• Minimal management intervention.

The typical enterprise Software Development Lifecyle (SDLC)

We see the enterprise as being either aligned or separate teams with clear sense of self-importance. Some organisations I worked in, we would think we were armies going to battle on a daily basis, other places there was a shared vision to deliver great solutions.

The mind-blowing occurrence of a different team taking an artefact, only to create their own version of it, to then send that onto another team who takes that artefact only to create their own version of it.

We write an application, add configuration to it, create a schedule, write release notes, create a runbook, work with testers, upon successful UAT, this is handed to the production release team who also want hand-holding, and even when hitting production - nobody really knows if it will be successful given the different data within the production environment.

Developer Operations promises to overcome this by providing tools and processes to reduce duplication and let teams reduce their process.

How Info Rhino solved our DevOps issues - for now

There are only two thing a developer should do;

1. See if there is some software that already does what they need.
2. Reluctantly write software to do what functionality does not exist.

There is always a balance between spending time souring and testing software, and writing it yourself. The reason I always go to code on places such as Github and Nuget is because they have normally thousands to hundreds of thousands of downloads with active communities, meaning they must have been tested to death. However, over these last few years, I kept finding external software doesn't exist. I am a reluctant developer, hoping to only write code when vital.

DevOps software I looked at and have experience with

• Bamboo, TeamCity, Jenkins, Octopus Deploy.
• Visual Studio MSBuild, Gzip, Nuget.
• Hangfire.IO, Quartz.Net.
• Microsoft ToDo, Trello, and many more.

You may say - they're not DevOps software, what about containerisation; Docker etc? Well, I looked at them too. I researched the cloud - to see what was on Azure.

My criteria for tooling successfully with DevOps

• Does the software lessen the learning curve? i.e. Reduce complexity?
• Does the software force your DevOps to tightly follow how the software wants you to work? An example of this is Atlassian's JIRA.
• How easy would it be to stop using the software? Is it portable? Can the information be used elsewhere?

When assessing software, they were not only expensive, they really wanted you to work how they wanted you to do it. There was definitely a steep learning curve, and I didn't feel - if I made the wrong choice, I could take that work and start again on a different DevOps architecture.

An obvious approach that sadly falls flat?

By now, you should see that I am predominantly a Microsoft Developer. Visual Studio provides fantastic tooling, and there are many extensions to pretty much let most of the building and deployment. I have worked with MSBuild and Nuget but this attempt to centralise just doesn't cut it when I may have many applications and architectures outside of the Microsoft IDE.

It has to be the case that we cannot rely upon Visual Studio - aside from anything else, the license cost is expensive per seat.

Focusing on the Job to be Done (JBDT)

I had two main types of applications; ETL, and Website deployments. Whilst the ETL can be largely instance based and dedicated - my web host is Shared Hosting, meaning unique settings per application. With the ETL, there was a need for Parallel processing - not just in process.

I needed something which was as close to the job being done without needing to manually work around this within the limitations of the software. An example is Control-M, having to set up parameters how they want you to do this. It is a given that setting up chains of job dependencies executing different applications, with different configuration is complicated. The question I had - can I avoid having to install lots of different software, all encapsulating their own settings and configuration?

Info Rhino Process Automation - what was built

I built the following applications;

• Job Processor.
• Zipper Application.
• Full Deployer application.

These applications are commandline, .Net/C# applications which does nothing much different to other DevOps software, except it doesn't have a user-interface (yet). However, it has a few great concepts which really helps simplify the deployment automation.

• Job Discovery - wildcard searches for executables.
• Configuration demotion - taking configuration from higher environments and saving it.
• Configuration translation and generation.
• Granular artefacts publication.
• Parallel processing.
• Readable instruction files (not YAML, but json).
• Job Status reporting.

The key focus, unlike most other software is to have most settings and progress in files which can be accessed by other applications. This makes the solution un-egotistical. Indeed, we don't care if people generate their own files, read our files, build their own versions on how our software works.

Where I think this software can help in the Enterprise

I will admit, Deployment Automation is a very complicated affair. Not only do we write software which may have many use cases and processing modes. We may need to string together many other services and applications. We may have different states, especially with relation to production versus test environments. I commonly find myself writing software in a development environment without data, with limited to know access to production data - it is like fighting with both hands tied behind your back.

So this software won't solve all the problems found in most artefact deployment scenarios. So what will IRPA solve?

Anybody can set up this software. Whilst it requires some knowledge, a small learning curve, it is nothing much more than setting up steps in a file or database, files which may need translating with different instance variables can simply be created.

A developer, tester, DBA, production release team member can all pick up the core concepts fairly quickly.

The solution is No-Code too.

Wrapping this up

I don't expect people to jump with joy at this software initially. However, it has reliably allowed me to do a huge amount in terms of deploying software updates, creating batches of data processing, generating environmental specific configuration, and minimising manual intervention without having to install multiple Enterprise DevOps tooling. As somebody who has been building entire platforms almost single-handedly, this software has really helped me manage my deployment automation.

If you are interested in helping see if this can be a truly amazing product, take a look under the beta section of my website.

https://www.inforhino.co.uk/beta

Written with StackEdit.