True Cost of Manual Operations

SAAS companies around the world function in some part from people doing manual repetitive jobs each day to ensure the continued operation of the business. It is difficult if not impossible to have a fully automated software stack. What can we do is drive down the known operational costs we have by visualizing what these efforts are costing us and prioritizing their automation or fixes.

Human reliant software is fragile and should be avoided. Reverse engineer your manual repetitive operations task checklist into development tickets and calculate their costs over time to be able to effectively prioritize.

The Concept: Every Task is a Bug

If you have a repetitive task which you must do on regular intervals to ensure operation of software, that is a bug and should be tracked as a defect against that component. Follow this exercise to learn more about how this arose. Consider the company cultural factors which also played a part in this.

Why are you doing this? What does this task actually accomplish?
What led to this task? What piece of new software, deployment or process change required this to now happen regularly?
How did it become ok to have this task? This is more of a cultural paradigm shift to start realizing and documenting anything which becomes manually repetitive.
What is the importance of this task? Must it be done to keep the company functional? This can help you to prioritize the fix.
Which part of the software did this task come from? Are you seeing trends where one part of the software is more problematic than others?
What development planning should be changed to avoid in the future? How can you influence the development process to avoid coding things which necessitate this in the future?

By answering these questions hopefully you can see why any types of issues like these are defects. Consider yourself an operational QA, attempting to remove the bug of humans being part of the required functionality of your software. Also think to the future and things you can learn to prevent this from happening again.

Reverse Engineered List

To continue this process, it is time to start tracking down each of these tasks from their owners in the company. From your own knowledge you can probably list many of them off the top of your head, others you know who to ask to get these lists. As time goes on you will continue to stumble across them and document as you find. I have seen these types of items tracked via:

Spreadsheets
Calendars
Sticky notes
Notebooks
Repetitive tickets entered with the same words

From each of the items you find, create a defect or bug in your development ticketing system.

Data to Capture

The data you will need for these defects is going to be different from any other type of issue you likely have right now. Be sure you have the ability in your ticketing system to add custom fields, as you are going to need to use them for adequate reporting. You will also need to be able to have fields which can be calculated based on the values of other fields. Some ticketing systems have this built in, for others it is a plugin or add-on. Here are the data points I would suggest:

Issue details tracking:

Frequency: days of month to run
Time(s) run per day (optional)
Area or component
Owner
Description
Workaround detailed instructions (link?)

Issue numbers:

When did this start?
Minutes spent per instance
Financial cost if any per workaround not including time spent

Calculated Fields

From the data entered above, once the ticket is created, it should start to auto calculate these values. This will be used later for reporting.

Number of instances overall = count (days of month to run) x times per day x months since start date
Overall cost = (minutes per instance x number of instances overall / 60 x average hourly cost per employee) + (financial costs * number of instances overall)
Number of instances in last quarter = (count (days of month to run) x times per day) / 4 x weeks since start of quarter
Last Quarter Cost = (minutes per instance x number of instances in last quarter / 60 x average hourly cost per employee) + (financial costs x number of instances in last quarter)
Quarterly forecasted future cost = ((count (days of month to run) x times per day x 3) / 60 x average hourly cost per employee) + (financial costs x (count (days of month to run) x times per day x 3)

Dynamic Data for Reporting

With all of the data captured into the bug / issue tracking system this can now be used for 2 purposes for all parties involved:

Reverse engineered daily checklist: you can now take the data you have from your bug tracking system and make a complete checklist from those defects. This is quite powerful as additions to this list now must be additional defects entered. Also when problems are solved, those items automatically disappear off of the checklist. This is easier to now give to a NOC team and have them follow, allowing other operations employees to be able to focus on more DevOps related efforts. Fields:

Time(s) to run
Component
Description
Instructions link

Numbers report: from the management perspective you can now use this same data to present costs per issue and rank. This report will dynamically update itself each time you run it as time goes on and people keep running these manually. Until the defect is closed, the report will assume these are being done regularly and keep adding time / cost to them. Fields:

Component
Description
Overall cost
Quarterly cost
Forecasted Quarterly Cost

Some additional reports to consider:

Applications with the highest number of manual workarounds.
Areas where new manual tasks are being added.
Total financial impact of these items per quarter or year.

Developers report: an added benefit of this approach is the developers can see the applications and workarounds from the data you have gathered. The description of the workaround followed by the NOC team should give them enough information to be able to come up with a solution from the data you have already provided without requesting additional information.

Instill a New Culture

With this new wealth of data gathered, analyzed and calculated show it off! Not just to the people making decisions, but to everyone. Get the word out that you are tracking this type of work. Most people should welcome the transparency into the amount of effort put on these tasks. It will be easy to then justify the costs of fixing or automating these issues. Also this makes it easier for development to weigh estimated gains from new development and compare to the costs of these manual tasks to see what is more financially advantageous.