Product Based Software Support Engineering Team Design

When looking at legacy support systems consisting of horizontal tiers (1-4) it can be easy to see how these systems were adequate in the past but now fail to meet the needs of the current competitive support environments. Expectations are high from customers to get problems solved quickly regardless of the complexity you face in implementing that solution. It is time to evolve this strategy and utilize the same vertical approach as the product department in your organization to structure your support team.

Treat your software support engineers (SSE's) as production defect product owners: assign them roles parallel to product owners. Break their responsibilities down into manageable parts of your stack so they become specialists who: know a specific area of your product from A-Z, can speak to the most recent production issues, impact, and proactively look for problems in those areas.

Problems with Horizontal Legacy Support Tiers

The legacy approach to software support engineering stems from limiting employees responsibilities into siloed areas of expertise. Each tier has a narrow area of responsibility usually limited to an employees technical knowledge and skills related to directly interacting with external customers. This list details some failings with the support models for Tiers 2-3:

• Too many layers of people to efficiently conduct business. When a problem has to make its way through 4 teams of people there is no way you can quickly solve anything. Reducing the number of layers helps to remove this timing issue so you are counting resolution times for your customers in minutes to hours, not days to weeks.

• Lack of accountability across the multiple tiers. This stems from the "throw it over the wall" mentality when you have isolated tiers of support. People feel it is their job to validate and move the problem to the next level so a problem is out of their queue. They want to ensure SLAs are met for their team. The problem here is two fold and lies with: 1) lack of quality of triage as the next tier will be more easily able to solve, and 2) lack of accountability as there is always another tier of support to rely on or point fingers at.

• Support personnel can be spread too thin having to support problems from all product offerings. When everyone is free to help pickup any ticket from the queue and triage this can occur. While this can help reduce lead time, it means the engineer looking at the ticket is not always an expert or familiar with the issue. This can lead to the possibility of longer cycle times, incorrect triage, and escalating known issues.

Solution: Collapse the Roles into Smaller Cross Cutting Slices

Merge the roles of Tiers 2, 3, and product role of Tier 4 into vertical areas of responsibility that match the scope of new product owners. Take a piece of your service catalog which is currently managed by an individual product owner, and mirror that to a single SSE to own for production problems. The role should span from Tier 2 data collection, Tier 3 triage, and the product owner role of Tier 4. This solution removes all of the deficiencies listed in the section above. When considering this approach:

• Scope the areas of responsibility to match your product organization. If a new development product owner is able to manage the scope of their assigned software, a support engineer should be able to do the same. Give them a broad enough area of responsibility so they can utilize all of their allotted triage time in this role, and become a subject matter expert.

• Assign a scope large enough to be able to remember current issues in the backlog. Quick triage is aided when you have a familiarity with the current set of issues within your domain. You can manage scoping of responsibility up and down as needed based on performance and growth.

• Have a team lead or manager per application / service which combines the problems from their support engineers into a unified picture of the health and critical issues of each applications or service. This person can also lead team intelligence swarming efforts on newly written issues to ensure faster lead / cycle times.

Product Focused SSE Responsibilities

The duties of this type of SSE will vary from the standard roles as they need to focus more on full cycle fixes instead of only responsibilities from their previous layer in the tier. The career benefits to this approach also mean engineers directly work with product owners, developers, DevOps, and managers from other departments. This will help them to decide if they want to focus on one of these other areas as part of their future career path. Suggested areas of ownership:

• Proactive research into issues. The most important part of software support, which some support organizations fail to address, is looking for issues before customer contact. Support engineering is not just responding to customer issues, it should focus on proactively finding them. Some examples of how to accomplish this: if an engineer is UI focused, look for errors in your user session recording software. For data focused engineers: setup data monitors looking for issues in the data and alert when error conditions are met. API support engineers can monitor call logs for errors in how customers are requesting and receiving responses.

• Triage of assigned domain tickets. The meat of what an engineer has been doing, and should be an expert at is: triaging and attempting to solve problems. However, this time with a narrow focus they should be able to triage issues faster as they will be subject matter experts, and should have familiarity with all recent issues. Also participating in team intelligence swarming sessions can help others to understand their issues and solve faster.

• Shift left knowledge documentation and training. With a narrower focus on one aspect of the software, it will be easier to see trends where Tier 1 or customers need more help and information. The engineer should write documentation and do training so future similar issues can be solved with no contact to support. Schedule regular reviews of common issues and brainstorm how it would be best to reduce their occurrence.

• New release verification and authorization for domain areas. A single product focused SSE will be able to laser focus on new release functionality and understand how it will integrate and work with any current issues in the production environment. They should be able to pinpoint any inconsistencies or incompatibilities which might cause issues post release. A SSE will be the ideal owner to authorize a release into production.

• Determine impact of all known issues and regularly update. SSE's should also do regular reviews of the real time production impact of their issues. New software deployed to production, environmental changes, and new customers using the software in unexpected ways can cause the frequency of any given problem to fluctuate. While some releases may inadvertently solve some issues, other production changes may cause problems to get worse. Examples of what can be measured is: javascript or http errors in a web UI or rest API, exceptions being thrown by applications, or counts of corrupted data.

• Prioritize the top production issues for development. With current impact of issues regularly reviewed, the logical next step is to provide development with the top list of issues to prioritize into each sprint. Ensure the most relevant customer impacting issues are prioritized in each development cycle using data driven reasoning. Define what the criteria is to measure the priority of any customer facing issue to assist the team with these decisions. Depending on your business model examples of this might be: most impacted customers, larger customers impacted, monetary impact to the company, number of support calls, time needed to manually fix.

• Implement solutions. Depending on the technical expertise of the engineer, they may be able to do the technical work necessary to solve the issue. If this is the case it also helps out development with not having to bear all the burden of production issue fixes. This also gives the SSE more hands on experience in a potential future career field.

In summary: narrowing the focus of your support engineers into a more manageable slice will allow them to approach problems as full process experts. They have to hold themselves accountable for the success or failure in solving issues in their scope. They should be able to see issues coming in as customers experience them and be able to gauge the impact of these issues. As a product focused support engineer they will be able to know which issues should be solved first to ensure the best possible experience for customers.