Lean Support through Better Tooling for Tiers 2 / 3

We all want to minimize costs of support, how can you make your support structures as lean as possible in a Tier 2 or 3 team? New tools are coming out every day which empower employees to have greater visibility into data and system stability and reduce the overall costs and time needed to support systems. You no longer have to be a nerd to get the technical information you need to solve a customer issue. Those same tools empower Tier 2 teams to have the same visibility as engineering / Tier 3 teams have.

Giving your Tier 2 and 3 teams the tools they need to give easier and quick visibility into system issues, status, actions and logs allows for faster triage times.

Focus Areas

---

Lets start by going over areas which are commonly looked at in the triage process and how enabling better visibility via tooling into those areas might decrease the amount of time support teams need to spend troubleshooting.

Logs, both file and database based. Logs are one of the critical components of debugging as they enable you to see (usually) the exact locations of errors applications are having. With this information determining the actual problem can be fast once you know the lines of code having the issue.

Monitoring and alerting for servers and running services. Its important to know the state of your applications and if they are up or down before your customers contact you. Also you need to know what errors are occurring or if backend services are slow or crashed.

User session visibility. Somewhat of a newer technology: being able to see the actions users are taking in your system provides exact information about what the user is doing and how they are interacting with your applications. This powerful technology means you don't need to ask the customer to reproduce as you are able to see their sessions and provide the exact error information to developers.

Email activity sent by applications to customers. Common issues with emailing is the complaint of I did not get the email or the content of the email was wrong. With these tools you enable support teams to see if the email made it past the customers spam filter and if they opened the email.

Performance and uptime metrics. Are there certain times when applications go down or are under load and responding poorly? There might be environmental things going on causing the issue from a variety of sources.

Current Toolsets

---

Here are some current toolsets which give the kind of visibility needed in the above aspects. Each has its own advantages and uses. In depth analysis of these toolsets will be done via individual articles and then linked below.

• Log exception analytics: Sentry, Rollbar, Rangun, Airbrake, BugSnag

• Log aggregators: Loggly, PaperTrail, Kafka, Raygun, Snare

• Application and system monitoring: Dynatrace, New Relic, SolarWinds, Cloudwatch, AppDynamics

• Performance and uptime reporting: Pingdom, DataDog, PRTG, ConnectWise

• User session recordings: LogRocket, SmartLook, Hotjar, Crazy Egg

• Email logging: SendGrid, SendinBlue, Amazon SES

Tier 3 Use of Tooling

---

Tier 3 is expected to make full use of the above tooling. Their understanding of the technology and applications should be sufficient enough to know how to debug an issue. They should know which tool provides which information and be able to directly access without having to logging onto physical servers and needing to know where the information is located.

Tier 3 / DevOps teams can also use the tools listed here to be able to proactively find issues before customers report them. If a quick fix is possible, you may be able to avoid that costly customer call and the expense of your support team handling the issue.

Empowering Tier 2

---

Luckily many of these tools are simple enough to use, they can also be utilized by Tier 2 teams. From my experience, many of the escalations Tier 2 had raised in the past was due to limited access to be able to effectively see the problem. With this impediment removed they can do more troubleshooting for you.

In effect the more information Tier 2 can now provide actually enables Tier 3 to be bypassed in certain circumstances and developers can be directly involved. As an example: when a user has an error in the web UI application, the Tier 2 specialist can see the error in the application from their UI recorded session, provide the url of that exact moment in the recording, and put in a ticket. The engineer can use the same tool, see exactly how the error was created, get the technical information they need about the error. You have now effectively bridged the gap in the teams using 1 tool, pretty cool.