What is an incident¶
An incident is an unplanned interruption to the hub service, or reduction in the quality of it. It is an external representation of an outage. The definition of an outage and the outage categories are tracked in more detail in the Outages documentation.
When is the incident response process triggered¶
Whenever an outage has been identified, the incident response process should be triggered by the person who identified the outage.
If they are part of the Infrastructure Engineering team, they become the Incident Responder, otherwise they should notify the Infrastructure Engineering team via the #pagerduty-notifications Slack channel or the #product-and-services-team channel.
The incident team¶
A team is formed when the incident response process is triggered. The goal of the team is to collectively and quickly resolve the incident.
The team is currently formed organically with the the available members of the Infrastructure Engineering team, and may include other members of the 2i2c team as-needed. A rough way to approximate this team is “the people that have communicated in internal and external channels to resolve an incident.”
This team needs to fulfill three roles, which ideally are filled by different people, but can be same person.
One or more Subject Matter Experts
Roles and responsibilities¶
- Incident Responder
- The Incident Responder has the authority to plan and delegate action to others on the team.
Their responsibility is to:
solve or temporally mitigate the incident, either themselves or with the help of the available team members
be the Source of Truth about the current state and action plan surrounding the incident
review the Incident Report after the incident is resolved
- Communication Liaison
- The person that communicates with external stakeholders about an Incident. This is typically the Support Triager
Their responsibility is to:
communicate the status of the incident
send updates about our current thinking and what we have tried, and any expected changes coming
write the Incident Report after the incident is resolved
- Subject Matter Experts
- A team member with expertise in an area of relevance to an Incident. They should be pulled in to the Response Team as-needed by the Incident Responder.
Their responsibility is to:
help resolve the incident by providing expertise and taking actions
inform the Incident Responder about their plan/actions/motivations either during or after the incident so they can take actions as-directed by the Incident Responder
the Tech Lead must always be included as a Subject Matter Expert, as they have unique expertise in solving certain problems and are ultimately accountable for successful incident resolution.
- Subject Matter Expert
- See Subject Matter Experts.
Accountability¶
Each of the roles is accountable for their responsibilities, and the Technology Lead and the Engineering Manager are ultimately accountable for the success of the incident response and how well the process has been followed.