Incident Response Planning for MSPs

The 6 steps to create an incident response plan

According to SANS Institute, there are six steps to create an IRP: 

1. Preparation

It determines policies, response plan, communication, team members, access control, tools and training.    

Policies: A written, documented set of principles, rules or practices that serve as a guide as to whether an incident has occurred in an organization. Clear policies help educate users to authorized/unauthorized actions and protect an organization from lawsuits and other legal action.  

Response plan: A strategy to handle incidents is needed. Prioritization of incidents should be based upon their organizational impact. Prioritization helps build the case to garner management buy-in and ensure the necessary resources are being devoted to incident response preparation.  

Communication plan: Communication is a vital aspect of IRP since it may be necessary to contact specific individuals (clients and their employees) during an incident. The plan should document who to contact, when it’s appropriate to contact them, how to contact them and why. Having an effective communication strategy helps minimize response time.

Team: The IR team is made up of several individuals from different disciplines to manage various problems that could arise during or from an incident. These may include technicians from your MSP organization and human resources, public relations, communications professionals and legal from the client organization. 

Access control & tools: Ensure the IR team has the appropriate resources, tools and permissions to perform their roles (these may be both digital and physical) and includes checklists, manuals and other such documentation.  

Training: It’s recommended to conduct drills at regular intervals to ensure each individual within the IR team is able and prepared to perform their duties during an incident. 

2. Identification

This step involves the processes through which incidents are detected and identified, as quickly as possible, for commencing remediation efforts. The IR team may use various sources, such as log files, error messages, intrusion detection systems, firewalls or other tools, to determine whether an event that’s taken place qualifies as an incident that must be responded to. Events deemed to be incidents must be reported as quickly as possible.   

• Monitoring may include the use of various tools to monitor client logs, networks, disks, firewalls, coupled with a SIEM (Security Information and Event Management) solution.   

• MSPs should aim to automate various IR tasks wherever possible. This may include notifications that help the right owners get notified at different stages of an incident as required, as well as keeping constant contact with end users and stakeholders until the incident is resolved.  

• Use templates to streamline communication efforts with clients across all stages of an incident. 

3. Containment

The goal of this phase is to limit the damage and prevent any further damage from happening.   

Short-term containment: Limit the damage quickly. It may involve isolating a network segment of infected machines or taking down production servers that have been hacked, and rerouting traffic to a failover host. Short-term containment is intended to limit the incident before it causes real damage.  

System backup: Before wiping and reimaging any system, it’s critical to take a forensic image of the client’s affected system(s) as it was during the incident, and preserve evidence in the event the incident results from criminal behavior. A backup strategy must be in place to ensure data remains accessible to clients and is safe as well. 

Long-term containment: Temporarily fix impacted systems to be used in production while rebuilding clean systems. 

4. Eradication

Eradication means actual removal and restoration of affected systems. Proper steps are taken to remove malicious or illicit content from affected systems. 

This phase is also where organizations should examine and evaluate their defense for improvement after learning what caused the incident so as to ensure the system cannot be compromised again.   

5. Recovery

Test, monitor and validate client systems that are being put back into production to verify they are not reinfected by malware or compromised by other means.  

Here are a few things to consider in the recovery phase:  

• Time and date to restore operations: System operators/owners (technicians) will make the final decision to move into production based on the guidance and advice of the IR team (client side) 

• Test and verify compromised systems 

• Duration of monitoring to observe for abnormal behaviors  

• Tools needed to test, monitor and validate system behavior 

6. Lessons learned

The last phase helps to educate and improve future incident response efforts. MSPs should take the opportunity to update incident response documentation with information that may have been missing, omitted or incomplete before the incident, plus complete documentation of remediation efforts to provide insight to clients. This gives a clear view of the entire incident, and clients may instill it as part of their security training or as benchmarks for future comparison.