--- title: "Resiliency and Recovery" canonical: "https://help.callroute.com/space/CKB/773881857/Resiliency%20and%20Recovery" format: markdown --- Callroute is a multi-service, geographically dispersed platform designed to offer the highest standards of availability and resiliency to our customers. Our system is built to withstand several points of failure without detriment to customer services. In the unfortunate event that our systems become unusable and suffer a total loss of service, the following objective restore points are enforced. ## Recovery Point Objective Callroute takes regulary backups of its databases throughout the day at 4 hour increments. At the end of each day, the incremental backups are rolled up into a daily backup. Incremental backups are kept for 24 hours. Daily backups are kept for 14 days. In the unlikely event of a complete failure / corruption to our databases, the maximum amount of data loss is expected to be no more than 4 hours prior to the time of the last incremental backup. Our application code and system configurations are stored in GitHub with daily backups made to remote storage. Data that will be lost during this time will be: - Call Detail Records of any calls processed including recordings - Configuration assignments to users, numbers, services within Callroute that have been applied after the last working backup. (This will not affect Microsoft Teams configuration within your Microsoft tenant) - New service deployments made using Callroute since the last working backup and point of system failure - New customer tenants created on Callroute platform between the last working backup and the point of failure When service is restored, any affected configuration, services or Callroute tenants will need to be reapplied manually. Unfortunately, we will not have a record of these, so it will be up to each customer administrator to reconfigure as desired. > ℹ️ For Microsoft Teams customers, the outage will not affect configuration already applied to your Teams tenant between the last working backup and point of failure. Once recovered, customers will be able to resync their configuration to Callroute to restore calling service. It is important to note that our database backups are created to protect us from a total corruption and loss of service affecting all of our database servers. We cannot use these backups to restore individual customer data. ## Recovery Time Objective The recovery time objective for outages differs depending on the severity of the incident and how it has affected our systems. Incidents due to a software or configuration malfunction are usually resolved within 4 hours with only minor disruption to services. Major incidents that have corrupted our systems beyond repair and require rebuilding from backups will take up to 8 hours to restore primary services and 24 hours to restore full service. During such an incident, our primary objective is to restore the most essential services first to bring the majority of our customer’s primary services back online. This will include: - Callroute web portal - Microsoft Teams Calling & Webex Calling - Core Carrier Services Once restored, we will continue to restore services based on priority including - Bring your own carrier - SIP Trunks - SIP Phone > ℹ️ As mentioned, we have multiple servers in high availability mode. The majority of incidents will go unnoticed by customers. The above recovery would only be initiated in the unlikely event of a total loss to our systems