Log files containing runtime information are a vital tool to assist root cause analysis following a recovery failure, however they can be extensive files containing unstructured information that requires manual filtering to find specific entries, and of course knowledge to determine which reported events are significant when determining any failure resolution.

Cristie Software are implementing the very latest compute efficient log parsing algorithms to transform raw log messages into structured log messages which can then be compared against a ‘known good’ recovery runtime outputs to train the VA in anomaly detection and categorization.

The recovery process and anomaly detection take place over several phases. The first step is the creation of a virtual environment in which to restore the system. The virtual environment is created according to a stored configuration and the system is restored from a backup or from a replica into the virtual environment. An attempt is then made to boot the restored system in the virtual environment with runtime logs collected during system startup.

The VA then parses the log files from the recovery simulation using machine learning rules based on known-good recoveries to identify any anomalies. The machine learning rules derived from the known-good runtime outputs train the VA on what log file entries to ignore so that only failed operations are collected for analysis. This training data is continually updated and reanalysed to ensure accuracy improves which each new software iteration.

The next phase involves categorization of any runtime anomalies found in order to create structured log groups containing matched log events. Log groups could include errors such as connectivity, network configuration, dependant systems and user permissions. As an example, one system may be a web application server, which will not work correctly without a database server (i.e., the web application server is a dependent system of the database server).

In the final phase the VA will determine any modifications required to the stored configuration of the recovery profile in the virtual test environment based on the outcome of the log file analysis and will attempt to make those changes automatically. The recovery simulation will then be repeated, and the resulting runtime logs will be parsed again to check for anomalies. This cycle will be repeated until the recovery simulation succeeds without errors in line with the domain knowledge provided by the baseline know-good runtime information. If the VA determines that manual intervention is necessary to resolve a recovery failure, then the ML algorithms will attempt to provide as much guidance as possible to the system administrator via the failure analysis panel within the VA user interface.