Automatic detection of software failures with hierarchical supervisors

Abstract

As the size and complexity of modern software systems grows, it becomes increasingly difficult to determine whether they operate as specified. Presently, the process is excessively dependent on human observation, limiting its scalability and accuracy. Accuracy and reliable detection of software failures would aid in the management and improvement of software reliability. An automated approach to detection of software failures is needed.

This thesis addresses software supervision, an approach to specification-based, automated detection of software failures. The work is focused on real-time reactive systems specified in a formalism based on communicating finite state machines. The supervisor, a separate unit, observes the inputs and outputs of a target software system. It makes use of the target systems' requirements specification. Discrepancies between specified and observed behaviors are reported as failures by the supervisor.

Supervision involves a number of difficult issues. A prominent one is the handling of specification nondeterminism. Specification nondeterminism permits the target system to generate several legal output behavioral alternatives for a single input behaviour. The supervisor must be able to consider all behavioral alternatives so that unwarranted failure reports are not generated. In some cases, the exhaustive consideration of all behavioral alternatives results in an excessive supervisor time and space cost.

This thesis presents a novel approach to supervision, called hierarchal supervision, that reduces the time and space cost of supervising systems whose specifications contain large amounts of nondeterminism. In a hierarhal supervisor, failure detection is carried out at two levels of abstraction: the path detection level and the base level. The path detection level determines the path or trajecgtory through the specification that corresponds with observed target system behavior. Effectively, at the path detection level, the behavioral alternative chosen by the target system is identified. At the base level, a detailed check of observed behavior along the path identified is made.

This thesis presents the underlying concepts of hierarchal supervision, the architecture of a hierarchal supervisor, the derivation of the supervisor model from the requirements specification, the definition of the interpreters for both the path detection and base supervisor levels and describes the derication of the time and space omplexities for both. The major research contributions of the thesis include splitting of supervision into two sub-problems (path detection and detailed behavior checking), making use of both target system input and output signals to track target system behavior, discussion of tradeoffs between the latency of failure detection vs the computational cost of supervision, development of an approach to prune behavioral alternatives from consideration and development of a base supervisor aimed at detailed behavior checking.

To evaluate hierarchical supervision, a demonstration supervisor was implemented. It supervised the control program of a small telephone exchange. Two key aspects, failure detection and time/space complexity, were evaluated.

The failure detection evaluation included both optimistic and pessimistic reporting. Pessimistic reporting refers to unwarranted generation of failure reports, while optimistic refers to not generating warranted failure reports. Experimental observations revealed that all failures were reported and no failures were missed. The time and space cost was evaluated by measuring the number of behavioral alternatives considered by the supervisor, which is indicative of its time and space cost. Experimental measurements showed improvements of over two orders of magnitude over the direct single-layer approach.