Evaluating Mission-Critical Self-Adaptive Software Systems: A Testing-Based Approach
dc.contributor.author | Li, Sen | |
dc.date.accessioned | 2010-11-24T21:15:47Z | |
dc.date.available | 2010-11-24T21:15:47Z | |
dc.date.issued | 2010-11-24T21:15:47Z | |
dc.date.submitted | 2010 | |
dc.description.abstract | Self-adaptive software is a closed-loop system that tries to manage, direct, or regulate its own behavior dynamically. Such a system aims at providing an automated and systematic approach to handling the increasing complexity of operation management. Mission-critical systems (e.g., e-business and telecommunication systems) are usually large, complex, and distributed. These systems must preserve their Quality of Service (QoS) at runtime under highly dynamic and non-deterministic conditions; therefore, they are suitable candidates for being equipped with self-adaptive capabilities. Although significant efforts have been devoted to modeling, designing, developing and deploying self-adaptive software since a decade ago, there is still a lack of well-established concrete processes for evaluating such systems. This dissertation proposes a systematic evaluation process for mission-critical self-adaptive software systems. The process is a well-defined testing approach that needs a post-mortem analysis, takes the quantified QoS requirements as inputs, and comprises two main phases: i) conducting system-level testing, and ii) evaluating QoS requirements satisfaction. The process uses Service Level Agreements (SLAs) as quantified QoS requirements, and consequently as the adaptation requirements of mission-critical systems. Adaptation requirements are specific types of requirements used to engineer self-adaptive software. Moreover, for the first phase, the dissertation discusses the uniqueness and necessity of conducting system-level load and stress testing on a self-adaptive software system, for collecting runtime QoS data. In the second phase, the process makes use of utility functions to generate a single value indicating the QoS satisfaction of the evaluated system. The dissertation mainly focuses on evaluating the performance, availability and reliability characteristics of QoS. An open source service-oriented Voice over IP (VoIP) application was selected as a case study. The VoIP application was transformed into a self-adaptive software system with various types of adaptation mechanisms. A set of empirical experiments was performed on the developed self-adaptive VoIP application, and the proposed process was adopted for evaluating the effectiveness of different adaptation mechanisms. To this end, the dissertation defines a sample SLA for the VoIP application, presents a report on the load and stress testing performed on the self-adaptive VoIP application, and presents a set of utility functions for evaluation. The experiments illustrate the validity, reliability, flexibility, and cost of the proposed evaluation process. In sum, this dissertation introduces a novel evaluation process for mission-critical self-adaptive software systems, and shows that the proposed process can help researchers to systematically evaluate their self-adaptive systems. | en |
dc.identifier.uri | http://hdl.handle.net/10012/5637 | |
dc.language.iso | en | en |
dc.pending | false | en |
dc.publisher | University of Waterloo | en |
dc.subject | Self-adaptive software | en |
dc.subject | Evaluation | en |
dc.subject.program | Electrical and Computer Engineering | en |
dc.title | Evaluating Mission-Critical Self-Adaptive Software Systems: A Testing-Based Approach | en |
dc.type | Master Thesis | en |
uws-etd.degree | Master of Applied Science | en |
uws-etd.degree.department | Electrical and Computer Engineering | en |
uws.peerReviewStatus | Unreviewed | en |
uws.scholarLevel | Graduate | en |
uws.typeOfResource | Text | en |