Extracting Non-Functional Requirements from Unstructured Text

Abstract

Non-functional requirements (NFRs) of a software system describe desired quality attributes rather than specific user-visible features; NFRs model stakeholder expectations about pervasive system properties such as performance, security, reliability, and usability. Failing to meet these expectations can result in systems that, while functionally complete, may lead to user dissatisfaction and ultimately to the failure of the product. While NFRs may be documented, tracked, and evaluated in a variety of ways during development, there is no single common approach to doing so.

In this work, we investigate extracting information about NFRs that may be contained in source code and source code comments, since they are often considered to be the ultimate source of ground truth about a software system. Specifically, we examine how often NFRs are mentioned explicitly or implicitly in source code comments by using natural language processing (NLP) techniques, and we evaluate how effectively they can be identified using machine learning (ML). We modeled the problem as a text classification problem in which the goal is to identify comments about NFRs, and we evaluated the classifiers using example systems from the electronic health records (EHR) domain. The best performance was achieved using SVM classifier, with an F1 measure of 0.86. Our results indicate that using supervised method for our problem outperforms unsupervised methods which try to find common NFR patterns in comments. Comparing our results to previous studies shows that NFRs can be extracted more accurately from source code comments compared to other software artifacts (e.g., SRS or RFP documents). Moreover, we found that bag-of-words features are more effective compared to more complicated features (i.e., doc2vec) for the problem of extracting NFRs from source code comments.