Software Engineering for Big Data Systems
Loading...
Date
2017-04-24
Authors
Kumar, Vijay Dipti
Journal Title
Journal ISSN
Volume Title
Publisher
University of Waterloo
Abstract
Software engineering is the application of a systematic approach to designing, operating and maintaining software systems and the study of all the activities involved in achieving the same. The software engineering discipline and research into software systems flourished with the advent of computers and the technological revolution ushered in by the World Wide Web and the Internet. Software systems have grown dramatically to the point of becoming ubiquitous. They have a significant impact on the global economy and on how we interact and communicate with each other and with computers using software in our daily lives.
However, there have been major changes in the type of software systems developed over the years. In the past decade owing to breakthrough advancements in cloud and mobile computing technologies, unprecedented volumes of hitherto inaccessible data, referred to as big data, has become available to technology companies and business organizations farsighted and discerning enough to use it to create new products, and services generating astounding profits. The advent of big data and software systems utilizing big data has presented a new sphere of growth for the software engineering discipline. Researchers, entrepreneurs and major corporations are all looking into big data systems to extract the maximum value from data available to them. Software engineering for big data systems is an emergent field that is starting to witness a lot of important research activity.
This thesis investigates the application of software engineering knowledge areas and standard practices, established over the years by the software engineering research community, into developing big data systems by:
- surveying the existing software engineering literature on applying software engineering principles into developing and supporting big data systems;
- identifying the fields of application for big data systems;
- investigating the software engineering knowledge areas that have seen research related
to big data systems;
- revealing the gaps in the knowledge areas that require more focus for big data systems
development; and
- determining the open research challenges in each software engineering knowledge area
that need to be met.
The analysis and results obtained from this thesis reveal that recent advances made in
distributed computing, non-relational databases, and machine learning applications have
lured the software engineering research and business communities primarily into focusing
on system design and architecture of big data systems. Despite the instrumental role
played by big data systems in the success of several businesses organizations and technology
companies by transforming them into market leaders, developing and maintaining stable,
robust, and scalable big data systems is still a distant milestone. This can be attributed
to the paucity of much deserved research attention into more fundamental and equally important
software engineering activities like requirements engineering, testing, and creating
good quality assurance practices for big data systems.
Description
Keywords
software engineering, big data, software engineering knowledge areas, SWEBOK