Measuring the Performance of Code Produced with GitHub Copilot

Abstract

GitHub Copilot is an artificially intelligent programming assistant used by many developers. While a few studies have evaluated the security risks of using Copilot, there has not been any study to show if it aids developers in producing code with better performance. We evaluate the performance of code produced when developers use GitHub Copilot versus when they do not. To this end, we conducted a user study with 32 participants where each participant solved two C++ programming problems, one with Copilot and the other without it and measured the running time of the participants' solutions on our test data. Our results suggest that using Copilot can produce code with a significantly slower running time.