Methods and models for the quantitative analysis of crowd brainstorming

Abstract

Microtask marketplaces provide shortcuts for automating tasks that are otherwise intractable for computers. Creative tasks fall squarely within this definition, and microtask marketplaces have been heavily leveraged to this end. Brainstorming is often an implicit component of these solutions. This thesis provides the first foundational study of brainstorming in microtask marketplaces, aimed at solving the open problems in brainstorming task design to make this process more accessible and effective. This is achieved by establishing techniques for coding brainstorming data at scale, models for quantifying desirable outcomes of brainstorming, and a qualitative deconstruction of brainstorming strategies employed in this environment.

Idea forests are introduced as a data structure to enable the disambiguation of ideas in large corpuses, providing natural measures of two metrics of primary interest in brainstorming research: quantity and novelty. They are constructed via a tree-traversal algorithm, restricting the subset of the corpus which the coder must be aware of when making decisions. A simulation approach is introduced to assess the validity of hypothesis outcomes derived from idea forest metrics.

The introduction of idea forests enables the core contribution of this thesis, a set of quantitative models for brainstorming outcomes. This thesis extracts several actionable conclusions from the parameters of these models: the rate of unique idea generation is subject to decay over time; individuals have a significant effect on the rate of idea generation, with productive workers generating dozens more unique ideas; and individuals generate their most novel ideas late in a brainstorming session, after the first 18 responses. Furthermore, a replication of findings by Nijstad and Stroebe is conducted, finding that workers take more time to generate ideas when changing semantic categories and are more likely to remain within a category than expected by chance.

Finally, a taxonomy of strategies employed by brainstormers is presented. In particular, this thesis discusses the phenomena of scoping brainstorming problems, providing partial solutions, and riffing on previous solutions.