Cognitive Testing of a Brief Dietary Screener: A Comparison of Analysis Methods
Loading...
Date
2022-09-21
Authors
Williams, Tabitha
Journal Title
Journal ISSN
Volume Title
Publisher
University of Waterloo
Abstract
Background: Dietary assessment is crucial to understanding the dietary risk factors that contribute to noncommunicable disease outcomes. It is therefore essential to ensure that instruments to measure dietary intake are reasonably valid and reliable, and acceptable to both researchers and respondents. Following the 2019 release of the updated Canada’s Food Guide, Health Canada identified the need to develop a brief screener to assess overall alignment of adults’ eating patterns with the guide’s ‘food choices’ recommendations. The process of development included cognitive testing, a qualitative method used to investigate how respondents understand and interpret a survey or screener to ensure that each question can achieve its intended purpose. Though cognitive testing is common practice in the evaluation of data collection instruments, including dietary assessment tools, methods vary widely and are often not well-documented, particularly in the case of cognitive testing data analysis.
Objectives: The objectives of this thesis were to: (1) Develop and conduct cognitive testing of a brief screener, in English and French, to assess overall alignment of adults’ dietary intake with the Canada’s Food Guide-2019 healthy food choices guidance; (2) Compare two approaches to the analysis of data from cognitive interviews conducted in English, including an informal approach using brief notes, and an in-depth approach based on the framework analysis method applied to interview transcripts.
Methods and results: The first manuscript (Chapter 5) describes the development of the screener, including defining guiding principles, scanning existing screeners, and mapping the healthy food choices guidance to inform questions and response options. Once a draft screener was developed, it was iteratively refined based on three rounds of cognitive interviews in each of English (n=17) and French (n=16) to assess understanding of questions and face validity, along with face and content validity testing with a panel of experts (n=13 English, 3 French). Notes from cognitive interviews were coded and analyzed informally to identify issues with inclusion and exclusion criteria, readability, keywords, and response errors. The testing indicated that the screener was well-understood overall but informed refinements to improve comprehension of the questions and their alignment with the healthy food choices guidance. The resulting Canada’s Food Guide-2019 Healthy Food Choices Screener includes 16 questions to assess alignment of intake with the key components of the 2019 Canada’s Food Guide healthy food choices guidance, including healthy foods and foods to limit.
The second manuscript (Chapter 6) presents a subsequent comparative analysis of two approaches to analyzing the data from the cognitive interviews conducted in English (n=17). The initial informal coding of interview notes was compared to the in-depth application of the framework analysis method to interview transcripts, to identify differences in issues identified, as well as time and resources associated with each method. The informal method identified issues with inclusion criteria, keywords, readability, and response errors in the screener, including some issues that were not captured by the in-depth method. The in-depth method identified infrequent issues with comprehension, recall and judgment that were not identified by the informal approach, as well as additional instances of issues identified using the informal approach. The informal method required less time, but necessitated a note-taker, whereas the in-depth approach took more time and required transcription and coding software. Potential implications of the differences in the issues identified by the two data analyses approaches for the screener were determined to be minimal.
Conclusions: The Canada’s Food Guide-2019 Healthy Food Choices Screener allows for rapid assessment of the overall alignment of adults’ dietary intake with the healthy food choices guidance within Canada’s Food Guide. Appropriate use of the screener can inform policies and programs to narrow the gap between current dietary intake and the guidance. Cognitive testing was imperative to the screener’s development to ensure it can be understood consistently and in the way it is intended among the target population. The comparative analysis of cognitive interview data revealed trade-offs regarding the results identified and resources required for each method. The in-depth approach required more time and resources to conduct but reduced likelihood of misinterpretation. Comparatively, the informal method was quicker and simpler to conduct and identified most but not all issues with the screener, however, employment of an additional person is required for note-taking. Overall, the identified discrepancies between methods were unlikely to have meaningfully impacted the Canada’s Food Guide-2019 Healthy Food Choices Screener. However, this thesis contributes to a scarce body of literature on cognitive testing methods and will help to inform selection of data analysis methods for future development of dietary assessment and other survey tools.
Description
Keywords
cognitive testing, dietary assessment, Canada's Food Guide, screener, questionnaire, dietary guidance, diet quality, cognitive interviews, content validity, face validity, food choices, translational equivalency