Effectiveness of AI Use in Qualitative Abstract Coding

Purpose: As the field of qualitative research continues to evolve, the integration of artificial intelligence (AI) methods into data analysis has garnered increasing attention. This research paper presents a comparative analysis that evaluates the effectiveness, advantages, and disadvantages of utilizing AI methods for qualitative analysis of innovation characteristics of grant abstracts from the NIH RePORTER database, compared to traditional manual approaches. By comparing AI results to manual coding results, this project explores the potential transformative impact of AI on qualitative research.

Methods: As part of the NIH HEAL initiative, researchers manually coded over 700 HEAL abstracts to examine innovation characteristics of grants funded. Survey questions included the type of study, study innovation, primary goals, and expected outputs. To explore whether these abstracts, when prompted in ChatGPT v4.0 would yield the same results as a human coder, we will input 100 of the 700 coded abstracts into ChatGPT, entering the same instructions given to the human coders. A comparative analysis will then assess and score the output in terms of detail, depth, relevance, and completeness of the responses.

Results: We expect there to be no major differences between human and ChatGPT coding. This would suggest that AI can produce valid qualitative coding outcomes, potentially streamlining some research processes that involve abstracting themes from textual data and opening new possibilities for large-scale qualitative analysis.

Conclusions: Ultimately, the findings of this study have the potential to shape the direction of qualitative research, offering valuable guidance to researchers and practitioners on the feasibility and reliability of integrating AI methods into their work. This research will contribute to the ongoing discourse on the role of AI in qualitative research and provide a foundation for future investigations in this rapidly evolving field.