The Biggest Drawback to Using AI to Write Your Literature Review

For my behavioral research class, I decided to use Microsoft CoPilot to help me write a literature review on a topic that genuinely interests me—how to design a gen ed assessment strategy that uses students as sincere collaborators. I had already written a 1500-word lit review on the topic, so this was like a comparison test. I wanted to know what generative AI did well and poorly so that I could better understand how it was helping/hurting students.

My biggest takeaway (and the biggest caution I can give) is that I don’t feel any different after having “written” the lit review. My perspective hasn't changed, I don’t feel any more well informed about the topic, and I don’t feel better prepared to take action as gen ed coordinator. It was an exercise in triviality—sort of like listening to French while you sleep.

[Below I have a video of the process and my thoughts about the process as it happened.]

What AI Did Well:

1. AI understood what I wanted

This was a surprise. As I was making lunch and thinking about how this would go, I expected to have to change my topic to align with whatever AI thought I wanted. But CoPilot recognized my topic and congratulated me on it. My topic was how to involve students as collaborators in gen ed assessment and course design.

2. AI Gave Me Bad Examples

At the brainstorming stage, there aren’t any bad examples. Bad examples help spark creative insights. I quickly rejected four of five recommendations and sort of liked number five. But the process of rejecting ideas helped me better understand what I wanted.

[In retrospect, however, I never got to develop my specific idea. I went with one that was "good enough.” While DW Winnicott thought this was fine for parenting, I’m not sure “good enough” is the path I want to take for becoming an excellent educator or scholar.]

3. AI Suggested Six Excellent Articles

When I asked, CoPilot provided six peer-reviewed articles and rationale for why I might use them. At least three were examples of exactly what I wanted to learn about: colleges that had thought to include students in the gen ed assessment process and how published the results. I tried finding such examples myself for 30 minutes and came up with nothing. AI did it in seconds.

[AI could not, however, provide me with the articles directly. The one I really wanted was behind a paywall and I couldn’t even find it through my university. I plan on tracking it down and reading it. Then I guess I can provide a side-by-side analysis of what AI came up with and what I came up with.]

Things AI did Poorly

1. AI Couldn’t Read the Articles It Told Me About

Perhaps its obvious, but since AI couldn’t share the articles with me it also couldn’t read the articles it was telling me about. Consequently, I learned more in my imagination than I did from the article discussions in the lit review it wrote for me. For instance, take a gander at this article description:

A particularly influential example of this approach is provided by Rowanna Carpenter, Emily Burgess, and Eryn Thorsrud, who described the development of a Student Assessment and Research Advisory Committee within a general education program. The committee involved students directly in interpreting assessment findings and informing assessment-related decisions. The authors reported that student involvement helped bridge communication gaps between students and administrators while generating insights that faculty and staff might otherwise overlook. Participants concluded that student partnership improved both assessment practices and program effectiveness

Note in particular how the authors “generat[ed] insights that faculty and staff might otherwise overlook”. I’ll describe this shortcoming in my next thing AI did poorly, which is…

2. AI Gave Weak Summaries of the Articles it Referenced

Going back to the AI excerpt: did students yield insights that faculty and staff had overlooked? It isn’t clear by the sentence if students contributed anything with their participation. This sounds to me like the AI is saying, “Students might have been more valuable than nothing at all.” It’s the word “might” that weakens everything. The AI doesn’t know. It’s guessing.

But imagine that student participation DID yield insights that teachers missed. Well, then tell me what those insights were! Help me understand the benefits of including students.

3. I don’t Feel any Smarter or Wiser for having Outsourced the Process to AI

It wasn’t until graduate school that I realized the importance and joy of writing. Before graduate school, writing was just a game that was all about information dumping and organizing it on the page. In graduate school, I learned that writing was the process of clarifying my understanding. A literature review, then was the process of clarifying my understanding of a topic while benefitting from the insights of others. The lit review was a learning process. I always had a different—a better, more well-reasoned and defensible— perspective after having written one. I might try to combine two ideas, for example, such as Goldstein’s concept of “self-actualization” and the Estonian biological theory of bio-semiotics. I imagined how they fit together, but then I would go to the thinkers themselves and try to fit them together. I would rub up against their thoughts—getting grease on my fingers and so on. Ideas change when you try to mix them together. You realize that you didn’t know them that well to begin with. By spending time understanding each thinker—each resource, each article—you change. The new insights are added to your ever-changing perspective. You come out the end of a lit review as a different person.

Outsourcing the process to AI shortchanged me on all that learning. I expect to forget everything I read about in one hour. I can say that confidently, because I recorded the process about 45 minutes ago, and I don’t remember what it was about other than the topics it gave me.