I Tried (and Failed) to Show Up AI for Writing an Article Summary

A few weeks ago I had Microsoft CoPilot write me a literature review on the topic of how general education programs could benefit from students’ perspective. I had already written a sample literature review, and I wanted to compare the work by my own two hands to that of generative AI. I was disappointed in what CoPilot produced, but mostly because I didn’t learn anything during the process (of outsourcing the research and writing to AI). I actually liked its proposed structure, but I thought the analysis of articles was weak.

The Design

I decided to do a deeper dive with AI and an article analysis. I picked one article in particular, because I thought it would be more helpful than the others. That article was “Engaging Students as Partners: Student Assessment and Research Advisors for General Education” by Carpenter, Burgess, and Thorsrud (2025). The description CoPilot came up with was as follows. I deleted the unnecessary clutter.

A particularly influential example of this approach is provided by Rowanna Carpenter, Emily Burgess, and Eryn Thorsrud, who described the development of a Student Assessment and Research Advisory Committee within a general education program. The committee involved students directly in interpreting assessment findings and informing assessment-related decisions. The authors reported that student involvement helped bridge communication gaps between students and administrators while generating insights that faculty and staff might otherwise overlook. Participants concluded that student partnership improved both assessment practices and program effectiveness. [jstor.org]

I criticized this summary for being vague. I wanted to know, for example what sorts of insights students provided and how the gen ed process changed as a result. According to this summary, CoPilot isn’t sure about the answers to these questions. I asked CoPilot to give a better summary of the article itself, but it apologized and said what amounted to “I could only read the abstract.”

 

I tried (and failed) to track the article down myself, so I sent an email to one of the authors. I got a PDF of the article within one week. I fed this full article into CoPilot and asked about the insights students provided to the gen ed committee. I got in return a four-bullet essay summarizing the ways in which student participation materialized in concrete changes. The breakdown was just a long-winded version of the summary, which came at the end:

The most important contribution students made was helping the committee see assessment through the lens of students' lived experiences. They pushed the program to ask different questions, gather richer information about student needs, improve the usefulness of reports, connect assessment with student support services, and include a broader range of student voices in decision-making. These insights led to immediate changes in assessment practices and helped the program develop a more student-centered approach to general education assessment.

Reading this, it is clear that CoPilot did a better job summarizing the article when I provided it with a copy. Let’s look at them side-by-side:

 

CoPilot without seeing the article

CoPilot seeing the article

The committee involved students directly in interpreting assessment findings and informing assessment-related decisions. The authors reported that student involvement helped bridge communication gaps between students and administrators while generating insights that faculty and staff might otherwise overlook. Participants concluded that student partnership improved both assessment practices and program effectiveness. [jstor.org]

The most important contribution students made was helping the committee see assessment through the lens of students' lived experiences. They pushed the program to ask different questions, gather richer information about student needs, improve the usefulness of reports, connect assessment with student support services, and include a broader range of student voices in decision-making. These insights led to immediate changes in assessment practices and helped the program develop a more student-centered approach to general education assessment.

 

And in Bullet Form:

 

Without article with Unanswered questions.

With Article with Unanswered questions

·      Informed assessment-related decisions. How? What is an example of one decision that changed? ·      Helped bridge communication gaps between students and admin. What were those gaps? What did the bridges look like and what did they accomplish? ·      Improved assessment practices and program effectiveness. What were the assessment practices that were improved? How were they improved?

·      Students helped gen ed committee see assessment through the lens of student experience. For example, …? ·      Helped committee as different questions and gather richer information about student needs. What were some of those questions? What richer information was collected? ·      Helped committee improve the usefulness of the reports. For example,  …? ·      Helped committee connect assessment with student support services. How did that connection come about? ·      Increased students’ voice ·      Process led to changes. Changes, such as…?

 

Summary of Results

The level of detail was much greater when the article was provided. But note how many outstanding questions I still had for the authors.

Comparing AI-Generated Article Summary with My Own Summary 

I read the article myself, and honestly I could not have done better than the summary on the right (the CoPilot-generated summary when the article was provided). Indeed, I gave up about halfway through the article when it was clear that the authors didn’t disclose any concrete examples. Perhaps this was motivated by a lack of IRB-review, but it resulted in a bunch of vague examples and suggestions. I still have all of the same questions after reading the article than I did after reading the AI-generated summary when the article was provided.

 

Perhaps this is a unique case in which the article was vague to begin with. The authors talk about a lot of changes that happened without actually describing even one of them. All I learn from the article is that I might expect a bridge to form between student experience/perspective/voice and gen ed assessment process. But I have no idea what that bridge would bring with it other than a bunch of useful stuff.

Surprising Conclusion

When it comes to summarizing an article (at least in this case), the LLM did a better job than I could or would. But to this I would counter: Why would a more detailed summary of an article be preferable when the article itself is poor? 

This leads me to a topic I hadn’t considered: LLMs are incapable of critiquing an article when the LLM isn’t given any instructions. E.g., it would be unable to do a blind review for a journal. However, I suspect CoPilot could critique an article if I asked it to. But then I think of all the potential problems the reviewer has to have in mind and I’m doubtful that the LLM could keep all of track of all of those concerns. I pressed CoPilot about whether this article did what I accused it of not doing, and CoPilot thought the article was swell. One of the examples CoPilot gave surprised me, so I reread that section. CoPilot wrote, “Rowanna notes that student advisors helped identify topics that were absent from existing surveys, including: student mental health, time commitments, hopes and goals for the academic year.

This surprised me, so I reread that section. The actual words from the article are, “The advisors helped me identify topics that were missing from the survey and then think about how to frame questions about student mental health, … [etc]” (Carpenter et al., 2025, p. 266). I think CoPilot misread that sentence. Students helped reframe these questions (although we don’t know how), but we don’t know which questions students suggested be added. It really comes down to an ambiguous sentence with two verbs: "identify topics" and "frame questions.” Which of these was related to mental health? Time commitments? And so on. Again, an ambiguity that is missed by CoPilot.