Supplementing and Enhancing Information Literacy using AI

In what follows I describe the learning outcome of information literacy. I explain what I have found AI to do well or poorly in red.

I asked my counseling psychologist friend the research-related skills she relied on most in her counseling practice. The number one skill was what she called “knowing how to evaluate a study,” which she described as looking at who wrote it, where it was published, and whether it was any good. The last part (“Was it any good?”) is applied scientific reasoning. The other two belong to the skill of information literacy.

The three information literacy skills I will focus on below are accessing the needed information, evaluating information and its sources critically, and using the information to accomplish a purpose. I define these based on the IL rubric created by Association of American Colleges and Universities.

Accessing the Needed Information

By the time students reach upper division college courses or graduate school, they probably already have a technique for tracking down articles. When I’m researching a topic for my next book, for example, I begin by looking for other books written on or related to my topic. I do this by searching Amazon by keyword. There is a whole process I use for books and then for peer-reviewed articles, which begins with looking at where and when it was published. This helps me determine its credibility and relevance today.

LLMs are great when it comes to finding books, articles, chapters, and blog posts related to any keywords you can think to give it. I personally don’t use this method, because LLMs necessarily follow the most well-trodden path. I like to think that I approach topics and problems differently than most people. Why else would I want to go through the process of thinking about something if the result will be redundant. After all, if 1,000 people have already had the insight I’m working towards, then I feel like I’m wasting my time. Not to mention the fact that I could easily be replaced by LLMs or by someone who is using an LLM themselves if that’s all I’m doing.

Evaluating Information and Sources

To evaluate information, you have to ask questions such as, “Does this information represent a single perspective (weak) or multiple perspectives (strong)?” “Is the information relevant for my topic or research question?” and “Is the information current?”

Scientific fields are under constant transformation. Entire schools of thought can be supplanted (replaced) by new ones. In 1980, for instance, a significant transformation happened in psychological science: the psychodynamic theory of mental disorders and treatment was replaced by the biological theory. In one year, the explanation for why someone was suffering changed radically from being caused by unseen drives in a person (psychodynamic theory) to neurological activity (biological theory). This of course doesn’t mean that there aren’t still excellent clinicians who use a psychodynamic approach. Hopefully you can see that there are nuances to consider when it comes to evaluating where your information is coming from. You have to think about your client’s background, their knowledge of the problem they’re facing, their belief systems and previous experience, and so on. As a prospective clinician, for instance, all of this will go into determining which information is relevant and useful for your clients.

I asked CoPilot to write me a literature review about gender discrimination in the work force. It focused primarily on wage inequality (a well-trodden but important path), but half of its references were to a blog written by what looks like a not-for-profit organization with a mission of helping women. In particular, a lot of the data came from surveys that the organization created. Because the results were not peer-reviewed, we would have to evaluate the survey questions directly to make sure they weren’t designed to inflate or shrink the outcomes. For fun, I asked CoPilot to evaluate the credibility of the source, and this is where the cracks started to appear. Like the examples of LLM responses I’ve shared before, CoPilot gave me three reasons why the source was credible and why it was not credible. But these were listed without any sort of magnitude of their importance. 

Strengths:

·      Large sample sizes

·      Longitudinal data sets

·      Strong organizational reputation (citing themselves!)

Weaknesses: 

·      Not peer reviewed

·      Potential organizational bias

·      Limited theoretical focus

The way these are written makes it sound like a balanced comparison—“Sure, there are some downsides, but there are also several upsides. Good and bad.” But when I look at the list, the weaknesses OVERWHELM the strengths. It would be like researching your next car and landing on a Porsche 911-GT4, stating that is has top-of-the-class reliability (strength) and is unaffordable (weakness). Its unaffordability trumps any strength, because it rules it out as a viable option!

Students will have to develop the scientific discretion for assigning weights of importance to the information LLMs generate.

Using the Information to Accomplish a Purpose

Students have probably written a handful of research papers by the time they reach college or grad school. They collect all this information, but what do they do with it? If they just summarize the information, then what they have is an encyclopedia: each new section of their paper is like a separate entry in the encyclopedia. This is how LLMs organize information: a bunch of unconnected sections. But if students want to make an argument, then they have to be willing to state their position clearly (thesis), and then take each bit of evidence and tie it back to the original position. Keep in mind that the pieces of evidence will vary in how strong or weak they are, and students will have to decide which ones to include to support their argument in the best way. This isn’t easy to do, which is why I have to write multiple drafts of an article before I will submit it for publication. It isn’t until I’ve been working on a project for a while that I’m able to determine the best sequence of evidence to use in order to make my paper the easiest to read.

As a college professor, I have read around 200 literature reviews that were written by LLMs. Some of these have been excellent while others have been poor. When they are poor, it is usually because they are vague (and don’t really tell me much about the topic), they lack detail that shows application to real people and real circumstances, and they lack syntheses (by which I mean a bunch of unrelated ideas are shared without making connections between them). 

But a growing realization I’ve had is that LLMs tend to give very simple summaries of the materials they reference. I can describe it by way of analogy that is hopefully still relevant today: the way LLMs reference materials would be like saying that the Disney movie The Lion King is about lions. For instance, “In the African wild, lions make up an important part of the food chain (The Lion King). This isn’t wrong, but it also missed the many lessons that the movie attempts to share. A lot of work goes into writing a book or article (or producing a movie), and one of the contributions can be summarized in a few words. But LLM research papers reduce all of the articles they include to summaries that are just a few words long.