Supplementing and Enhancing Applied Scientific Reasoning Skills with AI

Applied scientific reasoning means learning how to practice in your job the rigor, care, and precision necessary for conducting experiments. For professional psychologists, applied scientific reasoning is a set of skills that will transform how you listen to a client, how you observe symptoms and behavior, how you communicate your understanding with clients, and what conclusions you draw (and, more importantly, when you resist drawing conclusions).

The three basic skills of applied scientific reasoning I explore are evaluating research, interpreting data and results, and drawing conclusions. Because scientific reasoning is based on logical principles that are fixed and unchanging, LLMs are especially helpful at doing this. [AI integrations will be written in red text.]

Below are three skills for scientific reasoning and how AI is helpful or unhelpful in performing them.

Evaluating Research

Evaluating research means assessing how well a particular study has been done. This includes considering the history of the topic and how others have studied it, assessing the internal logic and rationality of the claims, examining the feasibility of the conclusions drawn, and weighing the impact of the study beyond the participants sampled.

In practice, this will mean that you will know what to look for when looking at an example of research—such as a book or a blog post. But let’s put this in context so you’re not just researching some random topic. Say you are a marriage counselor and have a couple who is interested in exploring an open sexual relationship, which means experimenting with sexual partners outside of the marriage. They explain that they’ve heard that open marriages are the best way to solve their particular marital problems, and they give you a link to a study that (to you) looks legitimate and credible (more on how to do this in Information Literacy, which I’ll post about later). Evaluating this example of research means that you will know how to look for and critically evaluate the research question, the research design, the data collection and analysis, results, and conclusion of the study. 

Research on therapy is extremely hard to do well, and it is likely that the authors of the study cut a few corners. This isn’t necessarily a red flag or evidence that they’re lazy. Corner-cutting is often necessary in order to complete a study at all. For example, it would be hard to find a meaningful sample size if the disorder you’re interested in is humans who compulsively eat paper and other bits of inedible household materials. Even a study with a small sample size will be better than no study at all. Each study has certain constraints the researchers must work within. (Actually trying to design a study yourself is a great way to realize the dozens of difficult decisions you have to make when conducting research.)

I asked CoPilot to evaluate the research design of a 457 participant longitudinal study examining long-term effects of child abuse. It’s a study I had handy, because it is a topic (child abuse) students claim to be interested in. I copied a detailed abstract into the prompt, and CoPilot gave a detailed summary of the study’s scientific strengths and weaknesses, listing five of each. For example, as a weakness the LLM said, “Some measures relied on self-report, particularly adolescent attitudes, peer approval of violence, and dating violence experiences.” Multiply that by 10 strengths and weaknesses and you have a lot of information that MIGHT be useful for evaluating the study. But CoPilot can’t help you decide which strengths are the strongest and which weaknesses are the weakest. It also can’t help you decide if the strengths outweigh the weaknesses or the reverse. It can only give you information. You have to develop the discretion for interpreting this information by understanding how those strengths and weaknesses will apply to the real problem you’re trying to solve in your job (such as whether an open marriage will be the best choice for this couple and, if so, how they can go about it).

Interpreting Results

One of my favorite problems to give unsuspecting students is interpreting a table of raw data—the results of a clinical trial. Here is a basic example of that (below). Ask yourself, is this drug effective? Assume that a higher depression score = more depressed.

 

Table 0.1. Effectiveness of Made-Up Drug “DK1” at Treating Depression

 

	Baseline Depression Score (before taking DK1)	Post Treatment Depression Score (after taking DK1)
Experimental Group	22	16

                        n=200

 

If you know how to read a table, then you’ll be able to see that, across 200 participants, there was an average decrease in depression of 6 points (27% decrease). The simple answer is yes, the drug seems to be effective. But in clinical research, it is imperative to include a control group that receives no treatment and a second (placebo) control group that receives a placebo treatment. Interpreting the results of the study means comparing the changes to depression across each group. You might be surprised to learn that a decrease in depression from 22 to 16 is actually the smallest decrease of all three groups, which would mean that DK1 is a worse treatment than is no treatment at all!

Interpreting results is an area where LLMs shine, as long as they don’t make any silly mistakes. We can anticipate silly mistakes will occur when our topics get more specific and nuanced, which is what happens as we leave behind the general topics of lower level psychology courses and begin applying principles to specific populations of people. 

When I gave CoPilot the question about DK1 and provided the table, it correctly interpreted the numbers and explained why there wasn’t enough information to draw conclusions about the effectiveness of DK1.

Drawing Conclusions

Conclusions apply research results back to the original research question. A good conclusion requires a consideration of the context in which the results appear. It is unlikely that you will find a study that answers precisely the question you are after. Say you have a male-assigned-at-birth client who is gender questioning. While this is a growing area of research—much bigger than it was 50 years ago—you are unlikely to find well done studies that apply specifically to your client’s precise social, geographical, religious, racial, and socioeconomic contexts. You will learn how to gather what evidence you find and draw conclusions from them as they pertain to your specific client.

This is an area where LLMs are weak. LLMs are good at generalizing to the broad population. They are bad at giving context-specific information, unless the context you’re after is super common. But if it was super common, then you probably would have learned it already in an introductory psychology or sociology class.

Following up on the long-term effects of child abuse question earlier, I asked CoPilot how the results might apply to children of color in Albany, Georgia. CoPilot didn’t have this data handy, so it generalized my question to all children in the state of Georgia. This might seem helpful, but if my population is children of color in a rural part of the state, then the LLM is perpetuating a deeper problem of systematically ignoring two important demographics of my population: race and rurality. By following the LLM, I am participating in the white-washing of my subject matter. I’m complicit.

Conclusion

I was genuinely impressed when I first dropped the table of data into an LLM and asked for an interpretation (ChatGPT was more thorough and detailed than CoPilot). Were a student to do the same, I’m confident that they could ask follow-up questions to clarify what aspects of the interpretation mean. 

The more I worked with it, however, the limitations of LLMs for doing my scientific reasoning for me became clear. They can only give me more information. I have to be the one to weigh the significance of each bit of information.

Think of it like asking for directions between Atlanta and Savannah. LLMs can give you lots of routes based on different goals (no highways, fastest route, most scenic), and it can give you strengths and weaknesses of each. But you have to understand your goals for taking the trip and interpret the information based on those goals. Putting the information into your context is your job.