This MIT prof says we don't know enough about AI to teach it

There is no shortage of discussion and debate over the role of AI in education — and that includes how students learn about the technology and how to use it. But my next guest says we just don’t know enough about AI yet to say we know how to teach it.

Justin Reich is a faculty member at MIT, where he studies online learning and how teachers learn. He’s also host of the podcast The Homework Machine and recently wrote a piece in the Chronicle of Higher Education titled "Stop Pretending to Know How to Teach AI.”

In the article, Reich quotes a guide his lab has put out for K-12 educators about using AI, which says, in part, “A guidebook of tying knots will show you exactly how to tie the knots the correct way. A guidebook on AI in schools in 2025 can’t possibly do that because we don’t even know what the knots are, let alone how to tie them.”

The Show spoke with Reich about all of this, and he began with what exactly he’s objecting to, relative to how especially universities are teaching AI.

Full conversation

JUSTIN REICH: I object to the idea that we know what to teach students about AI or about using AI that will help them be better prepared for their futures. We do not know that yet.

MARK BRODIE: So how then should schools be handling AI? Because as I don’t have to tell you, like, this is a huge topic of discussion in terms of do we allow AI, do we not allow AI, how do we try to if our students, we figure they’re going to use it anyway, what do we do with this?

So what do you think schools should be doing?

REICH: Yes, we’ve done this big investigative reporting project called the Homework Machine where we interviewed 120 teachers and students across the country. And certainly one of the things we heard most from teachers was our institutions have to do something that if they don’t do something, then it pushes those pressures down on the individual teachers to figure out things on their own. And it’s inconsistent between classes, and that’s no good.

What schools need to be doing is recognizing that there are, at this moment, there are no best practices to be found. We don’t know what the right way forward is. When you have to do something and you don’t know what it is to do, then you have to put yourself in a kind of experimental mindset, and a mindset really sort of shaped by humility, that our early guesses might be wrong.

So what I would encourage schools to do is to look out, look at their institutions, listen to their teachers, listen to their students, and say, “All right, what’s our best guess for how we might move forward.?” And then you’re gonna have to test things. You have to test policies, you have to instructional practices and really think about them as tests.

Think about them as, OK, this semester we’re going to try this and we’re going to gather some evidence from before this semester, before we had this policy and we’re going to gather some evidence afterwards and we’re going to see: Are teachers happier? Are students happier? Are they learning more? When we look at student work, do we really see evidence of better learning emerging?

Eventually — eight years from now, 10 years from now, 20 years from now — I hope that if we had a conversation like this, I would be able to tell you, hey, we’ve studied this for a while and, and we have a good sense of these are some pretty good ways of using GPTs and LLMs in academic instruction. And these are some ways we should really stay away from that.

BRODIE: That sounds like it might be challenging for schools though. I mean, in terms of just saying, “OK, we’re going to do this for some amount of time and then we’re going to do this, then we’re going to try something else.”

While scientifically it seems like it would probably make sense, I just wonder if logistically and policy-wise schools are really able to adjust quite that much.

MIT Teaching Systems Lab

/

Handout

REICH: Super hard. I’ll tell you a little bit about the alternative, which I think of as a really bad alternative. So Google was founded in 1996. There are other web browsers before then. And in the ’90s, we developed a series of methods for teaching students how to search the web.

And we now know that all of that guidance is really bad, that students who use those practices do worse on really important web challenges and web skills that we want every citizen to be able to do than people who learn more effective strategies. Those more effective strategies are called lateral reading.

But the point is we taught tens of millions of children and young adults provably ineffective ways of searching the web for decades. And we’re still doing it. Google was founded in 1996. The first peer reviewed paper with really effective methods for teaching students how to search the web was published in 2019.

So that’s, you know, that challenge took us 25 years to figure out in a really robust, scientifically rigorous way. So I completely agree that what I’m proposing is hard. My sense of the alternative is that we make a bunch of guesses, those guesses are wrong and we miseducate a generation of students. And that I also view as a really bad idea.

The thing that we want, which is that we can quickly guess effective strategies for teaching. This stuff is unfortunately not an option. And so that’s what we’ve got to work with.

BRODIE: Sure.

REICH: Tough choices.

BRODIE: Yeah. Well, I wonder if you see this and you reference the example of Google. In some way, do you see this as maybe scientific research, maybe more on the university level, where you’re asking a lot of questions and you don’t necessarily know the answer, you’re legitimately testing hypotheses to see what is effective, what is true, what is not true?

And in some of those cases, as you referenced earlier, you kind of have to lead with, “I don’t know, we’re gonna have to all figure this out together.”

REICH: I think we should be totally honest with students and our colleagues that we do not know and we’re going to have to figure it out together. That the challenge is not to like look out there for best practices — which don’t exist because no one has come up with them yet — but to say in our local context, like we’ve gotta do something.

Teachers are telling us over and over and we gotta do something and we don’t know what to do. So let’s be really clear and transparent about that.

The other reason to be clear and transparent about that is that we’re going to make mistakes. And we need to communicate to all the people in schools right now that some of what they’re learning, some of the early guesses they’re hearing are wrong. And they need to be ready to unlearn the those early ideas in five, eight, 10, 15 years as we get better knowledge and better science about these things.

And I totally agree that cultivating that kind of mindset about humility and uncertainty, it’s way harder than what we can do in other subjects where we can say, “Hey, we know pretty much how to teach you how to write.”

This is a different set of challenges that schools and universities face. And it comes at a terrible time for having these challenges. Universities are under tremendous strain from a kind of public and political backlash. Schools are under tremendous strain from the challenges they faced during the pandemic, the lingering effects of that.

It is a terrible time to have a giant machine released into the world that can do all the students’ homework for them. But that is what happened. And so now we all need to work together to figure out how to deal with it.

BRODIE: So how big of a mindset change do you think it would be for schools to take the approach you’re suggesting? We’ve talked about sort of the logistic, the practical challenges of doing it, but in terms of the mindset of not only not coming up with a set policy that they’re going to stick with for a certain amount of time, but also, to an extent, just saying “We don’t really know, and we’re going to have to all figure this out.”

Is that a big change in the way people think about education and the way that schools should be operating?

REICH: It certainly is at the kind of institutional level I’m proposing. I think if you get down to the classroom level, the sort of level of teachers and instructors, if you would be like, “Hey, do you sometimes change your practice? Do you teach in a different way from one year to the next, and do you see whether or not the new thing you’re trying is better than the old thing?”

I don’t think most teachers would be like, “Whoa, you’re talking crazy talk, Professor Reich. Like, what is this?” They’d be like, “Yes, of course. We’re professionals. That’s what we do from year to year.”

Also, everyone who has been teaching for six years or longer has an extraordinary experience of this is essentially what we did instantaneously and systemwide during the pandemic.

BRODIE: Are there schools that you have found that have taken this approach already?

REICH: No, not in the kind of systematic way. No.

I think the reason why I’m talking and writing like this is I think most schools are in the stage of saying either we have to find some best practices or we have to wait for some best practices to come out. I mean, this is like a pretty standard way of thinking about adaptation and change in schools. And is this idea that somebody’s going to figure this out pretty soon, and once they do, we’ll just take what they figure out and we’ll incorporate it into our system.

And I would just caution most institutions that we are quite possibly a really long time frame away from being able to say, “Ah, we pretty much have figured this one out.”