Co-Intelligence: AI in the Classroom with Ethan Mollick | ASU GSV 2024

[Music] I'm thrilled to bring on stage Ethan mik Ethan is a professor at Wharton at the University of Pennsylvania uh where he studies and teaches Innovation and Entrepreneurship uh and examines the effects of AI on work and education um his newest book co-intelligence just came out this month it's about AI uh he'll also be doing a book signing for it right after this um at 11: downstairs so if you enjoy the talk definitely go meet him down there for that um but in the meantime here to give us a talk about AI in the classroom

please welcome Ethan [Applause] [Music] mollik hi everybody it's so good to see you um we're going to talk about AI in the classroom and AI as a co-intelligence for teachers and as I try and do in every presentation I'm going to hopefully demo some stuff which might break we we'll find out out um but what I wanted to do is kind of just a brief moment of History so um I had uh the first time I came to jsv was last year where I gave a talk on the slightly smaller stage next door um and

at that point people sort of knew AI was coming but didn't know what it was and I first kind of went viral for my AI work with a uh my syllabus which I I've been using having people use GPT before uh in my classes I had an assignment where students had to cheat by creating an essay that would that would fake F uh fake me and turn it in uh and I did that even before chat gbt came out so when chat gbt came out this was my policy and it was shared everywhere I've seen

it been integrated into lots of other policy documents uh it was the idea that okay if you're going to use AI in a class you should use it in a way that uh is responsible and you're responsible for your own errors and this was great in the early days of chat gbd early days being just a few months after chat came out because um this is the test scores of gbt 3.5 so this is chat gbt the free version it's still pretty amazing by the way when I say that we have a um like a

65% on the gr qualitative exam the exam to get into GR out school that's not 65% right by the way for GPT 3.5 that is 65% of human test takers beaten by the AI so the free version already very good but still made mistakes and that was okay because then I could make my students accountable for their errors and AI was a tool they could use uh right around the time of gsv last year though GPT 4 came out the more advanced model and it basically blew every other kind of test out of the water

and there's open debates still whether it's the 99th percentile of the bar exam or the 60th percentile of the bar exam but that's kind of where we are at right now it's like beat 60% of humans or 90% And so it doesn't make sense anymore for me to assign my students even though I teach really smart students to they can use AI as much as they want to as long as they're responsible and they talk about the outputs because the AI is writing better than them in most cases it's answering more cleverly than them I

have to have a lot of effort to figure out if they do make mistakes or not and that kicked off what I like to refer to is the homework homework apocalypse um there is basically no homework assignment the AI cannot do if you think it can't do it you probably haven't tried enough or you probably haven't used the paid versions The gp4s the clae 3s the Google because all of them basically solve every problem visually or not um throwing whether it's a geometry problem or uh it gets a 3.34 at Harvard on essays which is

pretty good even with great inflation um and so we're seeing like a fairly large amount of you know anything you throw this it does right and it also works the same way in post-secondary education two studies I happen to really like on this the one the left is from Stanford Medical School that shows that given novel cases to solve GPT 4 beats all the second and third year medical students in terms of accuracy of outcomes on the right I like this one because it's from law schools what it shows you is Contra the grades people

get in a contra when drafting contracts the University of Minnesota and the um blue curve is the grade distribution curve before people got access to gbd4 and then in the experiment people given gbd4 you can see there's no longer any bad lawyers like it cuts off the end of the curve entirely right which is what we see over and over again so whatever kind of work we throw at this it does and this is kind of bad because homework turns out to be important schools based around the idea that we assign people work and then

they do them cheating was already something that was happening in classes but now we have a universal engine that even if you're not cheating you're getting help with you're it's helping you out and this is a problem because even in a world of ubiquitous AI right education's going to matter more than ever Baseline knowledge matters more than ever because you need to know enough to be able to work with these systems they are trained on human knowledge on the on all the concepts of human history and so we want people to know that to be

able to work with it we want them to be responsible consumers we want them to be responsible citizens so we want to make sure they know things that's what education is important for secondly um expertise matters more than ever so one of the arguments I make in my book is that AI is pretty good and a lot of things it's at the 80th percen of human performance when we look at its ability to be a high-end consultant or a programmer it does very well with these things but eth percentile is still not as good as

most people are at whatever they love and do best and the only way you're going to be better than the AI in those fields is if you build the expertise to do that and expertise is a grinding process of building Mastery and we still need to be able to build expertise and that's what we do in classrooms and then I actually think schools in the long term are going to be just fine AI is a huge amount of opportunities what I'm actually worried about from an education perspective is what happens after people graduate so I

teach at undergrads and mbas and postgraduates and what I do is I teach them to be great generalists and then I send them off to various companies you know whether that's Google or Goldman Sachs or you know or or Teach for America or any other organization and then they go through a period of apprenticeship right so even at this sort of college level they're going to work with somebody and that's the process where they actually learn how their job works and if they go to you know a financial firm they get yelled at for two

years and spend 180 hours a week yelling you know being working all the time and that's how they learn by doing the grind grinding work about what it takes to be an investment banker or a teacher or anything else this kind of process the first thing that people do with AI is replace their interns because I don't want to assign a task for a human to learn but I could just have the AI do that more easily the AI doesn't complain it doesn't cry it doesn't make mistakes right so I so what I'm seeing is

the first crisis is actually not going to happen in the schools it's going to happen when people graduate and we're going to have to think about how to bring that material back into schools to teach that so overall that means that I think education matters more than ever in a time of AI and that's why I'm so excited to be into this room because you all know this and this is not a replacement for teaching this is not a replacement for formal education this is something that actually Spurs us on to do more but we

have a system that can create a universal cheating tool so what do we do with that well we really have two two choices we can make it they're actually both valid choices so Choice number one is that we U is that we return back to the basics all right so this is the original Little professor calculator from the 1970s and the calculator caused a massive crisis in schools right it did math what do we about this so there was a fewe period of absolute kind of chaos around this and some Schools allowed it some didn't

how would we afford it how do we democratize access all the questions we're used to asking and then over time it went from parents hating using calculators in class to parents demanding uh and teachers demanding that calculators be used in class and we build an educational system around there's a certain amount of work you do by hand while you can't cheat with the calculator and then we move on to board Advanced topics and whether we like that or not that's how a lot of fields are going to end up working which is we're going to

end up doing a lot more inclass and there's going to be a lot more kind of activity based observational you know where people are in front of us and that's okay it turns out one of the few one of the really clear results from pedagogical research is that things like doing low stakes testing are incredibly valuable right low stakes testing increases student ability to remember things on related topics builds connection for them so it's okay to go back to like a more formal kind of like let's double down make sure we're testing people make sure

that we are doing uh low stakes test not let's say high grade tests and that we are returning back to sort of classroom monitoring that's a valid response if that's what you want to do and I think that there's going to be a lot of that happening a lot of basic writing might be done in class instead of outside of class with essays but I think we can also look at this as an opportunity for transformation as well so these are actual assignments um from canvas thank you any infrastructure people here um but uh that

I put for my class this year right so when I was teaching um my my undergrads and mbas um I AI was required and was deeply integrated into the class and every possible way so they had to work with the had to create things that one of the assignments actually was that they had to invalidate their own uh job they were applying for so they had to create a gbt to replace themselves for job interview so when they went to that interview they would hand that to their interviewer and they would say to them my

job's done got to get a raise now um and what was fascinating is I had 300 students in the classroom and they did all they were you know I had hip-hop promoters I had Navy Pilots I had because I was teaching at Wharton lots and lots of private Equity people um and all of them uh came up with ways of invalidating parts of their job of of doing their job for them I never would have realized you could file Navy flight plans or that this would actually do really good hip-hop promotional tweets or private Equity

deal memos my students were able to figure that out and do it so I was able to push out to them and give them the tools to help transform what they were doing and I think that lesson applies more broadly as well so who should be doing this transformation I'm in a room where a lot of you work for you know a lot of you are teachers instructors administrators some of you work for ED companies and I think we have to think about who should be doing this transformational work because I think there's a lot

of abstract conversation but let's be really specific about what makes AI interesting or different in this way so there are three things I think about with AI that I think people don't often realize the first of those is that there is no secret instruction manual I mean I talk to open AI I talk to Google talk to Microsoft anthropic all of these organizations on a regular basis and um I think people have this model that they know what these models do and what they're capable of doing in your field and they don't nobody knows anything

right that's either comforting or upsetting it's up to you but um it's not like they when when GPT chat GPT was released they thought homework would be destroyed as a result of this they didn't even think about that they don't know how good it is in you know in fifth grade social studies or in teaching negotiation classes or any other topic nobody has the answers and there's no instruction manual out there that you can hire somebody who knows the secrets there is no secrets everybody's figuring this out together the AI companies are basically releasing these

products almost as soon as they're out and trained and we're all discovering this together so if you're kind of waiting for instructions from above they're not forthcoming right nobody knows anything right the second thing um is that I think because AI is viewed as a technology product when I talk to non-te people they have a little bit of an aversion to getting their hands dirty with it because they're not coders there is some early evidence I've seen that suggest that coders are actually the worst at working with AI because AI doesn't work like code like

code shouldn't argue with you it shouldn't make moral objections to your points it should give you different every time AI does all those things what AI is best at actually is if working like a human being It's Not a Human Being don't make a mistake but it works like working with a human being and I find and in uh and in our research we're starting to see some indications of this early on that teachers are actually among the best workers with AI because what you need to do is understand the ai's mindset doesn't have a

mind but it thinks it works like a person if you understand how to give clear instructions you understand where someone's going off course you understand how to give her correction you're actually going be really good at proing Ai and one of the things I want to see is more Hands-On by less technologists and more teachers instructors Educators who I think will actually find they're very good at working with these systems because youve you work with many other Minds on a regular basis you're used to giving instruction you're used to see where someone goes wrong all

of those skills you'll find are very valuable in this case and the final thing is the I've been running an internal edtech company at Wharton for like the last 10 years and I'm very proud of what we've done but one of the things you learn very quickly is like teaching environments are different everywhere every school is different every classroom is different every teacher is different and they know their own environment and one of the failings of edtech overall has been that we have to build sort of large scale solutions that are not customized to your

needs and have to trickle down from above what's really exciting about AI is that instructors who know their own environment have the ability to build their own sets of tools or modify tools that are responsive to their environment instead of taking away Authority and making it more to do it's about giving yourself Authority and control in a way that we've never had before democratize in access to education technology we're already seeing a little bit of this when I talk to people a lot of people in classrooms this is one of our prompts all the promps

I show you by the way my I work with my wife Dr leik and all of this and we co-author all the papers together she's mostly our amazing prompt engineer uh and by the way no programming experience at all before teaching understanding our teaching Works understanding mindsets building edtech tools that's that's where this comes in but here's you know a simple one that creates quizzes right and if you give this any document you will end up getting a pretty good quiz as a result teachers are doing this already everywhere right I'm seeing a lot of

this kind of tool creation of like helping you prep for classrooms uh it's not showing up at every survey yet but some more surveys are finding teachers are adopting it for this reason less commonly I'm seeing people use this to build activities and explanators that are actually deployed to students so I find it kind of miraculous that I could say literally this paragraph to an AI create an interactive stimulator that explains the central limit theorum and I get a working website like a zip file or like a full-on website that I can just show to

my students it's like here's a simulator that shows you how this works it would take me a long time to code that myself if I even remembered how to do it and to deploy that but AI can do this entirely so it changes the frontiers of what we could do and I want to make the argument we can go even further and that um instructors are becoming coders now okay so I want to make that argument for you and show you a little about this this is an example of one our one of our prompts

it turns out you could put a four-page prompt that's basically like a assignment instruction rule set into Ai and it will Ex that pretty well most of the time so I'm going to try and show you some of this here there's nothing better than live demoing uh and um especially live demoing chat gbt because it breaks on a regular basis but we will find out uh whether it works here for us um so I want to show you an example of this and all of this by the way we share all these prompts um so

if you want to play with any of these you should um and hopefully we can have my screen up here okay great so hopefully you'll be able to see so I'm going to start with uh let's just try an exercise this is a simulator the one you see they saw on screen before um which is about practicing negotiation so I can just say here help me practice the negotiation and then what we know and I've been building simulators for 10 years is that you need to customize this for the student right so at first the

AI is actually going to ask you what's your experience level in negotiation I could say uh I am oops I am an undergraduate I'm just making this up as we go along art student who has no real experience okay and just hit enter and hopefully we'll see if the AI actually works for us today um and so we're having a whole like I'll show you again the whole prompt this is all in Pros here it's now giving us three choices of scenarios we can run right so an art gallery purchase notice how it's customizing for

us building customized instruction has always been a really hard difficult process it just does this stuff right out out of the bat so uh someone give me a number What scenario do you want to try two okay I heard two first so let's just do two okay um and um and it's going to set and it's going to give us all the details of the scenario right I don't have to Rite out it's going to tell us what our role is it's going to tell us um the the basics and the details of this particular

experience what our objective is and then it's going to as we go along give us feedback and information and then ultimately grade us at the end of this whole process and this is literally just a few paragraphs of text paste it into chat jpt there's no fanciness about this there's no outside AP or other system this is something any of you could paste in and have working for a student this way right um which I think is a very powerful transformation for how things work um and so you'll see it starts to tell us a

story here it's a nice little bit on the story too you're meeting the cafe owner of their establishment a cozy place with the vibrant atmosphere and it's quoting to us and we can go back and forth in negotiation and we'll get feedback at the end in this kind of way pretty incredible right um and how do I respond to start the negotiation on a strong note um you know we'll just I'll just say give me me the art now okay and um so um right and now it's going to give us give us some feedback

here um so you know uh but you know interesting right now will this I I do negotiation simulation so I know what's right and wrong about this but think of this as a teacher one of the things I could be doing is evaluating the tone of this and saying is this in the right heading in the right direction or not right and it's going to give us slightly different answers every time um and it's trying to help me because I seem desperate um but the idea here is be beyond that right I can also do

let's go back to our GPT list here so but I can go further than that right I can I don't have to just do let me see if I can find it here we have lots of gpts we're building again we share all this so this one actually will help you create a prompt like that negotiation simulator so I don't have to just give you this I can actually give you tools that we can anyone can create that will help you create stuff so let's say we want to learn about negotiation and this is for

a teacher to use it's designing a oh this is a tutor actually it's not the simulator but it's fine either way and there's two concepts of negotiation we'll cover we'll cover batna and we'll cover uh Zopa so batna is your best alternative to a negotiate agreement Zopa is your zone of possible agreement notice it understands the concepts and what they are if they didn't again as a teacher as an expert I would be looking at the AI and saying oh no it doesn't understand this well and I would be correcting it and thinking about that

right and what do they confuse it they confuse this with best offer which is a different idea um and so now it's going to say that's a good point and it's going to tell me how brilliant I am which is nice it's very supportive um you know if no one else likes you the AI will um so um I don't know you figure it out it will take us through a bunch of questions but um you know you can see um I just want to show you how this works here and now what it's going

to do is it's going to actually create a prompt for us that I could then paste in and anyone can use right so I it's not about showing you demoing an ed Tech product to you is demoing the idea that you can now roll your own edte and you can create a tool that you could share with other people you create a tool that creates tools and share that with other people and there's all sorts of exercise and I could just copy this code and and you know paste it in and that would turn the

AI oops if I can remember how to do this and that would turn the AI into a AI tutor is it the tutor you want to use I don't know you'll have to experiment with it and figure out what it's good or bad at I can't get answer that question for you you are the subject matter experts and the more we democratize out the use of these tools the better off we are and there's many other kinds of tools we could do for example let's just pull up one more in the couple minutes I have

um just to show you and again all of these are publicly available but let's take here's another nice example this is um this coaching that teaches you to inter leave topics together um let us find a good one here uh okay here's a nice one teach the AI exercise hopefully it'll run um and this one has the as every teacher knows teaching others is one of the best ways to learn because it fors you to organize your thoughts and put stuff together this has the AI Act as a student and your students can then teach

them a topic and the AI will respond one way or another do we want to be chatting inquisitive or skeptical and beused which one I'm hearing too many actual teachers in the room right skeptical beuse is is the right way to go um and we you know we could say negotiation um and then it will actually give you feedback on how good you are at teaching it one way or another could you start by explaining what it is and why it's important right so soliciting answers from you and these are just things we've come up

with right the idea is that you are a room full of people and everyone watching this remotely you're an audience full of people a world full of people who have not had the ability to make tools before we've always had to take other people's tools and try and adapt them or live with them or adjust to them and now in your class in your setting you can build the tools that work well for you and to me that is the most exciting kind of part about this kind of AI moment that we're in right now

um and if we can switch back to the slides for a second here so what I what I'd like to AR is that like this is the time for us all to build and what worries me the most is people viewing this as something happening to them that AI is like a thing that's going to impact education that's true but we finally have agency over this right this is not something that we have to wait for to roll down through 25 layers of administration and be turned into products and sold this is something we can

start building on ourselves that we can all experiment and learn from because there are no outside experts there isn't anyone who can tell you how AI is best deployed in education right it's kind of amazing to me in some ways that I'm still here A year later as one of the leading people talking about this there should be many many leading people discussing there already are it's bubbling up on all kinds of districts we're just not sharing enough we're not talking enough we're not building our own communities of practice and sharing how we can make

this better we're not taking ideas from edtech and from pedagogy and building pedagogy instructor first approaches to how education should work so here's my advice on what I think we should do um so first of all as Educators I think we should be experimenting legally and ethically with AI right and there are lots of caveats here but the truth is things like privacy are mostly solvable concerns at this point every AI company knows that they want their systems to be used by companies and educational institutions and they are figuring out privacy solutions that is not

a long-term problem so I I recommend inviting it to everything you do the only way you can learn what's good or bad at and by the way it's going to be terrible at some things and that's okay right you need to learn what it's good or bad at the way you do that is you use it for everything give it every one of your assignments before you turn them in use it for every you know trying to email a parent trying to develop lesson plan come up with ideas try it for everything right and you

will find what it's good at or what it's bad at as a result and that's how you learn the second thing is to dive in I you know we we I have a series of videos other materials you can look at but really the only way to figure this out is just to use it and the way to use it is just to talk to it like a person so if you it's not a person but if it's built that way if you're good at working with people you'll find that you're good at working with

AI interact with it tell it things correct it when it's wrong learn what it knows and doesn't know you'll learn all kinds of interesting things like for example one flaw in all the major large language models right now is they all believe that learning styles are real because it's been repeated so many times in literature that that's what it things is real so we actually have in all of our prompts we have to correct it say remember learning styles aren't really a thing um and and then it knows better we have to learn that set

of stuff and you have to figure out how to teach it you can't be waiting for someone to tell you and then most importantly um you have to share what you learn we have to start building more communities that are discussing this like one of the things you can do is you can give this information out you can share it with people and lets them build on it because what we really need to do is democratize in this moment and take charge of where AI is leading us to be positive for education to be positive

for instructors to result in more Security in the classroom not less to make it feel less precarious and we're only going to do that if we as instructors and as Builders and as Educators start to think about that kind of stuff in a serious way uh and then finally like this is the beginning if you're not on this curve right now the other concern I have is that like this is just the start I only have a few months of foresight nobody else does but like we're not done yet with this curve if you think

like okay AI is ready we'll adjust to in the next coming three or four years that's not what's going to happen right I don't know how long this Curve will go on nobody knows how long this curve is going to go on including the people who make it when I talk to people training AI systems they're divided on how long this goes and how good these systems get but they are getting better and if you sit this one out it's not going to result in this going away somehow or getting put back in a box

or disappearing so we either decide we're going to do something with this that's positive we're going to build on what's positive we're going to role model the positive behavior behavior that comes from AI we're going to show people how could be used to make the world a better place or other people make that decision for us so I I hope that you think about this and that we figure out ways to join together as a community of Educators administrators education technology experts and think about how to build kind of a better future that way so

thank you very much um I have all the prompts that I showed you are in more useful things.com we share them all um we have a bunch of academic papers and videos there too um and there's a bunch of practical guides too and I really want to start the norm of like sh sharing even stuff that doesn't work the stuff that breaks we need to be in innovating as a group I don't think we should be waiting for someone else to do it for us so thank you very much I'm excited to be part of

this journey with all of you thanks [Music]