[0:00:24] Very good. So good afternoon, everyone, and welcome to the inaugural Data-Intensive Social Science Speaker Series today, part of Love Data Week 2023 here at Johns Hopkins. My name is Charles Meneveau. I'm an engineering professor and serve as associate director of the Institute of Data-Intensive Engineering and Science at Johns Hopkins. This institute, IDEAS is the acronym.
[0:00:51] Participates and helps in many research projects across many schools and divisions of the university. Brings together hundreds of researchers, educators, and technology developers to pursue deeper understanding of the world guided by big data and increasingly by artificial intelligence. IDEAS also organizes and hosts workshops and events associated with big data challenges and opportunities.
[0:01:17] And among these, we've seen quantitative social sciences often at the forefront of really exciting big data and AI-related developments. So this new speaker series of data-intensive social sciences is really an exciting new initiative. So this brings me to the theme for today's event. I think, as we all know, chat GTP has kind of suddenly burst on the scene and has generated tremendous interest at many levels.
[0:01:44] I think it's been astounding us, but also confounding us as well. On behalf of IDEAS, I should say we're very grateful to the organizers for having put together this very timely event today of presentations and panel discussions that I'm really eager to listen to. Maybe as disclaimer, I should say these opening words.
[0:02:08] are spoken by me. I did not ask ChatGTP to write this, although I have to say it might very well have. So anyway, on behalf of Ideas at Johns Hopkins, I give a very warm welcome to all the participants today. And without further ado, let me pass the presenter mode to the organizers, Professor Ting-Lang Dai of the Carey Business School and Pete Lawson of the Sheridan Library. So Ting-Lang and Pete, please take it away.
[0:02:39] Thank you, Charles, for the really warm welcome. And thank you, everyone, for your interest in this event. We're super, super excited to have you. But let me just say a few words about this data-intensive social science speaker series. So at Johns Hopkins, we have scholars across a variety of schools and departments working on social science using data and increasingly using artificial intelligence.
[0:03:05] But we do not really have a university-wide initiative to bring everyone together. So this is our attempt to unify everyone. And we work on data-intensive research. And we're going to bring together speakers from Hopkins, but also outside of Hopkins. So we figured today will be a perfect start for the event. We're going to talk about how the CHILD GPT is going to reshape higher education.
[0:03:34] So I'm very grateful to have my co-op moderator, Peter Lawson. So Peter Lawson is at Johns Hopkins Library. So he's going to talk about the Lab Data Week, which this event is also part of.
[0:03:50] Yeah, so we're very, thank you, Ting Long. So we're very excited here at the libraries to be co-hosting this event along with IDEAS and Cary Business School. And this is part of a larger series this week, Love Data Week. Love Data Week is an international celebration of data. It's hosted by the Inter-University Consortium for Political and Social Research, ICPSR.
[0:04:10] And the purpose of Love Data Week is to raise awareness and build a community around research data. So to that end, this week at Johns Hopkins, we're having a number of events around data, and we're really excited to be co-hosting this inaugural data-intensive social science discussion about ChatGPT today. Thank you, Pete. And so now I want to start today's, let me just briefly say how today's panel will be organized. We'll start with...
[0:04:40] A talk by Professor Christian Tovich and then followed by another talk by Professor Jared Ebeling. And then we're going to have two panels. One focuses on faculty experience, the other focuses on student experience. We're going to explore how ChessGPT and also more broadly speaking, generative AI is going to reshape everyone's experience. And so let me just briefly say a few words by Professor Christian Tovich.
[0:05:09] So I see quite a few attendees here, actually, are my operations management colleagues and students. So for them, Christian really does not require any introduction. So he has co-written one of the most popular MBA textbooks, Matching Supply with Demand. And so Christian is a world-renowned operations management researcher and also educator. He's an Andrew Haller professor at the Wharton School and the University of Pennsylvania.
[0:05:38] And we have the Chair of Operations, Information, and Decision Department. He also co-directs PEN's Mac Institute for Innovation Management. And he's recently has published in public journals, ranging from management science to new and general medicine. So without further ado, I'll send the floor to you all. Well, thank you for the introduction, Tianlong. Thank you to all the organizers. Really thrilled to be here.
[0:06:01] And as any good business school professor, I cannot say more than two words without a PowerPoint slide deck. So let me, if the technology agrees with me here, let me switch over into PowerPoint mode and share a couple of slides that I've prepared for this meeting. Let me just switch over here.
[0:06:22] Let me first of all say that this is not really a study. I mean, I've written close to maybe 70 or 80 papers where I've gone through peer review and gone through all the pain of writing the paper. This came out of a dinner table discussion with my kids. I have one kid who is a designer and architect, and he has been using MidJourney and Dali.
[0:06:42] for his work and was really interested in AI. And my other son is a software developer and he has been showcasing me how ChatGPT has been doing well on a lot of coding assignments. And so over dinner, the question naturally came up like, hey dad, does this thing, is that thing able to pass your exam? And so I wrote this white paper, I literally wrote this over a weekend, sent it out. On Wednesday, I talked to a journalist from the Financial Times.
[0:07:11] They ran the story on Friday, and the thing has gone viral since. And it makes me kind of somewhat mixed feelings. I've written many papers that nobody has read. It took me years to write. This one has gone crazy, and I'm very excited about that. But really, we need to put out the disclaimer. I'm not an AI expert. This is more of a faculty user experience. So here's what I did. So I teach a course on operations management, policy, and productivity.
[0:07:38] For the non-OM folks, this is about making things faster, better, cheaper. This is about productivity, bottlenecks, capacity. And the course is a mix of teaching cases, Harvard Business School side, you might say, and more kind of analytic numbers, spreadsheet crunching. And let me share with you a typical exam question that was the first question that I gave to ChatGPT as an assignment. So process analysis question here.
[0:08:04] A bunch of resources in the process. You have to understand the idea of a bottleneck, that being the step with the lowest capacity. And you have to infer how to compute the capacity for the first reactor and the second reactor, because it's some form of a combination of how big the reactor is and how long the iron, in this case, needs to spend in the reactor to come out good. And so that was the first interaction I ever had after seeing my son explain me how to use bubble swords and how to explain bubble sword to somebody as if they were Homer Simpson.
[0:08:34] This was my first OIM example. And to just make the story short, it just blew it away. I mean, really perfect answer, finds the right bottleneck, and comes up with an explanation that was, to be honest, that was better than anything that my TAs or myself have written in the past, both for our students as well as the readers of our book. Really well done.
[0:08:57] I used similar examples. I asked you the question of working capital calculations for a given size of a business, given inventory terms, how much money would a founder for a retail business have to raise in order to support the business from a working capital requirement. And I asked you questions about the use and definition of the current system. A plus, no hesitation. Absolutely awesome.
[0:09:20] The next thing I did is I kind of put a question in that was a little harder. For the insiders in operations management, there's a famous case out of public business school called National Cranberry. I wrote a kind of a short synthesis of the case on this half page that you see here. The subtlety now is that there's, rather than having one product go through the process,
[0:09:44] We call it product mix. You have two types of products, in this case, dried cranberries and wet cranberries. Go through the process. And then make the determination, the calculation, the definition of the bottleneck somewhat more complicated. But I was optimistic. After all the time we test ACE, it has aced the first question. So how would it do?
[0:10:05] Well, there's only one way to find out. So let's see how it did. It did correctly find the dryer as the bottleneck. So 600 barrels per hour as the dryer as the bottleneck, it says at the bottom. So that looks good. The story, the narrative, the wording of the answer, again, super well-written, super smooth.
[0:10:27] But there are two issues that are worth kind of pointing out. The first one is when you have multiple products go through the process, it really depends on the mix. And I gave it here on the question. I instructed it to use a mix of 70% and 30% wet and dry.
[0:10:45] If you browse through the answer, you'll notice that it hasn't used these numbers. And so there was something fishy going on. And so I did what I thought was kind of cool, or I was at least impressed by Church EPT. I gave it a hint. Without retyping the answer, the question, I just said, look, remember to distinguish between dry-harvested berries and wet-harvested berries. I love the answer, because without a blank, the thing turns around and says, well, look.
[0:11:12] It is important to distinguish between dry harvest berries and wet harvest berries. So as if it was to be the most obvious thing on the world, it's now responding to my hint. It is weaving in into the calculation. It's weaving in this new information. And it does pretty decently. It says you notice that at the end of all the bottlenecks will depend on the mix. So apparently, it's not using the information, but it tells it at least in the first one.
[0:11:40] Super important. So it's really good at arguing without knowing. And so I think that's a very human skill that I've seen among many of our students. They can talk well, but sometimes just shying away from the math. And then I was, and I've taught this case like a hundred times, I kid you not, but I was looking more carefully at the numbers. Some of these numbers out of repetition I had known by heart. And there was one line in the initial answer that caught my attention. It was claiming.
[0:12:08] That the receiving station was the bottleneck. You see here, if I go back to the first answer, the 300 barrels per hour at the receiving station being the capacity at that station. Well, that is not really in line with the story that the 600 barrels per hour at the dryer would be the bottleneck. And so I was kind of looking at that calculation.
[0:12:32] It looks totally plausible, right? Five dogs, eight minutes per dog, 75 barrels per minute, that is 300 barrels per hour. Well, it's not actually, right? If you think about it, the math is just totally wrong. You have five dogs, they're 60 minutes in an hour. It takes the truck eight minutes on load, so the capacity is an eighth of a truck per minute per dog.
[0:12:54] There's 75 barrels per truck in the vehicle. So the total capacity is 2,800 barrels. So it's off by a factor of 10. And unless you really think hard and check every little bit of the math, the answer is totally wrong. And that one was making me somewhat nervous, right? That again, it is so confident, the answer is so well polished, but yet the math is so fundamentally broken.
[0:13:24] So overall, if I think about the grade I would have given this, and I think that was the number or the score that made the headlines in the media. The simple process analysis of the working capital and the common question, it would have gotten an A+. On the other questions, I had a curing question as well, where I had similar experiences with the primary questions.
[0:13:48] It needed a human hint. It did fine responding to the hint, but it has still some glitches at the end. Overall, I think I would have given this thing a B or a B-, which I leave it up to your interpretation whether you think that is absolutely great or whether it is miserable. I think in the media, you see there was this initial excitement that you can do that while it is so cool.
[0:14:09] And now that everybody expects that this thing is flying to Mars tomorrow, people are flooding social media with all kinds of simple failures and mistakes that the technology makes. But maybe more importantly for us here on this panel and in our meeting today, let me kind of speculate a little bit of what the implications are on our teaching. And I'm speaking here as somebody who teaches MBA students at the
[0:14:33] Fairly elite, privileged business school, as opposed to being a high school teacher in West Philadelphia. So a couple of observations from my side. The first one, I think, is we have to distinguish between what we teach and how we teach. I think as business school faculty, and certainly as operations faculty, we have to teach about AI and GPT just because they are part of modern work. And so I want to leave that part aside. We definitely all have to, as business school faculty, embrace it from...
[0:15:01] What we teach, but also does it impact how we teach? Does it impact the way of teaching? I think one question oftentimes got asked is that people ask me saying like, look, if a chat GPT can get the questions right, Christian, on your operations management exam, maybe you're teaching the wrong content, right? I mean, it's basically, why do I need a board and MBA if the bot can do that?
[0:15:27] I'm a little hesitant to say that, well, you don't have to teach foundations anymore, because if you think about technology as a stack, which I think in computer science is kind of a fairly common approach, there's a stack of, at the top of the stack is the application layer. Think about doing an empirical study. Then there's some form of a methodology, a regression model, an ordinary release square estimator. And deep down in the stack where the nuts and bolts in the machine room of the stack,
[0:15:57] There is, if you remember the definition of OLX, OLX, X prime X inverse times X prime times Y. There's a matrix inversion, right? And when I took regression analysis in college some 30 plus years ago, we would by hand have to invert that matrix. But I think we could all agree that you can do a good regression analysis and you can be an expert in econometrics without being able to convert a matrix by hand.
[0:16:21] On the other hand, if you think about another technology stack where you think of maybe corporate valuation is at the top of the stack, and that's the application, discounted cash flow analysis and compound interest rates might be methodologies that you use. Deep down in the machine room, you're doing pluses and multiplications, right? And so I think, or I hope at least most of us would agree that we still have to teach kids how to do plus, minus, and multiplication.
[0:16:48] As opposed to saying, well, a calculator can do this. So I believe we have to continue to teach foundations. I come out a little bit ambivalent on the testing side. I think before we can answer the testing question, how does this impact tests? We as educators have to examine ourselves, why do we test in the first place? The way I think about it, we sometimes test for skill certification.
[0:17:10] And just like you wouldn't be able to pass a driver's license exam if you would show up there with a self-driving car, I don't think you should pass an accounting exam for CPA if all you can do is use a bot. So I think for skill certification, we totally have to ban the technology.
[0:17:25] A lot of the testing that I do in my course is really geared towards getting a sense of where the students stand on their learning journey, how much of the materials they have understood and absorbed versus how much of the material they're still struggling with. Again, I think their chat GPT is highly counterproductive. It doesn't serve anybody if I teach level seven, week seven material if the students are still struggling with week two material. But I think it's really a way for engagement, right? I think the big opportunity that all of us can hopefully see here.
[0:17:54] This allows us to engage students in a new way and grab their attention to get them deeper into the material in a way that we have not seen in the past. I also think my primary example shows that you can't just use this as a calculator. You have to really teach the whole decision-making process. And that requires the students to learn how to validate an answer to how to develop a gut intuition of what type of answers might be right. And so it's really more than something that you would do with a calculator.
[0:18:24] I'm working right now on ChatGPT from a creativity perspective, and I thought this would be the most useful application so far, that you don't have it do the job for you. You're just asking for stimulus for an idea. Ideally, you ask it to generate four, five, six, seven, or more ideas, and then you just pick the best one, which I found, again, works really well if you need a starting line for a talk, if you need an idea for something to work on.
[0:18:51] And then finally, my final observation is kind of from a productivity perspective. I think lots of has been said of how this will impact the job markets, but I hope if the story is, this is a new technology that will improve certainly our productivity, say for sake of argument, it doubles our productivity. Now you might say like, well, let's lay off half of the teachers. I think that would be a very bad response. I think the beauty of education is that we can hopefully change the productivity gain.
[0:19:20] And do twice as good of a job as educating our students, as opposed to thinking about cutting the labor force in education. So let me leave it as this. I think I got detailed instructions to speak for 10 minutes. I'm a little over that. And Ting Long, I'll hand it back over to you. Thank you, Kirsten. Really, really informative, but also very entertaining talk.
[0:19:44] So now we are opening to really brief OpenAI and later on we will have more opportunities to ask Christian questions. But for now, I want to ask whether anyone have any immediate quick reaction to Christian's talk. Feel free to type using the comment chat or Q&A. So I saw one. Okay, so this question from Aaron Segal Christian, what are your thoughts?
[0:20:12] Competency-based teaching and learning. Not exactly sure whether I understood the question, but maybe you have a better understanding about this. Can you repeat the question, please? I didn't see it. Yeah, so what are the thoughts on competency-based teaching and learning? Well, I think you have to always ask, like, what skill do you want to learn the student in the first place? And I think at a business school faculty, we want to develop hopefully some creative thinking skills.
[0:20:40] And I think there I see a very immediate use for ChatGPT that it helps you with brainstorming, it gives you stimulus for creative thinking. And the other skill that I want to teach is critical thinking, some form of problem solving, analytic skills. And I think there we need to do a better job. I think we oftentimes pretend, as I did in my exam here, that management is just like solving a word problem with 100 or 200 words in there. I think we need to embrace the fact
[0:21:10] That management decision making is a much more complicated process from taking reality, abstracting the mess of reality and squeezing it into some form of mathematical model, then thinking about the model and making simplifying assumptions of how to solve that model, and then deep down in that stack is using specific formulas, formulas that we teach in operations management or elsewhere. And I think
[0:21:32] Being able to navigate that process from reality to abstraction to computational simplifying assumptions, I think navigating that spec and validating each level against the next higher up, I think that is a skill that I think we have to do a better job teaching. I'm certainly pointing the finger at myself. I feel like I haven't done good enough of a job as I should.
[0:21:56] Thank you very much, Christine. We have maybe one more question. I have two questions, both questions about creativity. So because, Christine, you are an expert in innovation management, so I should want to rephrase, reframe this question a little bit. From the innovation perspective, what do you see the role of chatGPT on more broadly generated AI?
[0:22:18] Well, there's a real opportunity here, right? I think we all know that the challenge that, if I think about an innovation course that we teach you, I think the innovation tournament method that we've advocated at Wharton, Calvary, myself, Sagan Edison, where we ask students in the first day of class to come up with some ideas. And we all see them struggle, right? They sit there with an empty piece of paper and say, well, yikes, I have to not come up with these type of ideas. Why not just go to chat GPT and say, well, look.
[0:22:44] I need five new ideas on digital health care. Give me some ideas. In the worst case scenario, you're going to get five OK ideas from Czech GPT. That's better than a white piece of paper. So there's already some progress. But even better, you might look at these ideas and feel like, well, look, those ideas suck. I can do better than that. And now you started jumping into the water, and you started being creative. And so I think it's a way of jump-starting. It's a way of prompting. It's a way of
[0:23:14] Starting the pump in the creative journey. And I think that's this enormous productivity potential, both in teaching as well as in practice, I believe. Thank you very much, Christian, for that really insightful talk and for the excellent Q&A. And so, as I mentioned, we have more opportunities to interact with Christian or with other speakers throughout the session. Feel free to post your question in Q&A. And if you have questions specifically to Christian or to Jared,
[0:23:42] Tom or Jenna, feel free to mention them. Otherwise, they will take questions to answer as I see them. So now I'm turning to my colleague, Peter Lawson, who's going to introduce next speaker, Jared Kaplan. Thank you, Ting Long. So yes, I'm very excited to introduce Jared Kaplan. So Jared is professor of physics at the Department of Johns Hopkins and is a theoretical physicist working both in particle physics and quantum field theory.
[0:24:11] If that wasn't enough, Jared has also been working in AI ML and is the co-founder of Anthropic, which is an AI safety and research company responsible for developing Claude, which is a cutting-edge AI language model rivaling ChatGPT, which, if you were reading the news, has been recently funded by Google for $300 million. So that's pretty exciting. Congrats, Jared. So with that, I'm going to turn it over to Jared.
[0:24:37] Great, thanks. So yeah, I guess for the sort of 10 minutes that I have, I don't have anything prepared to say specifically about the uses of Claude or ChatGBT in higher education. We can talk about that in the panel. But I think I want to talk about the broader trend here, because I think that when a new technology emerges, there might be an intuition that, oh, wow, this new technology is really cool. What does that mean for the world?
[0:25:06] But I think that chat GPT is just the tip of an iceberg and that it's basically part of a predictable trend. And so I think the right question to ask is, in my view, most likely a much scarier question of chat GPT is just the thing that we have this year. But next year and the year after, we're going to have something much, much, much more powerful. And so I sort of want to explain.
[0:25:34] How I came to work on AI and why I think that, because it might sound even crazier than chat GPT. So five or so years ago, I was just doing theoretical physics research, but I had friends who were working at OpenAI and I was on sabbatical. So I said, oh, it sounds like AI maybe is a big deal. Maybe I can hang out and work as a volunteer and play around. And I kind of ended up.
[0:26:00] Doing some research with other people in machine learning and other people coming from physics on what we ended up calling scaling laws in machine learning. So maybe I should flip to that. And the details of these figures just don't really matter at all if you're not in the weeds of AI and machine learning. But the basic idea is that what we found is that you can predictably at very high precision
[0:26:27] Make AI models bigger and train them on more data and get predictable improvements in how well they do at everything. And these were the original language models. So I did this work when GPT-1 and GPT-2 first existed. I then helped build GPT-3 and some of the code models as a consultant at OpenAI. And these trends that show that if you just
[0:26:54] Basically build a bigger model, spend a little bit more money, you get predictably better performance are I think the sort of driver of trends in machine learning to use more and more and more computation to train models. So you can sort of see there's a trend here that kind of culminates in AlphaGo in red. And then there's the current trend that we're on, which I think ChatGPT and Claude, et cetera, continue of basically using exponentially more computation.
[0:27:23] to train AI models each year. And we've reached a point now where if you remember back to sort of chemistry class, there's this Avogadro's number, which is this unimaginably large number of six times 10 to the 23. We're actually training AI models now that do more floating point operations in their training than Avogadro's number. So it's really, really enormous.
[0:27:44] But these are exponential trends that continue. And the expectation is that with the billions of dollars that Google, OpenAI, and other companies have, we're going to be training larger, more powerful models basically every year. And so GPT-3, I think, was actually fairly impressive. GPT-2 was impressive. They're all just points on this line.
[0:28:09] And in the next couple of years, we're going to kind of continue. Another interesting feature of this, it's not super important, is that in some ways, AI research has gotten simpler over time. It used to be that AI researchers working in computer vision did something completely different than AI researchers in natural language processing, who did something very different from AI researchers who are trying to train AIs to, say, play chess. And there's been a kind of, to use physics terminology, grand unification.
[0:28:39] And here we're basically the same AI model can be used to write a poem or generate an image or do anything else. So things are getting simpler and they're scaling up and companies are investing more and more money into training single AI models because they know that they're going to get more powerful AI models from that. And so my kind of personal view on this subject.
[0:29:05] I first got involved and I was extremely skeptical. I sort of thought, oh, maybe AI isn't really going anywhere. I mean, five or 10 years ago, it wasn't really very clear. And I sort of worked on this stuff involving scaling laws, GPT-3, et cetera. And I came to sort of believe more and more that AI was going to get better. But then very plausibly, people have come up with walls that would halt this.
[0:29:29] Continuing process in AI. Like maybe it's not possible to build a multimodal model. Maybe AI models aren't able to reason. Maybe they can't do any kind of long-term planning. Maybe they can't learn the way that humans do. And these problems have not been all addressed, but many of them are going away. So I remember long, long ago in the ancient days when we built GPD-3 two or three years ago, it seemed like GPD-3 was very bad at reasoning. It was very bad at...
[0:29:57] The kinds of problems that were just illustrated with ChatGPT where it had to reason through something complex, do math, and get the answer. Clearly, it's not all the way there yet because you've seen these mistakes from the last talk. But the sort of doubts that I had about this sort of general approach to AI addressing reasoning are generally fading as I think just
[0:30:20] Training larger models and basically developing very simple interventions seem to be addressing it. I also don't think that multimodal models are a problem. Obviously, people have talked about DALI and Mid Journey. I think there's kind of no bottleneck there. And I also don't personally think that there is any bottleneck preventing these models from starting to write longer term plans and then kind of execute them step by step. And so I think that my general view is that
[0:30:47] ChatGPT is a single moment on a very, very, very rapidly evolving trend. Computation is continuing to get less expensive. And although you might imagine that big tech companies are spending tremendous amounts on AI, they actually aren't. The kinds of models that we're seeing now maybe cost tens of millions of dollars.
[0:31:07] But there are all sorts of big science projects that involve hundreds of millions or billions of dollars that can be contemplated. So my general expectation is that this is kind of part of a trend. And I think it's going to be much, much more transformational than ChatGPT. And so I think that this is, I mean, there are a lot of lessons you can draw from this. You can be very excited or you can be very scared. I think I feel a mixture of both, but I think it's worth.
[0:31:37] It's worth thinking about how this can be scary, just in the sense that I'd like to believe in sort of this productivity bonus where everything will go well and people will just be more productive and it'll be great for everyone. But I think it is really uncharted territory in terms of how it affects society. And I think that if we're approaching, I mean, we have chess, we have we have AIs that play chess much better than people. We've had those for a long time. We had five or so years ago.
[0:32:06] AI systems that could play Go better than people. With ChatGPT, I think we're starting to get to a point where for some kind of unclear applications, some aspects of writing, we're getting to sort of human level performance, although we're also far away in the sense of these models are not reliable.
[0:32:26] They make mistakes that maybe people wouldn't make. But I do think that on this trend, there'll be more and more areas where these AI systems can do things that are sort of comparable to or possibly exceeding what people can do. And I think this is basically very scary. It's very uncharted territory. And I think it kind of invites a kind of deeper inquiry, not just about how ChatGPT
[0:32:54] Thank you, Jared. That was awesome. It's kind of cool to see AI kind of increasing along that sort of Moore's law, but it also does make me a little uneasy as well with the sort of implications and ethical implications as we get these larger and larger and more powerful models.
[0:33:23] A few questions in the chat. So let me go ahead and start asking those. And again, to the audience, if there's anything you'd like to specifically ask Jared, go ahead and put it in the Q&A. So the first one is coming in. How likely is it that we're going to see some sort of upper limit emerge on this? So you've got this kind of linear trajectory that you keep training these larger and larger models. And then is there a way of researching and training that's known right now that
[0:33:50] might be exploited to kind of break through that if there is some sort of cutoff? I mean, I don't think that there's any clear upper limit on, well, there's certainly theoretical upper limits on where we're going. So I mean, you can eventually run out of data to train on, for example, these models are sort of more data hungry than people are when they learn. But I think that we're sort of a few orders of magnitude away from that being a bottleneck. So I think that
[0:34:16] It's possible AI progress along this very simple trajectory will slow down in sort of five or 10 years, but I wouldn't expect it to sort of halt progress in the near future. I do think there are a lot of ways that you could get past this. For example, with reinforcement learning, it's less clear that there's any data bottleneck because models can explore and learn sort of from their environment. Right. Okay. Well, that's exciting. Exciting to see the potential that still exists.
[0:34:45] So kind of on the ethics side now, so as an AI safety organization, how does kind of improving these models so that you can train larger, more powerful ones with less work, less cost, less time, how is that going to possibly increase risks? So how does that approach with your sort of Anthropics mission of being a sort of a safe AI as you create these more and more powerful AIs?
[0:35:15] Yeah, so I mean, the techniques used to train chat GPT are at a basic level, kind of like five-ish years old, which in the AI field is a long time. The reinforcement learning from human feedback is that old. And I mean, the GPT models that haven't changed that much in the last few years were first developed three, four, five years ago as well. So I think that...
[0:35:41] There's a sense in which there isn't a lot of new stuff going on there, but things are scaling up. So Anthropic's approach to this was we actually wrote a paper about a year ago about how we trained a model very, very similar to JAT TPT called Claude, but we didn't release the model commercially or try to kind of create a lot of commercial interest or hype around it. Rather, we used it to kind of explore like,
[0:36:08] In what ways could it be red teamed? How could we further the research to make the model more honest and reliable? We are sort of going to be commercializing this model now in response, but we had a very much more cautious approach. We didn't want to sort of release these models very, very widely until they could be understood better. In terms of driving this capabilities progress, that was also sort of one of the motivations where we didn't really want to be accelerating the pace of
[0:36:37] AI development and deployment because we think that the sort of research around safety bias and the longer term implications kind of lags behind the rate at which this is kind of exponentially growing. But I mean, I think also as an organization that wants to be able to do that safety research on frontier models where it's most relevant, we sort of think that we need to be on the frontier to do that.
[0:37:04] Awesome. Most reassuring to hear that that is actually being thought about, and we're not just scaling these models without any thoughts of the sort of ethical implications and sort of the safety implications therein. So right now, I think it's time that we're going to switch over to our first panel session. So I'm going to switch over to my co-host, Ting Long, who can introduce our first panel.
[0:37:30] So now we move on to our first panel. So the first panel we have, for the next 20 minutes also, we're going to talk about how generative AI will impact the faculty life. So a lot of audiences, attendees here actually are faculty. And so that's why from a selfish perspective, we're going to talk about how it's going to impact us. So you have already heard two wonderful talks by Christine and Jared. So now joining us also we have...
[0:38:00] Professor Jenna Fry, who is a leader in art and design education for nearly 20 years. She recently joined the faculty of multi-disciplinary design program at Johns Hopkins University, where she explores problem-solving potential of mixing art such design with engineering. Also, we have Professor Tom Levencourt, who is a director of digital humanities and research assistant professor at Johns Hopkins University.
[0:38:27] with a primary appointment in the Alexandra Gras Humanities Institute and a secondary appointment in the Department of Political Science, where he is affiliated with the Center for Language and Speech Processing and the Human Language Technology Center for Excellence. So we have this perfect panel for this perfect topic. So let me start with asking the first question. So from a faculty perspective, have you been incorporating generative AI such as Charlie TPT and also new Bing search?
[0:38:56] into your daily workflow in both teaching and research. For example, I'm thinking about the aspects such as have you ever thought about how generative AI can impact the research and the publication process, including research design, writing, peer review, and also editorial decisions. Teaching perspective, how that influences the way you teach and also the way you interact with your students.
[0:39:26] So maybe we started with Christian. I mean, that question is alone half an hour to answer. So let me make two observations really quickly. The first one is I did update my course outline that homework assignments at the final exam have to be done without the help of AI. I think that one was easy, but it was relevant. On the research front, let me kind of start bullish because there are so many obvious downsides. But on the bullish side, I think the stimulation part is really good. Asking.
[0:39:55] Asking chat GPT for ideas and then worst case, rejecting them and working on something else. The stimulation I think I find is both ethically clean and easy to do. On many other things, I'm very concerned and confused, including the questions that are about to come. And so I'm happy to pass it back to you, Tingruang. I just want to reiterate, you don't have to address both team risks, just any aspects that you feel like has most got most significantly impacted.
[0:40:32] So what are the most significant impacts of ChatGPT on higher education? Yeah, from your own perspective as a professor, as an entrepreneur as well, but as a professor, how does that change your daily workflow? Yeah, so, well, I've been on leave at Anthropic, so I'm sort of out of the line of fire here. But I think that the, I mean, I think what I would expect to be the sort of biggest impact is that
[0:41:02] If the technology continues to improve, it's hard to take a clear stance. I think a thing that's more positive to start positive, as Christian did, is I think it can also be useful.
[0:41:16] My experience is I've interacted a lot with these models is that it can actually be fun to ask them to tell you about some subject and give you information. There is this issue of reliability where you're not sure that you can completely trust them, but there's some obscure area. They've read everything on the internet. And as you can ask them about sort of history of some particular period and they can kind of tell you interesting information. So I think that can be engaging. And I think that I would be optimistic that the technology
[0:41:45] Can be developed in the next year to a point where you could have specialized models that are kind of like tutors that you ask them about different areas and they engage with you in an interesting way. So I think that might be the way that I'd be most optimistic. Thank you, Jared. And maybe turn to Jenna. Maybe what it.
[0:42:09] Clark, Anthropic, Claude, OpenAI, AI policy, Import AI.
[0:42:38] A big part of, I think, the foundational part of higher ed and what we need to be helping our students make sense of. Thank you. And Tom, I guess you have been ready for this moment, right? So is this something you have been waiting for years or has it surprised you at all? I've certainly been waiting for years for it and thought it was going to happen eventually. I was very taken by surprise.
[0:43:07] How quickly it happened. In fact, I was gearing up to teach the first iteration of a humanities course introducing computational skills to humanists who are, of course, very driven by kind of close reading methodologies where you're doing very fine grained, extremely domain specific reasoning about primary sources and that sort of thing. And I was prepared to come into that class and just
[0:43:34] Go with my usual spiel, which was going to be along the lines of machine learning is wonderful for making these aggregate decisions and being able to see broad trends and get a blurry but very broad picture of things that humans are not very good at themselves. And then ChatGPT came out a few weeks before the class started. And so I was rewriting my first introductory lectures just to say,
[0:44:00] I think we might actually be pretty close to being able to do close reading using these models. Maybe this is a bit of a threat or an opportunity. And of course, it made the course even that much more critical, I think, for these students to be getting a bit of a taste of what computational methodologies can offer and maybe even taking a bit of the fear out of it by getting a bit of hands-on experience and understanding a bit of what is going on under the hood.
[0:44:26] And I also just want to reiterate what I think Jared put very eloquently, which is that we're really at just the first point on this, what is going to be a continued increase. And I think that a lot of the conversation right now is maybe not certain, being less productive than it could be because it's focused on
[0:44:48] What are the current state or what is the current possibilities with this particular model that we have in front of us right now? When really we are just, you know, this is one snapshot of an escalating process. And I'm just, you know, pretty much any objection that can be raised right now, I can see being overturned in a matter of, at this point, I want to say months because of how quickly these things are coming.
[0:45:10] Particularly when it comes to things like having fine-tuned models or smaller models that are more customized for individuals to use and more directed. So it's very exciting to me, honestly, but I can understand fear is mixed in with excitement all the time.
[0:45:28] So kind of taking that from sort of if you're implementing this in your coursework, but also now, what are sort of the ethical implications and ethical considerations surrounding the use of ChatGPT in teaching and research? And that's questions of accuracy, because we know ChatGPT has been known to create fake citations, but also intellectual property questions as well. What are your thoughts? Well, I might share.
[0:45:59] An example that I sort of already talked to some of the panelists about. So I have a student who's very interested in leveraging AI. And I teach a data visualization class. And we were analyzing a data set around candy preferences.
[0:46:14] The students had to come up with a pitch for a new candy based on the highest ranking candy qualities. And one of the students who really loves AI took all of the different categories and put it into an AI image generator, and it created this.
[0:46:31] Beautiful mess candy visual. And to me, like that felt ethically beautiful and wonderful. It was like it was it was helping him make a sketch. And that was how he made his sketch. I think where it sort of would go into territories of maybe being less comfortable is if, you know, I was doing an assignment where, you know, it was like more about your own original.
[0:46:59] Clark, Anthropic, Claude, OpenAI, AI policy, Import AI.
[0:47:26] You know, not riding on stolen valor, so to speak. Yeah, that's an interesting perspective, thinking of it as a participant. So, Tom, you, you know, talk about maybe using it for like close reading and things. What are your thoughts on this as, you know, sort of the AI as a self that, you know, you can, what are the ethical considerations?
[0:47:50] Well, when it comes to these kinds of humanistic research projects, one of the huge advantages that I see immediately from even its current state is its use when it comes to plumbing the depths of primary sources that we haven't been able to really get to in a way, because that relies so much on very intensive human labor.
[0:48:16] To kind of crawl through the depths of data and, you know, marshal it in the appropriate formats that we can draw comparable conclusions about this and that and the other thing. And it's really a problem that has plagued a lot of areas of inquiry for a very long time until you get into like the absolute most computational sciences. You know, it's always a problem of having comparable research and comparable insights and comparable materials.
[0:48:41] And so I see it as a very positive thing in that sense that it'll let us quickly explore things and let us build new interesting kind of digital artifacts that we can then kind of hang insights off of. But I mean, again, that view really depends on a fair amount of education of the users and the people who are going to be kind of directing this mechanism at
[0:49:07] into their domain. But I see it mostly as, I mean, honestly, maybe I'm just being an optimist about it, but I think that the potentials are, I see a lot of good in it. I see a lot of potential good in it. How about you, Christian? I know I saw one of your colleagues at Penn had actually developed an entire syllabus out of ChatGPT and was surprised by how well it worked. What are your thoughts on possibly using it, you know, the sort of ethical considerations using it either in research or in teaching?
[0:49:40] I think on the idea stimulation side, which again, that's sort of how I think about the syllabus generation, I think it's totally legit and I wouldn't even lose much sleep about giving the acknowledgements of the right references there. I'm very concerned about Jan's example in the world of art, right, in the world of art, images. Even in chat GPT, I'll give you an example for the fun of it.
[0:50:03] I asked JetGPT to write me a letter of recommendations for Christian Tebbish applying to the dean job at Harvard Business School. And it did a pretty good job. It's clear, though, that I'm not a particularly famous person. So you could sense from the words that it used, it picked up my language that I have on the website, because who else is going to learn about that? So I'm basically now applying the same logic for a scientific study. I'm asking the broader question. In some sense,
[0:50:30] I shouldn't reference Chet GPT. I need to reference a source that Chet GPT has been using the training data to make this statement. And I'm just concerned that we're going to lose giving credit and potentially money to the initial person who came up with that drawing that trained with Journey or Dali. And so I think I'm very concerned, especially I think on the tech side, it might be a little bit more traceable. But on the image side, this is this kind of fuzziness of art. I'm very concerned that we're not
[0:50:58] Paying the people who did the raw work as the artists, they were not paying them and giving them the intellectual credit they deserve. Thanks. Yeah. And Jared, do you have any thoughts? I know you're working more on Anthropic now, but so you've got that kind of interesting intersection between sort of working on the company side and then working on sort of the education side. Yeah. I mean, I think the ethical issues that I'm probably most
[0:51:31] Most concerned about are, yeah, I think whether we're sort of fairly compensating people for work and how this will affect employment. I think for things like generating a syllabus, it seems totally fine. I think that it can probably be used as a tool that helps students, but I think that a calculator is a very binary.
[0:51:57] Like whether you use a calculator is very binary. I feel like there are a lot of ways that I could imagine a student using ChatGPT for an assignment where they're not really cheating, but it's a lot blurrier. Like they might not be feeding the question in and getting the answer and transcribing it. They might be asking about the general subject, asking about other questions. And so I think it probably makes it harder to differentiate than it has been in the past. Thanks, Jaron. So let me just.
[0:52:26] I want to quickly refer to one question from the audience, which also happens to be a question we had for the panelists. How is chatGPT different from a calculator or from other tools? For example, math teachers in high schools, traditionally, or high school or middle school, are traditionally struggling with students using calculators to finish their homework. How is chatGPT different? And also this question from Professor Babich of Georgetown University. Why are we anthropomorphizing AI? Why start AI as a co-author?
[0:52:55] Would we cite a calculator or Excel or an MRI machine or any other tool as a cause? Anyone want to jump in? Well, I think the answer about why we're anthropomorphizing it is pretty straightforward, which is that it's capturing cognitive processes that have been solely the domain of humans at a very core level part of human identity for as long as humans have been around. So I think that one's pretty straightforward. Yeah.
[0:53:31] So anyway, I just want to maybe answer from teaching perspective how we're treating generative AI differently from what math teachers would treat tablets differently.
[0:53:44] Well, I think there's a calculator. There's also your best friend or your brother-in-law or your parents, right? And so I think you might say, like, there's a rule that I have whether I'm using a calculator or not. But I think if AI is somewhat on this continuum between your calculator on the left and a brilliant professor on the right, on that continuum, if you will, which rather than taking the exam yourself, since your uncle was a famous professor in physics.
[0:54:10] I think most people would say like, well, that person needs to be acknowledged, right? So I think AI is more of the latter than it's being a calculator. So I think in that sense, it's different from a traditional convex cell or calculator technology. So before we conclude this panel about, I mean, the part one of the panel about the faculty life, I just have a quick question for everyone, as faculty ourselves.
[0:54:33] Do you feel like generative AI has created more work or less work for us? Do you think it's going to be a blessing or it's going to be a curse? Maybe quickly just go around the circle. Maybe Jared? I mean, I think that it's sort of a reality. I mean, I think it could be mostly positive in the short term. I think I'm worried about its long-term implications.
[0:55:03] I think there's sort of a question of what is the purpose of the education? Is the purpose of the education to give students a deep understanding of a subject? Or is the purpose of education sort of to enable someone to accomplish a task in the outside world? And I mean, the fewer tools we give people to do something, I think the more we're focusing on a deep understanding and less on how would you actually do the task out in the world?
[0:55:32] ChatGPT and better models are going to exist. And so they're going to be useful tools. And so enhancing your own productivity is really just going to be a benefit for you in terms of being productive in, say, the workplace. But I think the key question is, is it detracting from your understanding of what you do such that you can't actually evaluate your own work? And Christine? I think, I mean, as Jerry put it, you have to start with the question of purpose. What is the job that needs to be done?
[0:56:06] What does a user need here? I think the discussion we're having is a little bit too technology-centered, right? I mean, if we think about the impact of technology in education, yes, let's talk about technology, but let's also not forget that we understand the customer needs, the student needs, the needs of the academic community.
[0:56:23] And I think, to be fair and mildly stated, we've done very poorly at thinking about these things. Our public education process has lots of weaknesses. Our teaching processes has lots of weaknesses and biases. And so I think we should keep those in mind and think about how we can use technology to make this better, as opposed to start with the technology and look for a need for it. Thank you, Christine. Reminded me of one of my colleagues saying that he used to go to a library.
[0:56:52] To find something really good, all the textbooks, where you can find the exam questions students can never find, right? I guess that's going to be. That's right. We cannot do any more. So that's it. So Jenna, with your overall perspective here.
[0:57:08] Yeah, I think I agree with a lot of what everyone else is saying is that I think it's dependent on the content for sure. And I think for in terms of whether or not it would make more or less work for me as a faculty member, I think that depends on how concerned I am about how they get their answers or the kind of ways that I use assessment in my classroom. I don't happen to assign a lot of papers or quizzes or things that chat GPT could help you.
[0:57:37] Thank you so much.
[0:57:56] Thank you, Jenna. And Tom?
[0:58:27] Put aside for a moment the learning a particular skill or hitting a particular threshold of certification type view of education. You think about the personal enrichment and fostering human curiosity and creativity side of things. Knowing that having a computer that can do a lot of these things is not going to take away from the personal enrichment that I will get from doing those things any more than it bothers me that my neighbor also reads Shakespeare.
[0:58:55] It doesn't mean that it's not worthwhile for me to go read Shakespeare too, just even if I come to the same insights from it. So yeah, I think that it's understandable that there's a lot of anxiety about this, but I think that some of it's going to fall away pretty quickly. Thank you, everyone. I'm going to turn to my colleague, Peter Lawson, who's going to move on to this part two of this panel.
[0:59:26] So I'd like to switch gears a little bit from the faculty perspective to looking at generative AI and chat GPT with respect to the student experience. And I'm sure a lot of the participants today are students and are kind of interested to see here how this is going to impact their lives.
[0:59:42] You know, ChatGPT has been viewed as sort of a threat and a boon both to higher education. I've heard it heralded as the end of writing, end of creativity, end of higher education as we know it. Others have just said it's hype, you know, it's prone to errors. It's just going to be a glorified calculator. But there's also, you know, deeper existential questions because I know higher education itself is sort of at a precipice. We have declining enrollment, skyrocketing costs, and there's this sort of justification that comes into play. You know, what is the value of...
[1:00:12] So, you know, there's these questions of how does ChatGPT fall into that and sort of what our institutional and our pedagogical responses in teaching. So to open it up to start, let's talk about, I know, you know, I've been seeing a lot of students submitting ChatGPT responses to exams. So what are your thoughts on student assessment in light of ChatGPT? How is that going to change?
[1:00:41] And maybe what types of assessments are being used. I think as I was sort of talking about before, I think I would imagine you're going to see a lot more project-based learning if it starts to become really monotonous for everybody to be turning in the same essay or the same sort of analysis. Then you have to shift to assigning things that can't be solved with that or can't be solved with that alone. So maybe there's more.
[1:01:14] Clark, Anthropic, Claude, OpenAI, AI policy, Import AI.
[1:01:46] And that's exciting. Yeah, I think that'd probably be more engaging for the students. Christian, what are your thoughts? I have seen that question a lot. I think it is the typical faculty question. Lazy us. We say like, well, look, we have to do something because now they're going to cheat us. We don't have to change. And that's the strategy, I think.
[1:02:10] Our job is to ask ourselves, as I said in my talk a little bit, like, why do we test in the first place? If this is about a skill certification, I like Jenna's example about the elephant. If it's about whether the student can draw an elephant, well, let's make the student draw an elephant and not have them use mid-January for that. If it is about engagement and having the student engage in creating something beautiful, that is much more of a project thing.
[1:02:37] Their technology becomes your friend. And we can actually create engagements that we could not create without the technology before us. So I think we have to be willing to question ourselves as opposed to just saying, let's get the toothpaste back in the tube. And it would be nice if we could teach the way that we have taught for the last 100 years. So I want to move on to the next question. And we don't have the other panelists to speak. But this time, the question is.
[1:03:06] How can ChessGPT help address challenges faced by students with diverse learning needs, such as those with learning differences or limited access to resources? I would also like to point out that, for example, in business schools, as we teach more and more data-driven courses, we realize that not everyone has the same level of programming skills. Do you see that generative AI could be an opportunity to...
[1:03:34] So this is actually one of the areas that I'm most excited about in terms of application. So I think that in Christian's talk at the very beginning, he mentioned the two facets to teaching where you have content and you have presentation. There's something similar going on here with ChatGPT and models like it.
[1:04:05] It will continue to evolve into where you have something that is both can have deep expertise about very niche topics or very arcane or Baroque topics, but also communicate, say, in multiple languages, multiple dialects to multiple communities. So, for example, right now, one of our postdocs is actually looking into how large language models could be used to help students from
[1:04:34] Clark, Anthropic, Claude, OpenAI, AI.
[1:05:02] You know, minority communities or dwindling language communities who don't have those advantages. And maybe it has a chance to equalize things there. It also, by the same token, has the possibility of moving a lot of the, I think, some of the most critical parts of education.
[1:05:21] Further back down the pipeline to the point of, say, high school or primary school education, where you might not have enough experts in particular parts of literature or science or history to adequately teach all the different nuanced.
[1:05:38] areas that you might want to say at the high school level. But maybe something like ChatGPT that has this kind of knowledge and has the ability to expand on that knowledge and explain it to an interlocutor could actually kind of reinvigorate that process and give us a better primary school education. So Tom, it's very exciting. You mentioned that ChatGPT can be used for that student application process. So which actually on the
[1:06:08] On the flip side, you could imagine schools could also use generative AI for this actual application review and mission process. So anyone has any perspective on this application and also a mission process potentially enhanced by generative AI? I think Jenna, you have. Yeah, I think this is an area where it concerns me, right? Because when it comes to evaluating a person's ability to be a gooder
[1:06:37] Not good fit for a college. It's already a process that's ripe for bias, right? And so if you're adding in, you're taking away the human interaction, I don't know, maybe makes it less biased, but there's a lot of, I think, evidence that.
[1:06:55] This technology holds its own bias. So I think I've heard about different schools using predictive AI models to see how good a student is going to be, a fit for their school, how long that they'll stay, and things like that. But I worry about the potential of that just further marginalizing already marginalized populations. So that's an application where it frightens me, where I think it's maybe not appropriate, but also...
[1:07:21] I don't know. I mean, I don't completely understand all of the ways that AI is learning. But from what we know right now, I think I would be nervous about it making predictions about who, it's already an exclusive environment, higher ed, and we already have problems with diversity and inclusion. I don't know how it helps that. Right. Right now, you can certainly just type an essay, ask a CHPT to evaluate, is this a good candidate or not a good candidate? But we don't really know the underlying algorithm, right?
[1:07:50] It's a very, very opaque. It's probably a lot more than- I hope so. Yeah, in-house everything already used right now. So anyone else want to talk to you about the admission process and potentially how that could be impacted by?
[1:08:07] Just make a counterpoint to that, a potential counterpoint. I'm not saying that this is actually going to outweigh some of the dangers of it, but the distance that gets created when you are using a computational sort of interlocutor for evaluating something or trying to do evaluation across a broad expanse of candidates or like a broad number of texts or whatever it is, computers are much, much better at applying rules consistently than humans have ever been.
[1:08:36] And along that same line, having that kind of distance from the model, from the algorithm, means that it's a bit easier for us as humans to triage out the things that we don't want to take into consideration. Obviously, that's a leaky process, and there are all kinds of ways for the model to get back at that information and so forth.
[1:09:00] Same goes for humans, evaluators. So I don't know how we calibrate that and measure the pros and cons of it, but I suspect there are ways forward that are going to lead to positive outcomes. Yeah, I would agree with that. And I'm motivated by hearing the different perspectives on the panel about ethics and morality and thinking about...
[1:09:27] Those concepts as, you know, the technology is being built out and grown. And I totally agree with you that there is so much that we can learn a lot from machines. But also, I think just taking, you know, wondering how they're learning to be so aggressive sometimes when they get something wrong, you know, in a conversation and how that like, you know, how to mitigate that to have a more humanity forward.
[1:09:54] perspective, especially when it comes to something like college admissions. Well, moving from the admissions side to sort of the student-centered experience and specifically around creativity and critical thinking, there's a question that came into the Q&A. So they asked regarding creativity, it sounds exciting and promising for these students and researchers to utilize generative AI tools to jumpstart. But for individuals, AI tools can increase creativity.
[1:10:23] But then aggregately, the question was, will AI tools lead to most people being more alike to each other, to sort of converging to the sort of sameness? And how would you respond to that as an educator? I mean, it's the Peter Thiel argument from zero to one versus from one to n, right? I think the argument that you can make is that
[1:10:54] Basically what chat GPT or similar technologies would do is create a convex combination of what already there is. And you come up with new permutations, combinations, but you never kind of make the step from zero to one. So I think there's some truth to that. I wonder if we can be pushing that process and explicitly force ourselves when we select ideas to think out of that box. But I very much like the observation that is part of the question here. So I think of that.
[1:11:23] More is a really good observation as opposed to a question. So I totally agree with that comment. Yeah, I think there's a lot of evidence that culture can be manipulated by algorithm. I think we have some pretty good evidence of that vis-a-vis Facebook, vis-a-vis Instagram, and how that changes and shapes culture. So I would imagine that there is absolutely some
[1:11:52] Clark, Anthropic, Claude, OpenAI, AI policy.
[1:12:09] I was thinking a lot about when students might use it versus not. And if they're turning in, let's say, a paper that's based on a chat GPT answer, well, maybe they just weren't really very motivated by that paper. And to me, that seems like a bigger crisis than that they used a chatbot to solve it. It's like, why aren't you interested? What's in the way of that? And how do we get you to be more interested in your education? So does that make sense?
[1:12:38] Yeah, that makes a lot of sense. I mean, I like that, you know, I like that this is sort of an inflection point towards improving the student experience rather than just a crisis. So coming back to maybe looking, you know, outward at the model rather than inward at the experience. So how is, given, you know, ChatGPT is trained on, you know, a huge corpus, basically everything on the internet, which means that
[1:13:07] It recapitulates any of those biases or the marginalization that's encoded in that data. So how does use of ChatGPT impact the representation and marginalization of these underrepresented groups? You're not looking at this tool critically. You're just using it and putting it into your classroom without thinking it necessarily about.
[1:13:33] Critically about the model itself. And then how can you address those concerns as an educator? Prompt engineering, I guess. I mean, I think that one way that we're using it to look at some underrepresented corners of the world is through kind of careful engineering of kind of careful probing for non-standard language.
[1:14:04] And kind of explicitly doing that. But that requires a certain amount of facility working with the model to begin with. Or kind of understanding like what the, I don't know, it's not a precise technical skill. It's almost more of an artistic skill of kind of knowing how to have that particular kind of conversation with this particular kind of cognition. And so I'm not really sure. I mean, I think it's...
[1:14:30] Probably too early to say much more about that than that this is a skill that needs to be cultivated if it's going to be used effectively in kind of like by individual educators in their own classrooms. Just sort of as a technical comment, there are sort of two phases to the training of these kinds of systems. They're sort of trained on, like as you put it, a lot of stuff on the internet, and then they're fine-tuned with
[1:14:58] Clark, Human feedback or or I mean in in our case a sort of AI feedback in various ways to sort of self correct, so I think there's two quite two considerations, one is like. Clark, Is there material missing from what the model knows that you think should be exposed and there's sort of a concern that students aren't seeing that and then there's a secondary problem is the model reproducing content that's that's harmful and I think that.
[1:15:24] The second problem of the model reproducing harmful content, I mean, it's still a problem, but it's significantly ameliorated by this sort of fine-tuning process that kind of tries to train the model to be more helpful, honest, and harmless, although it's not entirely successful. The absence of certain kinds of information in the model is certainly not fixed by that. I think then there's a secondary question of, does this process of becoming less harmful address bias? And I think...
[1:15:54] We have some work on further processes that we think helps with that. But I think this basic process doesn't necessarily. So in some ways, that's still there. So that's just kind of a technical problem. Comment about kind of how the training works. I do think these models are, it's not as simple as the model is just going to reproduce every harmful thing on the internet at this point. But on the flip side, it's definitely not completely.
[1:16:21] Completely addressing all such concerns either. Thanks, Jared. Yeah, that might have been a gross oversimplification on my part, but I appreciate the sort of technical kind of background on how that's actually being done. So I think one of the really interesting points made earlier was that you typically provide relatively uniform answers. If you do Google search, right, you typically, I imagine most people look at the first 10 or 20 results.
[1:16:56] Before deciding which one to click, whereas with chatGPT, it's more like a closed loop system. I'm just wondering from students' perspective, this seems to be now, of course, they can still use different prompts, but given similar prompts, people are getting pretty much similar kind of findings. Does that create a different learning and risk experiences?
[1:17:26] Yeah, I think for sure it points, it maybe points out something that's been a problem in education for a while. And that is that if all of your problems can be answered with one similar question, and maybe it's not that good of a problem, or it could be more interesting or more robust. But yeah, I think that's, I just lost my train of thought. But yeah, going back to what we were talking about earlier with like how this can be
[1:17:56] Like maybe harmful to marginalized communities. I feel like the thing is we want to make sure that we have to have really good conversations about skepticism and critical thinking and putting things into perspective so that that's the part that's missing a lot is the context. I think when you Google something, if you Google something and you get a list of 10 results, you're at least.
[1:18:22] You kind of get the sense that there's maybe a different perspective, and then maybe you could go in and find something. But this idea that you can just put a problem in, get an answer, that's a harmful construct, right? And that's something I think we would need to work on as educators, is that that's just a starting point to then ask 10 more questions. Right. And I think people point out that even before CheckGPT, there are services.
[1:18:49] You know, you pay $3, $5 to write an essay for you, right? Right. Yeah, exactly. It's not, none of it's new, right? There was, you know, and I was thinking about that too, like, you know, back in, when I was in college and like teachers were like, don't use Cliff's notes, you know, that was like the big thing was, you know, and, but, you know, maybe you don't use that to, you know, write the whole paper, but maybe it helps you understand something in a way that you didn't before. And it makes you more curious to then read more and ask more questions. Sometimes you just.
[1:19:18] As a student, you need help finding another way into the material. In that case, there's no wrong way into the material, but it still relies on that ability to be critical about what you're reading and what you're taking in and how you know to verify your sources. That's the fundamental issue more than the technology, I think. So, Kristen, you actually talk about it.
[1:19:42] The National Cranberry case, which is very famous, a classical teaching case that we use in schools, is also a very local case, right? So if you Google it, you get millions of results, a lot of solutions posted online already. But I think the difference is that as professors, we change the numbers. The students still need to do quite a lot of thinking, a lot of homework to really understand the logic underlying that case. So do you see...
[1:20:10] The emergence of generative AI poses a threat such as that we can no longer do that, so we have to do newer settings, newer cases. No, I think there is a slight mistake that you made in your initial assumption, which is the notion that the device is closed loop and deterministic. Actually, if you play around with JetGPT, you'll notice it's highly stochastic.
[1:20:37] One option is you can press generating a new answer and oftentimes get a very different result. So kind of an easy question that I like to play with is the old consulting interview question of how many ping pong balls fit in a Boeing 747. And so try this with chat GPT and try just ask for a new answer every time if you get answers ranging within the effect of 100 from each other. So I think.
[1:21:02] You have still this randomness, which has to do with, ultimately, this generation part, which is a stochastic component in the algorithm, if you will. So I'm not that concerned that we're all converging on the same solution to the problem. I mean, I think some of the other issues that you raised, I'm clearly concerned about. But that thing that we're all going to have the same, we're asked about the same question, all gets the same answer, I'm a little less concerned about. That's on top of the worry list.
[1:21:33] And now we have talked quite a lot about business education and humanities education. Now actually questions from the audience about engineering education. So I actually want to make the broadly applicable to hands-on kind of education. For example, in business schools, we have a lot of projects, but a lot of really hands-on, experiential learning experience kind of education.
[1:22:00] In medical schools, I imagine a lot of things cannot be purely theoretical. It has to be hands-on. So how is generative AI going to change the way we design hands-on education? So maybe, Jared, and so I know you said you're a theoretical physicist. Do you think generativity has the potential for disrupting experimental physics in some way?
[1:22:34] Well, I don't know, like, I don't know whether I should, how crazy I should be. I mean, so I, I mean, I'm very, very sort of self skeptical. And I think that like, whenever you sort of predict something really unusual that's going to happen, you should, you should continue to ask yourself whether you're crazy. But I mean, I do generally just see this trend that AI systems are going to get better and better. I think
[1:23:02] From my point of view, it's like, oh, from the perspective of the public, suddenly ChatGPT has crossed a threshold where people are impressed. But like six months ago, you could have gone to OpenAI's website and used InstructGPT. InstructGPT will do almost as well at a lot of the tasks that people are excited about with ChatGPT. Just somehow that didn't have enough exposure or it wasn't in a dialogue format, et cetera. So basically what I see is just this trend.
[1:23:29] And it just seems very plausible to me that if the trend continues, that essentially most knowledge work will be doable by these kinds of systems in five to 10 years. I think companies are going to invest maybe 100 times more in the most powerful AI system in two to four years than was invested in ChatGPT. Their algorithms are going to improve a bit.
[1:23:55] They're going to learn lessons from what's going on now. And so I really just don't see it just seemed plausible to me that I mean, for example, I would I would guess that all of these questions that the chat GPT got to see on this year, a model sort of available sometime next year. We'll just we'll just ace all of those without any problem. And so and so I do sort of see this as as as continuing. There's also work.
[1:24:19] It's very obvious if you think about it, there's work out of Google that has now been done where Google uses large language models to control a robot. Because if you have a large AI model that can understand language, it can understand images, then it can sort of see what the robot's doing and say, robot, do this, robot, do that, instead of usual sort of robotic controls which are more based on more primitive AI systems. So I think that...
[1:24:47] You can easily imagine a world in five to 10 years where you have AI systems that are kind of comparable to humans at a lot of tasks, maybe better at some, and that are being asked to say, observe through a camera and provide instructions to a robot and then do work, could even involve things like experimental physics. I think in the near term, I would expect more from just like being able to do knowledge work.
[1:25:11] AI systems that can use web browsers, that can navigate using a computer. But I do think there's a very good chance that this is going to happen in my view, even though it sounds really crazy. I think there's also a good chance that I'm just wrong and that there are going to be bottlenecks. This isn't going to happen. But I think it's at least likely enough in my view that it's something to be prepared for and to be thinking really hard about how that affects society, what the risks are, et cetera.
[1:25:39] Thank you, Jared. I actually made this bold prediction three days ago that changeability will make theory more valuable than before because it does not really produce new theory. I do want to hear maybe what the panelists think about this prediction. I mean, just to quickly weigh in, I don't think that that will be the case. I think it can produce new theories. I mean,
[1:26:07] I think a common reaction is to think that the thing they're missing is the human spirit, and the human spirit is tied to creativity. And I think that these models can be creative. And I mean, people who have experience with generating theories, your theories are mostly wrong. So I think it's not just generating theories, but it's sort of finding the right theory and building on it. But I don't see a reason why these systems can't do that.
[1:26:37] Thank you everyone. I think we are about, we are approaching the end of today's session and thank you everyone for your participation. But before we close the panel, I do want to ask maybe, you know, just every panelist just to give your overall perspective from today's conversation or anything else you feel like you didn't get a chance to discuss before or feel free to respond to any questions as well.
[1:27:06] So maybe with starting from home. I think the only thing that I might have wanted to talk about that we didn't touch on was my excitement for what's going to happen when we get smaller, distilled, more compressed models that are more accessible to academia, for example.
[1:27:32] And that we're able to fine tune ourselves using the kind of limited resources that we have at the university level and begin to actually really dig into what these models can do for human understanding of various domains. Thank you.
[1:27:50] As you just where my head is right now, I have so many questions. And one of the things we didn't talk much about, but I'm curious to learn more about is the effect of like interacting with ChatGPT from like a disabilities lens. Like how does it work with screen readers and how like other kinds of disabilities and what that access is like. That's just something I don't know a lot about and I'm curious of. So that's what I'm ending on. I want to know more about that. Thank you. Jared? Yeah, I mean, I think.
[1:28:23] It really would be beneficial for more people in the academic community to kind of think hard about the implications of these systems broadly, because I don't know where it's heading, but I think it's something to be excited, but also concerned about. Thank you. And Christian? I think it goes back to the philosophical question of what is your view of good life? And if the vision here is that our students are writing college applications that are written with CHET-GPT, the university applies CHET-GPT to admit students.
[1:28:55] And then we have AI tutors teaching these students. I mean, if that's your vision for the future, I don't wanna be a part of that, but I think we can do better than that. And so I think that's what we should be working towards. Well, thank you everyone for this amazing panel. Jenna, Christian, Tom, Jared, thank you for taking the time. Thank you for kind of giving us your perspectives. I'm not sure if I'm more excited or more nervous about the future.
[1:29:29] of higher education and chat GPT and generative AI. But I have a lot to think about and I'm sure our audiences do too. So everyone, if you could, you know, clap hands or, you know, in the chat and thank our participants, our panelists today. Thank you. Thank you, everyone. And this has been a great inaugural session of the Data Intensive Social Science series and Love Data Week. And we appreciate everyone that came.
[1:29:56] And again, thank you all. And have a wonderful, wonderful rest of your day or wonderful rest of your week.