{"url": "https://www.youtube.com/watch?v=oq7NpSoi9kc", "page_url": "https://www.youtube.com/watch?v=oq7NpSoi9kc", "title": "The Marketplace for Intelligence", "show_or_channel": "MIT CBMM (MIT150 Brains, Minds & Machines)", "date": "2011-05-03", "duration_s": 8962, "type": "panel", "speakers": ["Demis Hassabis", "Tomaso Poggio", "Peter Norvig", "Amnon Shashua", "David Ferrucci", "Andrew Blake"], "confidence": "medium", "notes": "MIT150 closing panel \"The Marketplace for Intelligence\"; Hassabis (then Gatsby/UCL) is 1 of 7 panelists and gives a short talk; 149 min total.", "people": ["demis-hassabis"], "id": "2011-05-03_mit-cbmm-mit150-brains-the-marketplace-for-intelligence", "audio_duration_s": 8961.567375, "worker": "jc-w13", "transcribe_s": 219.3, "model": "large-v3", "segments": [{"start": 11.15, "end": 34.18, "text": "I'm especially proud of this panel and thankful to the panelists for allowing us to put together a group of people that perhaps is even more interesting than each of its very interesting components. The panel idea is to go back to the main theme of the old symposium, which reads as follows."}, {"start": 34.18, "end": 64.69, "text": "Over the past 50 years, research in AI and machine learning has led to the current development of remarkably successful applications, such as Deep Blue, Google Search, Kinect, Watson, and Mobileye. While each of these systems performs at human level, or better, in a narrow domain, none can be said to be intelligent. So at the end of the symposium, we are going back to this theme."}, {"start": 64.69, "end": 95.38, "text": "And I want to point out that often successes in science are measured in terms of publications or prizes given by other scientists. Now, this is fine. But from the point of view of machine learning, that smells of overfitting. And it has dangers. So the ultimate test of success for science is really to have an impact over the long term, of course, on real life."}, {"start": 95.38, "end": 125.15, "text": "And for instance, on the marketplace. And so this panel is intelligence and marketplace for intelligence. Now, this close interplay between pure science and application is actually one of the genes of MIT. This is not the way our provost put it in a recent article, but he said, MIT pursues and values the most abstract, curiosity-driven, fundamental research"}, {"start": 125.15, "end": 155.34, "text": "And the most applied, market-oriented innovations. And so, again, this panel is very much in this spirit. You have to show that the science you're doing can lead to things that work in the real life. Each of the panelists today represents one of the successful systems in the engineering of intelligence, which I listed in the theme of our symposium."}, {"start": 155.57, "end": 183.42, "text": "They will cover state-of-the-art achievements in narrow domain of intelligence, as I said, from Google Search to Watson to Kinect to Mobileye to medical devices and to future approaches. And each of the panelists also represent a company from well-known ones, such as Google, IBM, and Microsoft, to less known but successful ones, like MediNol and Mobileye."}, {"start": 183.42, "end": 211.76, "text": "Two really baby startups, Navya and DeepMind. So behind the companies and products and ideas, there are, of course, great people. And I'll introduce each one of them in turn. So first, Peter Norvig. I find Google search magic is better than any team of librarians I ever had when 30 years ago has changed so many things in the way we do things."}, {"start": 211.76, "end": 239.95, "text": "It has changed the way I do research. I look for what has been published before, what is known, before I start really trying to solve a new problem. Peter is director of research at Google. He's a fellow of the American Association of Artificial Intelligence. He has been faculty in various universities. He's the author of"}, {"start": 239.95, "end": 269.86, "text": "A very successful textbook, but also of a wonderful PowerPoint presentation, the Jettisburg PowerPoint presentation, which I recommend every one of you to look at. David Ferrucci. Let's see. I sent this email on February 14, 2011, to David. I said, David, sorry for the many emails. I'm watching the competition tonight. This was the Jeopardy competition."}, {"start": 269.86, "end": 297.49, "text": "I'm watching the competition tonight in a very crowded MIT classroom. Could we invite Watson to come to our workshop? And David answered almost immediately. He said, not sure if Watson can join me, but I plan to attend. I hope he's still good enough. Watson steals the show. Thanks. David is the lead researcher and principal investigator for the Watson Jeopardy project."}, {"start": 297.49, "end": 310.61, "text": "He has been a research staff at IBM Watson Research Center since 1995, where he led a team of 28 researchers and software engineers specializing in various areas critical"}, {"start": 310.61, "end": 333.39, "text": "to AI, software architecture, natural language processing, and so on. And I want to say that even more than Watson, I was impressed by the organizational feat that David pulled out of leading and integrating in Watson so many different fields and people in his team. Andrew Blake."}, {"start": 333.39, "end": 360.82, "text": "Kinect is another magic application that just came out. It's this box that used to play games with the Xbox, the Microsoft Xbox. It recognizes your gesture, the poses of your body. And the brand of Kinect, the machine learning technology which is inside, comes from the team of Andrew Blake, who is the managing director of Microsoft Research Center in Cambridge."}, {"start": 361.01, "end": 390.32, "text": "Prior to joining Microsoft, Andrew was in academia for 18 years, lately on the faculty at Oxford University, has published several books, was elected fellow, among other things, of the Royal Society a few years ago, is an old friend, a real pioneer in computer vision, and among other contributions, he pioneered the use of probabilities and machine learning in computer vision, which really transformed the field."}, {"start": 391.15, "end": 414.48, "text": "Amnon. Amnon Shashua, I already introduced him yesterday, so I'll be brief. He holds the Sachs Chair in Computer Science at the Hebrew University in Jerusalem. He received his PhD in Computer Science at MIT, working on computational vision, where he pioneered work on the most mathematical"}, {"start": 414.48, "end": 439.7, "text": "Part of computer vision, which is really geometry, multiple view geometry, bounding with this part of mathematics called algebraic geometry. I'm proud to say that Amnon was a student and a postdoc of mine. He's a great friend and a great colleague. And I said yesterday, I'll repeat today, that Mobileye is on top of my personal list of"}, {"start": 439.7, "end": 464.38, "text": "The successful applications of machine learning and computer vision in just a very recent past. Demis Assabis is a Wellcome Trust research fellow at the Gatsby Institute in London and a former well-known computer games designer. He was a child chess prodigy."}, {"start": 464.38, "end": 493.65, "text": "He created the classical theme park for electronic arts, one of the best selling games at the time. And after selling a game company, he started a PhD in cognitive neuroscience at University College in London. And he recently co-founded a new high tech startup to conduct research into general AI techniques. Among other things, he has won"}, {"start": 493.65, "end": 516.78, "text": "The International Mind Sports Olympiad for five times, I think. He's a visiting scientist in my group, a wonderful dreamer of very real dreams. Vikash. Vikash is a co-founder and the CTO of Navia Systems, which is a startup building probabilistic computing machines."}, {"start": 516.78, "end": 546.4, "text": "He received his PhD at MIT in CSAIL at BCS, was a student of Josh Tenenbaum. His PhD dissertation received the MIT award for the best dissertation in computer science in 2009. I still look at him as one of the bright students that make life a challenging fun."}, {"start": 546.4, "end": 571.79, "text": "At MIT, for me, is a great scientist, an entrepreneur, trying nothing less than changing the current paradigm in computation. You will hear from him how this will happen. Coby Richter. Coby is one of these very rare people. I know of two of them."}, {"start": 571.79, "end": 600.67, "text": "Including him, who is very good at everything he does or did, from flying fighter jets, to all kinds of sports, to art, to science, to business. He's a gifted multidisciplinary researcher in computer science and neuroscience, with a PhD in, broadly speaking, neurophysiology from Tel Aviv. He started recording from visual cortex while at MIT, about the time he arrived here in 81."}, {"start": 600.67, "end": 625.26, "text": "Among other things, he was helping me negotiate with tough guy types used car dealers in Boston at the time. And at the same time, he was programming on the least machines in my office at the iLab and starting a company with Shimon Ullman in the field of computer vision called Orbotech, which is still a world leader in the field of automated visual inspection machines."}, {"start": 625.26, "end": 657.95, "text": "We went on to develop new designs for coronary stents, helping to create a dramatic transformation of therapeutics in cardiology over the last decade. So with this, I will ask Peter to give his presentation, around 10 minutes. OK. Welcome all. Now, one of the undercurrents of this conference has been let's"}, {"start": 657.95, "end": 687.39, "text": "Compare the good old days to today and see how things are different and this idea that maybe we missed something and maybe things were great back then. What I want to talk about, and maybe some of the others will talk about, is the great things that are going on right now. And yes, there is an opportunity to do more in the future, to change what we're doing. And there's lots of good opportunities in the future. But there's also lots of great things going on recently and going on right now. And since this panel is supposed to be about marketplace,"}, {"start": 687.39, "end": 714.26, "text": "I'll focus on the market impact of those great things rather than the scientific impact. So let's try to estimate the economic impact of Google. And by this, I'm exaggerating and using Google to stand for more than just Google itself and stand for the whole infrastructure of the web, because what Google does is enable you to get to the web in the same sense that trucks have a big"}, {"start": 714.26, "end": 743.82, "text": "And then, of course, there's the other side of the coin, which is that we don't make the goods ourselves. It's the same thing for Google. It's a transport system. And Hal Varian took his tack at this at the Web 2.0 conference recently. And you can look up his numbers. My numbers are going to be slightly different, because I'm going to do the whole world, whereas he was just doing the US. But the main thing you should note is that there are really big error bars. And any of our guesses can be off by big factors."}, {"start": 744.56, "end": 766.16, "text": "Now, the first thing to note is that we have advertisers. They pay us some money. They get some value back. And we can calculate that the value back to them is about seven times what they spend. So 88% of the economic value of Google is going back to them rather than staying with us. And that works out to about $210 billion per year."}, {"start": 766.74, "end": 786.3, "text": "Then the next thing is, what about the values to the searchers rather than to the advertisers? And there's a few trillion interactions per year at the various Google properties. But it's hard to tell the economic value of that. If you watch a YouTube video of a kitten,"}, {"start": 786.3, "end": 815.22, "text": "How much is that worth to you? Well, we can estimate it by saying how much are you willing to pay for a first run movie and divide by some constant. But maybe the economic impact is more negative rather than positive, because you should have been doing something useful. So these are wild estimates. But let's say something on the order of $7 to $73 billion for YouTube videos, depending on if you're willing to pay a penny or a dime to watch one."}, {"start": 815.41, "end": 843.39, "text": "And then somewhere in the range of $37 to $370 billion for the searches at Google. And I'm only allowed to say that there's more than a billion searches per day at Google. We haven't updated that number in a while, but that's the official number I'm allowed to use. And how much is the search worth? Maybe it's worth a dime. Maybe it's worth a dollar. If you have cancer and you discover the treatment that works for you on Google, that's worth a lot more than a dollar."}, {"start": 843.39, "end": 875.1, "text": "If you use Google as a bookmark bar and you type in Yahoo and we send you to Yahoo, that's probably worth less than $0.10. OK. And then the value to websites. And there's a direct economic impact that there are publishers that show Google Ads on their websites. And we can measure that at $54 billion per year. But then there's the indirect of all the websites that get"}, {"start": 875.1, "end": 904.1, "text": "Clicks in advertisements, or not advertisements, but get redirections from Google for free. And maybe that's $250 billion. It's hard to tell. And so the bottom line is that it works out to somewhere around 1% to 2% of the world GDP has been accelerated through this access to information. So that seems like a good economic impact. What else is the impact of"}, {"start": 904.1, "end": 931.89, "text": "Artificial intelligence, of machine learning, of statistics, of information technology. One is in experimentation. And Ronnie Kahave from Microsoft Research talks about this. Quick, of these two, which one do you think would be better at getting you to click and make some money for MSN real estate? How many think it's A, raise your right hand? How many think it's B, raise your left hand? People aren't quite sure."}, {"start": 931.95, "end": 954.05, "text": "It turns out A was 8.5% better. Now look at these two. Which one is going to make you buy this product? Right hand if you think it's A, left hand if you think it's B. And this time, A was 60% better. And you can play this game, and you don't have to do more than three or four before somebody's got it."}, {"start": 954.05, "end": 981.34, "text": "At least one. Everybody's got at least one wrong. And the idea is that your intuition just is not reliable. And you've got to run these experiments. And having all this ability to do it online lets you do that. And then having the data analytics lets you make sense of that in a larger sense. And there's lots of places now where data is available. We can do more analytics. There's all sorts of intelligent applications that have been enabled. So speech recognition, machine translation."}, {"start": 981.34, "end": 1011.74, "text": "Self-driving cars, automated spam detection, genomic analysis. All these things are coming through. New breakthroughs in machine learning, new capabilities for data analytics, and so on. Communications is important. And there's applications like question answering and dealing with email and information overload, and lots of opportunities there. And then in education. So the US spends $120 billion a year."}, {"start": 1011.74, "end": 1036.3, "text": "And the US is a quarter of the GDP, and maybe half of the education comes there, and half the citizens pay for themselves. So it's a trillion dollars a year in education. And currently, electronic learning is only $50 billion. And so I think there's a big gap there where it could be doing a lot more. MIT has 10,000 students, but the MIT OpenCourseWare website has 70 million students."}, {"start": 1036.3, "end": 1063.41, "text": "And so what's the impact there? I don't think we have quite measured it yet. The Khan Academy has 35,000 students a day, and the University of Phoenix has half a million students. That's a distance learning university. Here's the Khan Academy. It has a bunch of videos that are online that are available for free, and students watch them and learn from that, either as a substitute or an augmentation to their own learning."}, {"start": 1063.41, "end": 1088.75, "text": "And here's two examples, one big and one small. The MIT OpenCourseWare, thousands of course available where you can follow a whole course all the way through. And then we're beginning to see applications as the e-readers come online of things like the Amazon Kindle has this sharing of highlighted passages. So you can highlight a passage, and you can choose to share it. And then you can see what other people think is important, and you can learn from that. And so according to Amazon,"}, {"start": 1088.75, "end": 1110.69, "text": "The most highlighted, the most important passage of all time is Jane Austen's It's a Truth Universally Acknowledged quote. Now, of course, in part, she's helped because her book is out of copyright. And so it's free to download compared to the other ones that you actually have to buy. And she can't actually benefit from this feedback being dead."}, {"start": 1110.69, "end": 1138.08, "text": "But for other people like me, being a textbook author, wouldn't it be great if I could get this feedback to say, on this page, this part worked, and this part didn't work, and now I can go and fix it. And that's being enabled now. And then in conclusion, let me just say that there's lots of big markets out there, some for which we've already had a huge economic impact on, and others for which there's just money lying on the floor. And if you pick it up, you get to keep some."}, {"start": 1138.08, "end": 1167.02, "text": "And like Google does, you get to share most of it with the world to make the world a better place. So go out there and pick it up. David. So thank you very much. It's really an honor to be able to get a chance to speak to you today."}, {"start": 1167.02, "end": 1186.61, "text": "Yeah, I want to make a couple of points. I don't have those kinds of numbers that you've just seen about the economic impact of Watson. Of course, we're very excited about it. We have a tremendous interest from industry, from lots of customers coming to IBM. We've sort of captured the imagination of the broader audience. And all of this is having very positive."}, {"start": 1186.61, "end": 1204.19, "text": "I want to talk about three main points, which is during the course of this project, we learned a lot. And one of the things we learned about was the opaque perceptions of AI and computers that we found as we were communicating to the general audience about"}, {"start": 1204.19, "end": 1228.32, "text": "You know, about what Watson does and how it does it. And I wanted to share a little bit of that with you. I also wanted to share a little bit of how we focused on engineering and engineering for science and what some of the commitments we made to succeed with building Watson. And ultimately, you know, the importance of the right interaction can really change the business. And I put interaction in quotes there because it's really about how you interact with computers and how you communicate."}, {"start": 1228.32, "end": 1250.64, "text": "And with the user and with the general audience and how powerful that can be in impacting business and the understanding of how to apply technology. And I got an opportunity to see a couple of the panels before. And I learned two things, I think. One was that brains are really sexy. And a warning, there's not a single picture. I couldn't resist. There's not a single picture of a brain in this presentation."}, {"start": 1250.64, "end": 1277.6, "text": "But I think the other thing I learned is that in this few just 10 minutes I have, that these audiences can absorb someone speaking really, really quickly. So that's what I'm going to do. So first off, these bipolar impressions we got. I actually did a radio interview where someone asked me pretty much exactly what was there. OK, so you put the questions and the answers in. And then when the clue comes up, Watson speaks the answer, right?"}, {"start": 1277.6, "end": 1300.03, "text": "And I was like, no, we don't know what the questions or the answers are ahead of time. And the host said, well, then how do you do it? And I said, exactly. So true story. But you have very different impressions. On the other hand, we would get people who would see some of this and say, oh my god, it's frightening. It's Skynet. When's Watson going to enslave us? So you had this."}, {"start": 1300.03, "end": 1329.33, "text": "Complete opposing things. Even within IBM, people looked at this initially and thought, gee, it's too easy. In fact, some people thought it was name that tune. Other people on the other end of the spectrum thought, this is too hard. This is pure folly, and you're just going to fall on your face and embarrass us. So it was interesting to me that you had these bipolar impressions about what was going on. And this was really not just a study of the audience and fun. This really affected to what extent IBM, my company, was willing to make an"}, {"start": 1329.33, "end": 1358.54, "text": "I had a unique opportunity to talk to many, many former Jeopardy! contestants who came and sparred against the computer. And I had this challenge. It was amazing that they were more interested in how the computer did the strategy or how it"}, {"start": 1358.54, "end": 1387.76, "text": "I'm going out on a limb here, but they're unconscious often of how they actually answer the questions. It happens in a blink of an eye for them, what's going on. And what they are conscious of is how they computed a bet, or how they did the math, or how they did the strategy. And of course, that was largely the easy part. And I don't want to undersell some of the cool things we did there, but largely the easy part with regard to the technology and the effort, but something they really"}, {"start": 1387.76, "end": 1415.7, "text": "So one of the things I did was talk about what's easy and what's hard for a computer. And you had a long time now to look at that. Anybody know the answer? No. So I bet you weren't even sure if it was less than or greater than 1. So this is easy for a computer. Looking through tables of information, here's a pseudo SQL query, is easy for a computer. And this helped communicate that this is how computers work. And it's a paradigm we're all sort of very used to."}, {"start": 1415.7, "end": 1439.44, "text": "And even people who are not experts in computers use spreadsheets. They use databases. And they know if you put your serial number, you can look up your address. So this is kind of what they expect when they see a computer just getting a question and answering it. They're expecting you're just looking it up in a database. And what I found was that, at the speed of light here, what I found was what helped communicate the challenge that we have with natural language processing and how to control"}, {"start": 1439.44, "end": 1468.75, "text": "Contrast that with database tables was this notion of what goes on with, here you have this table, and you wonder where somebody was born, and you could look it up in structure information. But what if you had the same question, and you had no idea where to go or how to answer it, how to interpret it, and you had to read where was Einstein born and infer that from that text? One day, from among the city views of all, Mato chose a water culture sent to Albert Einstein as a remembrance of Einstein's birthplace. You have some cues there, but now you have to understand there are people, there are places, there are relationships."}, {"start": 1468.75, "end": 1479.79, "text": "Here's another example. X ran this. Now, sure, if we knew that question was answered by that table, we can look it up. But instead, if we had, if leadership is an art, then surely Jack Welch has proved himself a master painter at his tenure at GE."}, {"start": 1479.95, "end": 1509.55, "text": "We have a slightly different problem here. So with some confidence, we may deduce that Jack Welch was a painter at GE. But that, of course, is not what was intended by the question, nor really the answer. And this helped people sort of scratch their heads and go, oh, I'm starting to understand really what the challenge is. And I think what was neat about Jeopardy from a scientific perspective was it helped us tackle some key points. The broad open domain, so you can't anticipate what the questions were going to be about or exactly how they were going to be phrased. Complex language. And you had to have a very, very high"}, {"start": 1509.55, "end": 1534.82, "text": "But moreover, the computer had to know what it knew in the sense that it had to know it was right. Because every time you say you know an answer, and you get in there and you buzz first, if you get that question wrong, you lose the dollar value associated with that question. So there was risk in answering with the wrong answer. So it had to compute, essentially, a probability, an accurate probability, as accurate as we can get, that you got the answer correctly."}, {"start": 1534.82, "end": 1564.11, "text": "You got it correct, so you could take that risk and buzz in. And you had to do this very, very quickly. And this was really the foundation of what the engineering feat was. And it involved starting with really, really crisp metrics. And one of the most important metrics we had was represented by this Winners' Club. And I can't tell you how important having a crisp metric was for both the science and the engineering of the problem. So we didn't start off with this general notion, we're going to build this big AI, it's going to be really smart, and there's going to be Maxwell equations for doing question answering."}, {"start": 1564.11, "end": 1585.52, "text": "We sat there and we were very, very practical. In fact, someone came to me and said, I'm a cognitive psychologist and I really, really want to be in your project. And as much as I'd love to philosophize about cognition, I had to get really practical really quickly because I had to meet with the senior VP quarterly who said to me, Dave, you're mucking with the IBM brand. Are we going to win?"}, {"start": 1587.57, "end": 1611.76, "text": "So this was very much an engineering feat. That baseline was where we started with the QA system. And we had to drive that confidence curve up so it was basically splitting right through that or cutting right across that winner's cloud, because that's where winning players were performing. So if they were confident enough and fast enough to get a chance to answer 50% of the questions on a board, they were doing between 85% and 95% precision."}, {"start": 1612.18, "end": 1639.41, "text": "I'm going to skip some of these slides because I don't have the time for it. But I want to get to a few guiding principles. And this is kind of, for me, where a lot of the AI theory sat for me. So we made a couple of key commitments. Large crafted semantic models will fail. Too slow, too narrow, too brittle, too biased. So we needed to acquire and analyze information from as-is null sources. So we consumed things like Wikipedia and many other encyclopedias, dictionaries of this size."}, {"start": 1639.41, "end": 1660.48, "text": "They saw our entire plays, books from Gutenberg Press, the Bible, Shakespeare, and you name it. We had the expectation that there was no one algorithm that was going to be able to do question answering, but that intelligence would come from many, many diverse algorithms, and that each were going to react to different weaknesses in the system based on"}, {"start": 1660.48, "end": 1690.26, "text": "Doing error analysis and repeating and improving the core NLP that was in there to attack each problem we would start to see and generalize from. And the relative impact of an integrated metric must be automatically learned. In other words, there was no presumption that if you got your parser from 95% to 98%, or if you got your information extraction from 75% F measure to 80%, that this was necessarily going to impact the end-to-end metric of the question answering system."}, {"start": 1690.64, "end": 1716.72, "text": "So instead, what we had to do was do this empirically, run many, many, many experiments to understand how these various algorithms and features interacted to affect the end-to-end metric. Another assumption was that, well, we're going to put many, many algorithms in there and let different hypotheses compute throughout the system in order to deliver the latency which we needed, which was to answer a question, come up with an answer and an accurate probability in three seconds."}, {"start": 1716.94, "end": 1746.54, "text": "We were going to have to scale out. We were going to have to have an algorithm that was embarrassingly parallel, which is exactly what we did. And this is the picture, a high-level picture of Watson's brain, if you will, even though I promised I wouldn't show a picture of a brain. I meant a human brain. So it takes questions and categories. It comes up with many different interpretations of that question based on doing deep language parsing and semantic analysis. It generates many searches. From those, hundreds of possible answers. Each of those become the root of a tree."}, {"start": 1746.54, "end": 1757.15, "text": "where it would collect evidence to support or refute those answers. And from there, it would apply many different scores, hundreds of different scores that would analyze that content from different directions."}, {"start": 1757.15, "end": 1783.92, "text": "It would weigh those various features based on machine learning models. I have an end result of a ranked list of answers with the top answer, with all of them having a probability that they were correct. And if that probability was over a threshold, the knots would want to buzz in and answer. And I'm going to go over just a little bit. We populated that architecture with hundreds of algorithms. We had a goal-oriented metrics. We did regular integration testing."}, {"start": 1783.92, "end": 1812.18, "text": "We needed the hardware necessary to support that engineering, which gave us the results we needed to push the science. It took two hours to answer a single question on a single CPU. We ended up scaling that over 3,000 cores to be able to answer fast enough. In the end, we were able to drive that performance from that baseline all the way up to where it was cutting across that winner's cloud. And this was a big test on blind data, 200 games worth of blind data. Does it guarantee a win?"}, {"start": 1812.18, "end": 1842.18, "text": "Absolutely not. But you can see you're competitive there with grand champions across that curve. And the last thing I want to say is, actually, this is not the last thing. I have two more things I want to say. One is that the integrated metric, in other words, the system level end-to-end metric that we focused maniacally on, with the presumption that winning on Jeopardy was something that was going to advance the size of NLP, we still had to focus on that end-to-end system metric. When we were done with that, we went back and looked at some of our core component"}, {"start": 1842.18, "end": 1852.08, "text": "We looked at how it evolved with regard to independent component level metrics, the kinds of things you typically publish on. And we're going to write all these papers, but we were leaders in"}, {"start": 1852.08, "end": 1876.51, "text": "In all these areas, from parsing to disambiguation to relationship extraction to passage matching and passage scoring, in every one of them, we were leaders based on those metrics. So that was a very, very satisfying thing to discover. If you choose that problem right and you have metrics and you push that, you can get there. So it was very satisfying. The last thing I want to talk about is evidence profiles, and then I'll stop."}, {"start": 1876.51, "end": 1894.22, "text": "In Watson, we had many, many features, many, many algorithms producing many, many features. Just to get a handle on what was going on, we had to build tools to understand how those algorithms were interacting and impacting the correctness of an answer. Eventually, we got to what we called an evidence profile."}, {"start": 1894.22, "end": 1923.7, "text": "Where you'd be able to group of hundreds of features, you would group them into semantic categories. And then based on the machine model weights, you'd be able to see how they contributed to a right and wrong answer. This allowed us to do error analysis to understand where the system was failing. This is a very, very powerful capability ultimately. And what to do to solve problems. There's a number of things going on here. But I want to point out that one of the issues in getting this question wrong here, you'll find Bethel College and Seminary in this holy Minnesota city."}, {"start": 1923.7, "end": 1936.53, "text": "Bethel College and Seminary in St. Paul, Minnesota and South Bend, Indiana. We didn't weigh location information high enough. But there was also another path to answering this, which is through the pun."}, {"start": 1936.53, "end": 1965.74, "text": "The association between holy and St. Paul, the right answer. We actually independently, one person came up with some ideas on how to detect puns in language, was able to integrate that into the system, train it, and that was having impact on the end-to-end system, and we were getting more pun questions right. We did this very, very rapidly because of that architecture we depended on. The notion of an evidence, my last point, the notion of an evidence profile and the communication of confidence ended up becoming very, very important. When you watch the game,"}, {"start": 1965.74, "end": 1991.68, "text": "Where all you saw was the computer answering when it knew an answer and getting it most often right, you thought, gee, I must be just looking it up in a database. What's really going on here? When you saw just a glimmer of what was happening inside the machine, when you saw those confidence values and that histogram, the general audience starts to scratch their head and say, wait, there's something else going on here. You mean a computer can"}, {"start": 1991.68, "end": 2021.23, "text": "Can maybe be wrong? You mean it's not sure? You mean it's thinking? I mean, we think of it as computing probabilities based on analyzing evidence. But this got people to think, wow, this is really interesting. And this inspired both the general public to think about, what is AI really all about? And it also inspired our customers to think, you know what? This notion of dealing with uncertainty and dealing with language effectively helps me address a problem I didn't think of that way before. And we got lots of interest both from the general public and from our customers. Thank you."}, {"start": 2029.81, "end": 2064.43, "text": "Andrew and Microsoft. Well, thanks very much for inviting me, Tommy. It's been an amazing workshop the last two days. I guess one thing you didn't mention from my CV is that I studied here for a year and I didn't get a degree at the end, so I guess that makes me an MIT dropout, which I guess must be something to be proud of. So I just wanted to talk a little bit about technologies for..."}, {"start": 2064.43, "end": 2091.52, "text": "Natural interaction and some of the progress that I think is happening there. You know, there are many contexts in which we want to do natural interaction. Somewhere here there's a laser pointer. Yes, here it is. So the one that is sort of the source of my talk, if you like, is gaming, and that's kind of obvious. But people are thinking about all kinds of other things, like in medicine, for example."}, {"start": 2091.52, "end": 2120.75, "text": "Bringing some kind of intelligence, albeit superficial, to bear in analyzing what's going on in the human body. And here's another investigation of natural interaction, this time in the operating theater. It's not really practical for a surgeon to get up and start poking at a keyboard and manipulating a mouse. If they do that, they'll have to go and scrub up again afterwards. So it would be great to have some more flexible, non-contact kind of interaction with the environment."}, {"start": 2120.75, "end": 2147.34, "text": "Cars. Well, Amnon is going to talk a lot about cars. More and more of the high-end ones, at any rate, at the moment are coming up with head-up displays. And so there's a lot of opportunity there for a more natural interaction with the information that's displayed on the screen. For example, pointing navigational information, which is maybe projected onto the world. Here's a rather beautiful robot that the Anybots Corporation has built to take"}, {"start": 2147.34, "end": 2174.43, "text": "It'll be your proxy at a meeting. It'll turn up to the meeting, and people will talk to it. And well, I don't know if that's going to catch on. It's a fascinating idea. What else? Well, of course, there's a minority report. I don't know if we really believe in that, that that's what we actually want to have as the interface for interacting with information. But again, a fascinating idea. So the technologies for interaction are coming on. We've had a history of"}, {"start": 2174.43, "end": 2203.26, "text": "You know, necessarily clunky things getting more and more flexible, now touch and multi-touch, and then the new generation that I want to talk about, you could call it no touch or action at a distance. So the Kinect camera that Tommy referred to came out earlier this year, and yes, that's what my team has been working on, putting the machine learning into it. And here is the team. I'm going to tell you just a little bit first about what we did."}, {"start": 2203.26, "end": 2232.74, "text": "Oh, but before that, in case you didn't play with it yourselves, here are the kind of things you can do. You play games with your buddies. I'm not a gamer myself by any stretch of the imagination, but I must admit I have found some of these Kinect games are quite compelling just because they're social and they're easy to interact with. They're not too kind of head-banging. So what do we have to do to make the Kinect camera work? Well, it's sensing the pose of the human body."}, {"start": 2232.74, "end": 2252.19, "text": "And so the task is to receive the signal that the body generates and reduce that to a set of joints, rather like the motion capture systems that they use in movies a lot. You know, in Titanic, when the guys all drown in the sea, actually what they do is they..."}, {"start": 2252.19, "end": 2282.64, "text": "Put the guys on a slide, and they capture their motion as they slide into a pool, and then transfer that into the scene. So what we needed was to build something like that, but without having to attach all the instrumentation to the body of the actor, because, of course, that doesn't make sense in the interaction setting. So we take the body in in the form of data, and what we have to put out is the skeleton inferred on the right there, and represented mathematically in terms of positions and angles and so on."}, {"start": 2283.18, "end": 2305.6, "text": "This is a hard problem because the human body is amazingly pliable and variable. Even your body can do many things. And then, of course, there are many different kinds of bodies varying in size and with unfortunate accidents of appearance and in very complex settings and so on."}, {"start": 2305.6, "end": 2334.86, "text": "to make significant inroads into these complexities. And it's a kind of commonplace of AI. One of the lessons that AI has established is that trying to program rules explicitly to handle situations of this complexity doesn't really work. So we realized we had to do this somehow by machine learning. What we thought we would start with was principles based on the work of Dario Gavrila. He did this work in the Daimler-Benz."}, {"start": 2334.86, "end": 2349.52, "text": "He also was anticipating bringing vision to cars. And he had a fascinating system of hierarchical templates of the human form, capturing the position of the human form in a cluttered scene."}, {"start": 2349.52, "end": 2375.73, "text": "We thought that was not quite going to be enough because of the need to capture all of the articulations of the limbs, which you don't really have to do to detect pedestrians. So we were thinking in terms of something involving templates for parts of the body, but then you'd have something to do afterwards to harmonize those templates if they had all detected their parts independently. Meanwhile, Jamie Shorten on my team had been working on..."}, {"start": 2376.75, "end": 2405.2, "text": "I guess we can call it an AI problem. At least there are an awful lot of people working on this at the moment, which is let's get back to object recognition now with what we've learned in the last 15 or 20 years about low-level vision and see if we do any better this time around. And, well, we don't have human-level object recognition yet, but amazing amounts of progress have been made recognizing up to 100 objects at once in a scene. So Jamie wanted to use this kind of..."}, {"start": 2405.2, "end": 2430.1, "text": "Technology to try to solve the problem of detecting the human body in depth maps because the Kinect sensor is a 3D sensor. And my contribution at this point was quite crucial. I told him that it would never work. And he nonetheless went on playing with it. I said, well, that's fine. Try it. Why not? But we'd better pursue both tracks because yours is never going to work. And then after a month,"}, {"start": 2430.1, "end": 2457.49, "text": "His was working surprisingly well. I mean, it's still not working properly. So I said, well, I think we should still hedge our bets and do the two approaches. But fine, you play with that if you like. Your Friday evening project, except every day was Friday. And after a couple of months, it was really working amazingly well to an extent that I never thought it would. And so we kind of gave up on harmonizing the"}, {"start": 2457.49, "end": 2488.34, "text": "Template's idea and went with the recognition. So this is what we ended up doing. The depth map that you received from the sensor is on the left here with close stuff coded as bright and far stuff coded as dark. And we ended up with an output which is like a kind of noisy clown suit. Actually, the input I'll show you soon, at least the training data, is a kind of perfect clown suit. But the clown suit here is colored according to the body parts."}, {"start": 2488.34, "end": 2519.63, "text": "Guy moves, you see the clown suit is moving pretty much with the guy. And this is what convinced me finally, okay, the clown suit doesn't fall off when you move around, so the labels really are sticking to the body. Here we have many different poses all labeled with the same clown suit. Of course, there are 31 parts of the body labeled here, and various things happen which I don't have time to tell you about to generate hypotheses which eventually boil down to tracking a number of..."}, {"start": 2519.63, "end": 2525.23, "text": "So that was the story vastly accelerated."}, {"start": 2525.23, "end": 2553.01, "text": "That kind of capability takes its place amongst, I think, a number of capabilities that are coming out that will change the nature of interaction with machines. And of course, our aim is to make the computers disappear in our interactions with machines that we find around the world. So we don't want to think of our cars as computers or assemblies of computers. We want to think of them as cars. And I think this is true for all the machines with embedded computing that we use. So of course, speech technology is"}, {"start": 2553.01, "end": 2577.1, "text": "has advanced incredibly over the last 15 years. And I think that is one of the huge areas of progress. I would like to disagree with Patrick, I guess, for what you were saying a couple of days ago. This is a field which I think has been transformed out of all recognition. And hey, let's call it artificial intelligence, if it exactly is or isn't. But let's claim it."}, {"start": 2577.1, "end": 2607.07, "text": "It has been a huge piece of progress. Eye trackers, incidentally, are something which all of a sudden seem to have come of age. You can get them embedded in computers now, other kinds of sensing. I'm just going to show you one or two things. A couple of things from MIT. This is something I saw from Roz Pickard from the Media Lab. It's amazing that you can sit in front of your laptop now, and without anything special attached to the laptop, it'll tell you how fast your heart is beating. That is truly astonishing when you first see that demo."}, {"start": 2607.07, "end": 2631.12, "text": "And another thing which migrated from one Cambridge to the other, Rana El-Khaloubi, who was in Cambridge University and now is here in the Media Lab, has made amazing progress in the interpretation of emotions. For a long time, computer vision researchers were grinding on about, is it the six basic emotions, and recognizing those. And it turns out not to be very difficult. Once you get into kind of complex"}, {"start": 2631.12, "end": 2660.75, "text": "Then the emotions have trajectories in time and everything gets more complex. And Arana's work is so beautiful because it exploits autism databases that were constructed in Cambridge, Simon Baron Cohen, for the diagnosis of autism and turns them into oracles for what emotions are as reflected in the face. Something else we're doing in Cambridge is trying to..."}, {"start": 2660.75, "end": 2687.95, "text": "Make a commonplace out of the business of gesture recognition. So at the moment, when games designers want to... I think I'll... The guy is controlling a DVD player, by the way, using his hands. And, of course, you could build this into the Xbox with Kinect. What's special about this is that... I'll just advance it so it stops making that noise. What's special about this is that we never actually built any kind of a recognizer for..."}, {"start": 2687.95, "end": 2715.25, "text": "The gestures. All that happened was we recorded a DVD with the examples of the different gestures on it, fed that into a machine learning algorithm. Out of that dropped out a recognizer module. So you can imagine a designer, somebody who doesn't want to think about computer science, building an interface to their machine using this capability. Well, I'm over my limit, so I'm just going to skip to my"}, {"start": 2716.91, "end": 2740.8, "text": "How about this slide? So just one other thing I wanted to mention is just the whole idea that perception, which is what we're trying to build into these machines that give us flexible experiences with devices, is enhanced or relies on having probabilistic reasoning built into the machines. And a kind of lingua franca for..."}, {"start": 2740.8, "end": 2768.37, "text": "Probabilistic reasoning, much of this theory was developed here in MIT. And elsewhere is the idea of the probabilistic graphical model. Here, for example, is a graphical model that encodes asthma, the different causes and effects, and possible intermediate concepts. Here is a probabilistic graphical model that encodes game playing and the way in which your skill will affect your performance and how that plays out with multiple teams."}, {"start": 2768.88, "end": 2787.66, "text": "Just to finish with, I just want to say that one of the things we're working on in the lab is trying to embody this in a package which could be handed out to developers, and developers are trying it as we speak. It's on the web for people to download and play with. We call it Infer.net, and it's encapsulating all of that graphical model."}, {"start": 2787.66, "end": 2811.44, "text": "Language, if you like, in a way which we hope ultimately will become a commonplace in software development. We're not really there yet. So at the moment, it's still a good idea if you have a PhD in machine learning to program with this stuff. But we hope that in the future, this will become commonplace. So just to conclude, a lot is happening, I think, which gives us reason for hope."}, {"start": 2811.44, "end": 2858.18, "text": "So the title is a bit presumptuous. Given the gap between our engineering successes and the wide spectrum of human perception abilities, the gap is staggering. So the title itself doesn't have any meaning. It's not well defined unless we narrow."}, {"start": 2858.18, "end": 2886.98, "text": "The domain, and this is the secondary title that I'm going to talk about, the domain of making the cars see. And I'll be modest. I'll be talking about technologies that I think can be implemented within five years, five to 10 years. And where this field of making cars see has the potential to reach within five years, and what are those technologies that we'll need to develop. So just a slide of a recap."}, {"start": 2886.98, "end": 2914.46, "text": "Today's technologies center around a camera looking forward, front-facing, doing about five different functionalities, lane departure warning or lane analysis, detecting cars and performing forward collision warning and braking on cars, detecting pedestrians and doing the same thing on pedestrians, detecting and reading traffic signs, about 60 to 80 different..."}, {"start": 2914.46, "end": 2942.82, "text": "Traffic signs. And also doing certain lighting functions. At night, be able to automatically determine whether the high beams should be on or off. And where this is going is towards autonomous driving within the next five years. In terms of the computer vision technologies, the three main areas are object detection, cars, people, traffic signs. There's a lot of motion estimation going on, both"}, {"start": 2942.82, "end": 2965.97, "text": "Understanding the motion of the host car, how the car is moving, and understanding what other cars are doing, kind of a friend and foe determination. And roadway understanding, which is like a light scene understanding. So the next slide, there are these clips that are running in a loop. So you can look at them in a relaxed mode. So the one on the top."}, {"start": 2966.16, "end": 2995.82, "text": "is detecting cars and pedestrians. What you see here is detecting lanes and road shoulders and all what is necessary in road analysis. Here you see some motion understanding. Whenever there is a lateral motion, it is colored. The dots are colored in red, and all the static points are colored in green. So there is a necessary ingredient here is to detect the ego motion, the motion of the car, in order to know what is moving and what is not moving."}, {"start": 2995.82, "end": 3020.14, "text": "And here is the friend or foe estimation. The red rectangle is the car in a collision course, in a possible collision course. And the white color is the cars in the neighboring lanes. The system needs to know how many lanes are in the field of view and what the cars are doing, the average speed of cars, whether there is an escape lane in case of an accident, and so forth."}, {"start": 3021.2, "end": 3034.0, "text": "In terms of what remains hard to do in this context is one is the scalability of object recognition. Engineering success is really limited."}, {"start": 3034.0, "end": 3060.05, "text": "to a small number of object classes. It means if the task is given an object class, go and detect it reliably in the image, this is possible. If the task is you have 1,000 object classes, go and find instances of these object classes in the image reliably. This is still an open problem. And the second hard thing is the situation awareness. I've seen understanding in the context of automotive."}, {"start": 3060.05, "end": 3089.94, "text": "It's what is happening in the scene. It's not only where are the objects, but what the objects are doing with relation to one another. So in terms of scalability, as I said, we can conjecture, given the engineering successes that we have seen so far, that all of one classes can be done today. Now, this is a controversial conjecture, but let's bear with me and let's agree that this is possible."}, {"start": 3089.94, "end": 3112.26, "text": "If we assume a linear scalability, that means if we need x computational resources for one object class, then it will need two x resources for two object classes. This linear scalability will not work when we have thousands of object classes. And thousands of object classes is what human perception is all about. We effortlessly can find instances of thousands of object classes."}, {"start": 3112.26, "end": 3137.17, "text": "The linear scalability means that we'll need to replicate resources, and this is definitely not something that is workable, even in the context of very large computing power. In terms of in our, in the automotive space, the areas in which you find a need for multiple objects, in traffic signs, you have there."}, {"start": 3137.17, "end": 3164.64, "text": "A sample of the different kind of traffic signs. In addition, there will be stop signs, yield signs, traffic lights will be detected. There's guardrail, concrete barriers, road edge, poles, bridges, and then the list goes on. And there's about 100 different object classes that are being identified today. And if you want to go into autonomous driving, there's at least 1,000 different object classes that you need to know where they are in the image in order to"}, {"start": 3164.64, "end": 3191.22, "text": "The clip that you are seeing is the system detecting all the relevant objects and also understanding what those objects are. This is an example of a clip where it's a challenge for any computer vision system. So you have a camera moving in space, both outdoor and indoor, and any object that it sees in its path, there is a label of what that object is."}, {"start": 3191.22, "end": 3220.02, "text": "Or whether it's an obstacle without a label. This is something that I'm gearing towards, the ability to store thousands of object classes and find them effortlessly in real time on a portable system. This is something that I think can be done within the next five years. In terms of situational awareness, what are those scene understanding elements that are needed?"}, {"start": 3220.02, "end": 3237.49, "text": "For example, find the lane boundaries when there are no lanes. Simply knowing what other cars are doing, where other cars are moving, the position of static objects like parked cars also gives you a clue what are the boundaries, what are the lane boundaries, even if there are no painted lanes."}, {"start": 3237.49, "end": 3251.62, "text": "Travel planning, how many lanes, traffic dynamics in neighboring lanes, whether there is an escape lane or not. Where are the objects of interest located? For example, is the pedestrian located on the road? Is the pedestrian located on the sidewalk?"}, {"start": 3251.62, "end": 3277.57, "text": "Where is the face or the posture of the pedestrian located? Is it located toward the collision course? Is it located in another direction? What do the pedestrians do? What do other cars do? This is also something that is necessary to understand when you go into autonomous driving. In terms of new research directions, again, I'm limiting it to this particular space."}, {"start": 3277.57, "end": 3306.69, "text": "So one of them is the scalability. As I mentioned, the linear scalability is not workable. One needs to develop a principled way of reaching sublinear logarithmic growth of resources as a function of the number of classes, which means that when you go and do classification, it's not a one versus all classification, which is the popular way of dealing with multiclass today, but it's subset of classes versus subset of classes."}, {"start": 3306.69, "end": 3334.62, "text": "And the challenge here is that if you have n classes, there are two to the n subsets. So the question is, how do you, in a principled way, avoid this course of combinatorics and do something that is theoretically understood and works? It's not that there's no research in this direction here at MIT. There's a nice paper by Bill Freeman and Antonio Tarabla. But still, it's a drop in the water. It's a drop in the ocean. This is an area that is"}, {"start": 3334.62, "end": 3356.38, "text": "This is understudied and needs to be pushed forward significantly in order to make room for the issue of scalability. Processing video. By and large, our community is single frame centric. If you look at all our data sets for machine learning and computer vision, they're all single frame."}, {"start": 3356.38, "end": 3382.18, "text": "In the work that we have been doing in Mobileye, the fact that you process video is critical. All the engineering success comes from the fact that you are not looking at a single frame, but looking at that video. I have here a second. No, I don't have. So I wanted to tell a story, but I'll put that aside. Microprocessors. Most people are not aware of the fact that the hardware"}, {"start": 3382.18, "end": 3408.7, "text": "The vision that you have will also determine the kinds of algorithms that you can run. Most applications in vision require a portable system. It's not that you'll be running the algorithm on the cloud like Google can do, but it has to be done on a microprocessor, low power consumption because it is portable, and there is a hardware consideration."}, {"start": 3408.7, "end": 3422.45, "text": "Array accelerators that exist today, these are DSPs, GPUs, FPGA, were not designed for computer vision. They were designed for signal processing and for MPEG, JPEG compression, decompression."}, {"start": 3422.61, "end": 3448.64, "text": "If I put here a slide of the spectrum of a microprocessor, there is the RISC machine, CPUs that you are all aware of on one end of the spectrum, which allows you to run any branch and bound general code. And on the other side, there is the vector accelerators, the DSPs, FPGA, and GPUs that you may have heard of. Maybe you don't know exactly what they do, but those are vector accelerators. Computer vision."}, {"start": 3448.64, "end": 3474.37, "text": "Requires both, and requires also the spectrum in between. Now, early visual processing, like edge detection, interest point location, can be done using vector accelerators that exist today. But once you identify certain areas of interest in the image, the type of algorithms that you run are very complex algorithms. These are not algorithms that can enable to convolution type of processing."}, {"start": 3474.37, "end": 3503.65, "text": "So a vision coprocessor is not a CPU and not an existing vector accelerator. I have here a list of the existing accelerators, each of them with their pros and cons. And the point is that none of them is designed for computer vision. At Mobileye, one of the things that we have done, we designed our own chip and our own vision coprocessors that are targeting the types of"}, {"start": 3503.65, "end": 3531.36, "text": "Complex algorithms that are being done in computer vision, especially the 2D type of processing compared to 1D type of processing that is done in signal processing architectures. But still, there is a long way to go to build hardware that is efficient, scalable, low cost, low power consumption that will enable to do very complex computer vision algorithms."}, {"start": 3531.36, "end": 3579.7, "text": "So I'm going to talk about the rich connection between games and artificial intelligence. So in fact, games and AI have a very long history together. In fact, one could argue that this mutual relationship dates back to before computers even existed. So here, for example, is a picture of the Mechanical Turk, the original one before Amazon rebuilt him."}, {"start": 3579.7, "end": 3608.66, "text": "He's a chess automaton that was built in 1770 and was destroyed by a fire in 1854. But before its destruction, the story goes that it beat Napoleon in a game of chess. And Napoleon was apparently so disgusted with losing the game that he ran his sword through the machine. But unfortunately, the human chess master dwarf that was hiding inside apparently escaped unscathed."}, {"start": 3608.66, "end": 3631.47, "text": "It's a happy ending. Now, in fact, beating a human champion at chess has been a dream of AI since the dawn of the field. And Alan Turing, for example, famously composed a chess program in the 1950s, which managed to beat a few weak amateurs when he proceeded to run it by hand on pen and paper."}, {"start": 3633.36, "end": 3658.86, "text": "This dream, of course, was finally realized in 1997 when IBM's Deep Blue defeated Garry Kasparov in a titanic six-game match that captivated the world. Of course, that was not the end of the story, and the field moved on to tougher challenges such as Go and Poker. But even those games have become relatively tractable in recent years. So, for example, Go..."}, {"start": 3658.86, "end": 3688.11, "text": "Which is thought to be a much tougher problem than chess for a machine to play well, mainly for two reasons. One is that it has a bigger branching factor. So the average number of moves in any one position is larger in Go than it is in chess. And probably the more difficult problem is building a good evaluation function for Go is very difficult because of its esoteric nature. However, a French group from INRIA built a program called MoGo."}, {"start": 3688.11, "end": 3712.7, "text": "that used state-of-the-art Monte Carlo tree search simulation to overcome these problems and recently became the first program to beat a professional at Go on a 9x9 board with no handicap. And poker is also tough because it is an imperfect information game with hidden states in terms of the opponent's cards."}, {"start": 3712.7, "end": 3740.21, "text": "And again, recently here, Polaris, a University of Alberta bot, beat some top professionals at Heads Up Limit Poker in 2008. So there's been some tremendous successes in terms of building AI for traditional abstract games. But probably what's less well-known is the impact, the role that AI has played in another sort of game, namely video games."}, {"start": 3741.78, "end": 3755.01, "text": "So the video games industry is a big business, not as big as Google, but it's a global business worth more than $10 billion a year in revenues. And AI has played a big part in that success."}, {"start": 3755.01, "end": 3779.25, "text": "And it's found in nearly every AAA top title in some form or another. For example, with pathfinding for agents to find their way around the world, character behavior, including squad tactics for war games, and AI opponents in all kinds of games, racing, sports, and strategy games. And of course, even with things like Kinect, even involved with the controller itself."}, {"start": 3780.3, "end": 3805.65, "text": "But AI is not just being used as a tool or component for games. It is also being used for a certain genre of games as the central gameplay element upon which the game is actually built, the game itself is actually built. So I'm going to talk about a few of those games in the history of the evolution of that kind of game. And it's a bit of a personal take on that history. And I'm going to focus on some of the games that I personally worked on."}, {"start": 3806.19, "end": 3816.38, "text": "These types of games, games that used AI as a core gameplay component sort of started in the early 90s with games like SimCity and also a game I worked on called Theme Park."}, {"start": 3816.38, "end": 3846.74, "text": "The idea of theme park was that the player built their own Disneyland. And then once you built this Disneyland, hundreds of autonomous agents came into your theme park and played on the rides and bought things from your shops. And underlying this game was a fairly complex for its time and economics model, the main challenge of which playing the game you had to balance. So the better you made the rides, the more fun the people had and the more you could charge them for things like Pepsi."}, {"start": 3846.74, "end": 3875.66, "text": "That led to a whole genre of games called management simulation games. And then in 2001, we worked on a game called Black and White. So in this game, the idea was that the player was a kind of god and was battling against other gods. And your physical presence in the game world was in the form of these mythical beasts, the mythical creatures. So here you see a lion here. And the player actually reared these creatures from birth."}, {"start": 3875.66, "end": 3898.42, "text": "It's literally from very young. And depending on how you treated the creature, they became good or evil, and which would have its uses depending on which way you went. So this is a good lion. And evil ones would stamp on the villages and your followers and so on and eat them. And even today, this is still one of the most complex reinforcement learning agents ever to have been built."}, {"start": 3900.43, "end": 3930.03, "text": "And then with management simulation games, this came to fruition for me with Republic the Revolution, which was done by my company in 2003. And in this game, the idea was that the player was a revolutionary. And we actually, for this game, simulated a whole living, breathing country, a fictional ex-Soviet republic called Novostrana. And this game world had literally had hundreds of thousands of people that went about their daily lives, went to work, went out in the evenings, and had political affiliations."}, {"start": 3930.03, "end": 3938.9, "text": "You as a player had to influence this world. So the idea of this game was that we wanted to make you feel maybe a glimpse of what it would be like to be Che Guevara to inspire evolution."}, {"start": 3939.86, "end": 3962.35, "text": "And the main difficulty with this game, actually, the main innovation on the AI side was what we called level of detail AI, which is that these kind of games have to run on standard home PCs. So we don't have supercomputers to run these things on. This is in 2003. So in order for the home PC to be able to run all the graphics as well as the AI, we created this system called level of detail AI, which is that..."}, {"start": 3962.35, "end": 3978.27, "text": "Any agents of which there were hundreds of thousands that weren't in the current cone of view or near the player, we would compress and run in a lossless state and then decompress them as the player moved around the game world. And finally, no"}, {"start": 3978.27, "end": 4006.13, "text": "No discussion of AI in games would be complete without talking about the most famous one, the range of games under the title The Sims. And this was designed by the game genius Will Wright for his company Maxis. And there's several iterations of this. And this version I'm showing here is Sims 3 done in 2009. And here, the scale is completely different from a game like Republic, where now we're focusing on a single family and your interactions with that family."}, {"start": 4007.06, "end": 4031.9, "text": "So you can sum up those kinds of games with Maslow's hierarchy of needs, which is a classic theory of human motivation. I haven't really got time to go into this. But basically, games have so far dealt with this bottom level, the physiological level, breathing, food, water, sex, sleep, and so on, and a little bit of the next level, but nothing up here of the higher levels of things like human morality, creativity, and so on."}, {"start": 4031.9, "end": 4043.76, "text": "Whilst a lot has been achieved in games, and collectively the games I've talked about have sold millions of copies, there's still a ton of stuff to be done in games. So I'll just finish with a couple of slides about the future."}, {"start": 4043.76, "end": 4069.55, "text": "Where do I see the direction of AI going in games itself? Well, we want to create game characters with more depth, with things like emotion and memories and hopes and dreams. Games agents and opponents that learn and improve and adapt to the player. Of course, there's the old chestnut of conversing with game characters, which of course will need natural language processing advances. And also using AI to make educational games that actually make learning fun."}, {"start": 4069.55, "end": 4093.58, "text": "But the one I'm most excited about is what I call dynamic storyline generation. So I came on this when I asked myself, you know, what is it that makes games special as compared to other entertainment media like films and novels? Well, I think I would argue that it's the ability to allow the audience, in this case the player, to actually influence the outcome of the narrative themselves. So that's something that none of these other media can offer."}, {"start": 4093.65, "end": 4113.89, "text": "So if that's what makes games truly distinct from these other types of genre of entertainment, then one can think then and one can argue that the ultimate game is therefore to give the player complete freedom of choice, but somehow also ensure that whatever choice they make, even if it's a stupid choice, is always compelling and fun. So this is a really difficult balance to make."}, {"start": 4113.89, "end": 4142.86, "text": "And so game production teams know this, and there's really only two ways of dealing with this problem if you follow this line of argument. One is to special case all decision points, which is incredibly expensive and needs incredible amount of talent in terms of writing and so on. So you literally special case every decision a player could make. Or the other alternative and more practical alternative is to make the environment intelligent and allow it to react intelligently to the way the player interacts with it."}, {"start": 4143.38, "end": 4167.78, "text": "And then the dream would be then to create games where every player's experience of the same game ends up being unique to them. And then my final slide is perhaps to me now more interesting even, is what can games do in the role of developing AI? So I believe actually that games can end up being a very useful platform for developing AI algorithms themselves."}, {"start": 4167.78, "end": 4187.92, "text": "So there's two types of platform I'm thinking about. One is to use current state-of-the-art games, say, for example, real-time strategy games like StarCraft. And this, I think, is the next step up in difficulty from games, abstract games like Go. So we have, in games like StarCraft, massive action sets and continuous and dynamic environments."}, {"start": 4187.92, "end": 4217.23, "text": "But the sort of thing that I'm working on is actually creating bespoke platforms for AI development that use games technology. So for example, rich 2D grid worlds, which include realistic perception and allow for a wide range of highly configurable problems in a unified framework. So this gets around the problem that we have in a lot of academic research where we special case the solutions. So I think it's the right balance of complexity between toy problems that we often use in academia that can be special."}, {"start": 4217.23, "end": 4245.46, "text": "And finally, I think there's another thing that this gains us is avoids robotics. And I think robots are cool, but they're very intensive in terms of their engineering effort. And that engineering effort can sometimes be a distraction in terms of concentrating on what makes them intelligent. And then finally, I think this is a good way to commercialize AI improvements. And this is actually the approach that we're using at the new company that I've co-founded called DeepMind."}, {"start": 4245.46, "end": 4288.32, "text": "So just to summarize, I think games are a great commercial domain for the application of AI. And also, games are the perfect platform for the further development of AI algorithms. Lukasz. So I like to think about computer science as actually an early outgrowth of artificial intelligence. And my favorite illustration of that is actually the word computer."}, {"start": 4288.32, "end": 4318.96, "text": "People of my generation often forget this, but computers were once people. They were trained people who did arithmetic calculations by hand, sometimes with mechanical tools, usually in service of science and engineering. So a helpful metaphor. It's kind of like a very early version of Amazon's Mechanical Turk. Now, once we started to understand the processes involved in arithmetic, we started to understand how to break them down"}, {"start": 4318.96, "end": 4340.0, "text": "Epistemologically, and then in terms of simple elementary logic operations, we were able to build machines that could do the same thing. Therefore, that led to this notion of the electronic computer, which is now spread everywhere. But there's something interesting about where computing has gone, especially in relation to"}, {"start": 4340.0, "end": 4368.7, "text": "The boundary between computing and artificial intelligence. So we've heard throughout this symposium, and actually really beautifully today, that at the frontier of computation, the distinction between the problems we think of as easy for computers, like calculation, adding 2 plus 2, and the problems we'd really like that seem to require AI techniques, the heart of it seems to involve inferring causes behind data that we observe and choosing good actions in response."}, {"start": 4368.7, "end": 4388.4, "text": "So one of my favorite examples is personalized medicine, pressing problem. We have a wealth of genomic data and health information about patients, and yet we still don't really know, except in some very simple cases, which combinations of genetic markers are really causally responsible for response to which treatment."}, {"start": 4388.4, "end": 4416.59, "text": "You see the same theme that inferring the causes behind data is at the heart of AI computations in many other areas. So I won't go into all the details. But the interesting thing is that although we've seen sort of remarkable successes, when we do this kind of work with our computing machines as opposed to arithmetic, it seems very hard. That difficulty ultimately boils down to the fact that when you're doing inference as opposed to calculation, there's just a whole lot of possibilities to wrangle with."}, {"start": 4416.59, "end": 4443.09, "text": "So we heard about how many different types of bodies there are in the connect system to deal with. This is one example. Or how many different configurations of pedestrians in the world. And so although there's a very simple mathematical theory that in principle lets us get good answers, and this Bayes rule here is one expression of it, just directly implementing it by calculation seems hopeless. And that's basically because there's too many causes. So in the genetics example, let's say there were only 200 genes."}, {"start": 4443.31, "end": 4469.9, "text": "There are more possible combinations of genes if there are just a total set of 200 than there are particles in the universe. So enumerating them would be completely hopeless, let alone weighing them in light of the data. So this leads to this art of AI engineering, where people try to approximate the dictates of this formula. We've been exploring a different approach."}, {"start": 4469.9, "end": 4498.74, "text": "So rather than try to use devices designed for calculation, where you put in a program which specifies some function for turning an input into an output, we've been developing a different model of computation designed around inference by guessing. So the idea is this. Instead of giving the computer a function to calculate, you give it a space of possible explanations to consider. So you do that by telling it how to guess one of them."}, {"start": 4499.54, "end": 4525.1, "text": "So the machine's job, then, is to take that way of guessing and some data and explore the space of possible guesses the machine could have made in light of the data and hand you back a probable one. And our hope is that by making machines that are built from the ground up to do guessing instead of calculation, we can avoid some of the tractability issues that people have struggled with in artificial intelligence."}, {"start": 4525.1, "end": 4547.97, "text": "We currently simulate these machines on top of commodity hardware, but we've also just begun to explore what the hardware substrates would look like if you were building physical machines for guessing instead of arithmetic. So I'm just going to give a couple of illustrations of what these machines look like and how they behave. So let's take one illustrative example worked out"}, {"start": 4547.97, "end": 4578.14, "text": "By one of my colleagues, Keith Bonowitz, breaking CAPTCHAs. So a CAPTCHA is nice in that it's a thing that's very easy for a machine to generate some image. But it's supposed to be hard for a machine to work backwards from. It's supposed to require human intelligence in some sense. So we're going to break CAPTCHAs by just taking a program that generates random CAPTCHAs and feeding them into a machine along with a particular CAPTCHA and asking the machine, how did you probably run to produce that CAPTCHA?"}, {"start": 4578.14, "end": 4607.97, "text": "So how might one write a program that guesses CAPTCHAs? Well, maybe you write something that chooses some letters and puts them in a random location. So maybe you guessed five, and then renders an image, and then adds some noise to make it harder to read. So you blur it and add a few errors. So writing this kind of program is pretty straightforward. And then we're going to say, OK, machine, this program produced this particular CAPTCHA. How did you probably run? OK."}, {"start": 4607.97, "end": 4634.82, "text": "So the code to do this takes about a page in our probabilistic language. And here, what I'm showing is on the top an input CAPTCHA. And on the bottom, we're going to watch a random walk through the space of possible explanations for this CAPTCHA in light of the program. So you can see, actually, as soon as it started, it pretty quickly got kind of into the right ballpark. And now it's sort of exploring alternate names for my company."}, {"start": 4634.82, "end": 4664.94, "text": "It's slowly going to delete the letters until eventually it'll converge on the right explanation for the CAPTCHA. So it's bizarre. But when you have machines that are designed to make guesses, they start to behave as though they're imagining alternative explanations for the data that they observe. So what are we doing with this kind of capability commercially? Well, our first product is based on the following observation. Although information technology has made it enormously easier to capture and query digital data,"}, {"start": 4664.94, "end": 4693.07, "text": "The thing that really gates people getting use out of it is humans telling stories about it, humans with statistical training. So machine learning is a part of that process, modeling. But really, there's this complicated iteration with the human expert in charge in many applications. So we built a system called Veritable, which we like to think of as sort of a statistical analyst or a statistical intern in a box. And what it does is it considers a hypothesis space of"}, {"start": 4693.07, "end": 4710.26, "text": "Very, very many causal explanations for tabular data, including things like what all variables might possibly interact with each other and which ones might not, using the same inference machinery that I showed you earlier. And then it can make predictions to fill in all of the missing values in a given table of data."}, {"start": 4710.42, "end": 4725.14, "text": "One example. So here's a table from a health economics problem, where each row is a hospital in the US. And each column is a variable measuring how costly care at that hospital is, what the reported quality is, and so on."}, {"start": 4725.17, "end": 4744.96, "text": "So in just a few minutes of computation, our system was able to figure out, for example, that only the other quality scores with very high probability are causally related to quality. So there's something out there in the world that makes hospitals good, and that has nothing to do with how big they are, how much capacity they have, how much it costs them to run."}, {"start": 4744.96, "end": 4774.78, "text": "And that kind of statistical finding, which really requires a whole lot of internal validation, was actually the centerpiece of a New Yorker study that was actually somewhat influential in the health care debate. OK. So now we've gotten just sort of a flavor of a couple of different applications of machinery designed for inference. Let's talk a little bit about what the hardware to support it might look like. So there's this kind of bizarre thing"}, {"start": 4774.78, "end": 4802.8, "text": "That happens when you do a lot of probabilistic reasoning with computers. You realize, especially when you start doing it in the style that I just showed, where the machine's job is only to make guesses, not really to calculate probabilities. You have these chips, which have billions of elements on them. And you're requiring them to do billions of steps in a row exactly the same way every single time. And they were manufactured with enormous precision and cost. And yet you're using them for operations that are fundamentally random."}, {"start": 4802.86, "end": 4831.25, "text": "So I actually think Watson illustrates this beautifully. If you compare and contrast the efficiency of the Watson computer to its human designers, it's kind of crazy, right? It requires maybe 50,000 times the power, and it's maybe 10 million times faster in terms of its switching elements. And that's all because it's frantically trying to calculate probabilities really, really quickly, as opposed to the brain, which might actually be doing something that's a bit more organized around guessing."}, {"start": 4831.25, "end": 4861.3, "text": "So we've been exploring building hardware architectures designed to implement this model of computation for guessing by moving from- That's only 80 watts. That's only 80? Oh, beautiful. All right. Where instead of having gates like AND, the AND gate, which implements a logical connective, we have randomized gates, so RAND gates that implements stochastic systems. They flip coins. And the noise is not an error. That's part of the behavior that lets the system make good guesses."}, {"start": 4861.3, "end": 4889.23, "text": "So an early architectural application we've done using this is to make a simple probabilistic video processor, which is designed to guess the causes behind video motion, basically, to guess what the motion field is that explains some movie. And then we get some pretty interesting speed advantages. So this processor and the movie was made by my colleague Eric Jonas, who's a good sport."}, {"start": 4889.23, "end": 4914.64, "text": "I just want to close by saying the thing that's exciting to me about this time in AI is that we've seen that many, many applications really boil down to just a handful of primitive operations, the operations of reasoning under uncertainty. And although they seem very, very hard technically, and we've had some great successes, there's actually hope for maybe removing a lot of that complexity."}, {"start": 4914.64, "end": 4963.57, "text": "So what if we reconsidered what computing is and what computing machines are, and we built our machines to do inference by guessing from the start, and maybe spread them everywhere that devices that calculate have already gone? Thank you. Kabi Richter will close the presentations. Good afternoon. You must be tired and bored. I'm the last and most probably the least speaking after Google and IBM."}, {"start": 4963.57, "end": 4987.04, "text": "It reminds me, when we started Mednol, most of the employees we recruited were immigrants from the former Soviet Union that came to Israel in millions. And I had about 15 former railroad engineers that we made into stent."}, {"start": 4987.04, "end": 5017.97, "text": "And I told them that the only thing I can think of that links the Israeli railroad system with the Russian one is the width. The same way I feel after Google's presentation that the only thing that combines me with Google is that we also have a dollar sign in front of our numbers."}, {"start": 5018.58, "end": 5049.9, "text": "Presentation, the short presentation I hope I'll be making is in honor of the contribution of David Marr that I was lucky to come here to MIT to work under in his last year of academic activity. And I will start with I was wondering, who is David Marr today? And when you want to do that, you Google him."}, {"start": 5050.13, "end": 5078.13, "text": "You go into Wikipedia, and I was interested to find that Wikipedia knows, or the world knows, who he was and that his seminal work had to do, first of all, in collaboration with Tommy Poggio, had the tri-level hypothesis with the three levels of"}, {"start": 5078.61, "end": 5107.02, "text": "Analysis, computational level, algorithmic or representational level, and implementation level. The work of David Marr, the very short but amazing contribution, was summarized by Tommy Poggio about a year after his death. And the one thing that I found very inspiring is that"}, {"start": 5107.02, "end": 5136.82, "text": "This was said about 30 years ago. I worked together with Shimon Ullman mainly on trying to find the way some of his ideas of early processing are implemented in the human."}, {"start": 5136.82, "end": 5166.03, "text": "Brain, we started with David Marr basically started by saying that, first of all, showing that the Laplacian of a Gaussian could be useful in efficiently representing images, and that the difference of Gaussian that is implementable in combination of cells in the retina is very similar to it. He showed how useful the zero crossings."}, {"start": 5166.03, "end": 5193.82, "text": "Difference of Gaussians could be in analyzing images. Later in his work with his students here with Ellen Hildreth in 80, he showed how the processing of images into zero crossings helped to very efficiently do three dimensionality, motion analysis, and so on."}, {"start": 5193.82, "end": 5223.41, "text": "Together with Tommy and Francis Creek, and later Poggio, Alman, and Fale, improved it, showed that hyperacuity, our ability to judge relative position of lines, for example, in a 10 times better resolution than the resolution of our receptors, was enabled by similar processing in the retina. The first thing I did together with Shimon Ullman is"}, {"start": 5223.63, "end": 5249.01, "text": "Developing a retinal model based on comparison to all the physiological recording that was available to us at the time with the exact response, transfer function, and time delays between receptors, horizontal cells, bipolar cells, and into the ganglion cells with the"}, {"start": 5249.01, "end": 5279.34, "text": "We analyzed the inhibitory effect of amacrine cells. We analyzed the spatial combination of such receptive fields of ganglion cells. And the time, the temporal, and for the first time was able to model very strange things, like recordings from monkey ganglion cells."}, {"start": 5279.44, "end": 5307.47, "text": "In this case, to flashing grating or moving gratings and the very strange notch in here that turned out to be well modeled by the delayed inhibition of the amacrine cells. We satisfied ourselves by showing that recording from the monkey in insects, in bipolar cells, amacrine cells, and ganglion cells were well."}, {"start": 5307.47, "end": 5334.43, "text": "modeled by our model that then created a much the delay in time between center and surround, created better predictions in the center panels, B, of the real recordings than the one without temporal that basically"}, {"start": 5334.43, "end": 5362.78, "text": "Did a year before that. So was the case with the Y cells. And I'll skip that for time. We then went to the LGN cells and satisfied ourselves that in response to different edges moving across the receptive field, at some point, the model predicted there would be another extra."}, {"start": 5362.78, "end": 5391.36, "text": "So three zero crossing for two edges. We actually did find exactly that in the LGN and went into recordings from cells in the visual cortex, hoping to find a response to that phantom zero crossing, if you will. I was lucky enough that Peter Shiller agreed to give me his lab to work in."}, {"start": 5391.36, "end": 5420.64, "text": "Before that, a very known electrophysiologist refused to give me his lab to work in. He said that it's a bad idea to start research with a hypothesis. It biases you. And I said, it may bias me, but it's much more interesting to aim at the target than just shoot in the dark and turn the lights on and see if you hit anything."}, {"start": 5420.64, "end": 5451.84, "text": "So as you can see here, the prediction was that if there is a cell that responds to zero crossing at some distance between the edges, there will be a third response. Our recordings from cells, surprisingly, in more than half of the cells, did show that third response to that third."}, {"start": 5451.84, "end": 5477.62, "text": "And when the zero crossing disappeared, the response disappeared. It was very rewarding to see that. And from here on, I moved to an industrial commercial applications. And the reason I showed you only 30-year-old"}, {"start": 5477.62, "end": 5501.3, "text": "Science is because that's when I finished doing it. The first application that we used some of the algorithmic level was in automated optical inspection systems in a company called today Orbutech that Shimon Ullman and myself were two of the founders of the company."}, {"start": 5501.3, "end": 5523.92, "text": "It was implemented in a system that looks very much like what you see here, that automatically inspected printed circuit boards, found defects in them, and as a matter of fact, also guided laser systems that corrected the panels."}, {"start": 5524.18, "end": 5553.26, "text": "That system, because of the use of zero crossing and hyperacuity, was about 10 times higher resolution for the same price and kept more than 80% market share until today. The next application, I then moved to something very different, to developing coronary stents. Coronary stents."}, {"start": 5553.26, "end": 5583.33, "text": "These are very tiny devices. As you can see here, they are about 3 millimeter in diameter and made of very tiny struts or beams, basically 50 microns or so wide. And they should support a vessel. Basically, if you look here, you'll see in red dots the perimeter of the heart. And these are two of the main vessels that irrigate."}, {"start": 5583.33, "end": 5611.62, "text": "The heart muscle, in order to give it oxygen and nutrients, you do see at the arrows two narrowings. This is a very sick person who can hardly climb a single floor or so. But after the implementation of two stents where the lines are shown, he's back to normal life."}, {"start": 5611.62, "end": 5632.06, "text": "When the stents are there, they are subject to very severe bending, repeated bending during heartbeat. And whether or not they'll fracture is critical to the health of the patient. So the inspection of the stents is a major"}, {"start": 5632.06, "end": 5662.7, "text": "I will not go much more about that, because something I heard yesterday, I decided to respond to. Is language as external representation?"}, {"start": 5663.47, "end": 5692.34, "text": "In two different languages, stable over the internal representation is an issue that Noam Chomsky brought first and then was discussed yesterday by Jenara Kierkegaard and Susan Kerry and others. It's not only an academic issue. Here is your 100 million question."}, {"start": 5692.34, "end": 5722.91, "text": "In real life, stents were very, very competitive field. And we developed the first competitive stent that was very flexible because between the zigzag rings that allowed the stent to expand, we placed looped connectors that allowed the rings to become longer on the outside of the curve and shorter on the inside and the stent to be flexible."}, {"start": 5722.91, "end": 5751.5, "text": "In our patent, we had a claim that said the scent is made of zigzag rings with connectors having a loop between rings. The next thing that happens is five years later, a big company comes up with a scent that looks almost identical with one exception. The connector has two loops."}, {"start": 5751.5, "end": 5780.54, "text": "Susan Carey and Gennaro Kierkegaard told us yesterday, no problem. A loop is at least one loop, right? Everybody believes that. Well, we won the case in more than $100 million in the US because this was recognized as that. But then we go to Europe. And in Germany,"}, {"start": 5780.54, "end": 5812.5, "text": "Our patent is translated into German. And a loop is translated into Ein Schleife. But Ein Schleife is at the same time one loop or a loop. Now, the judge and the court interpreted it as one loop. And they said, definitely no infringement here. This is two loops. It's a $100 million issue."}, {"start": 5812.5, "end": 5839.44, "text": "And it's external representation of a language. Don't worry. I had enough time, and I did issue a new patent in which I actually explicitly translated a loop into at least one loop. Now the German translation was, for instance,"}, {"start": 5839.44, "end": 5867.71, "text": "And now it was OK, and I want it in Germany. But it brings to the question of, is the internal representation stable? Are the courts stupid or inexperienced? Or is external representation affecting the internal representation of language? I want to, I don't have."}, {"start": 5867.71, "end": 5891.82, "text": "A lot of time left. As a matter of fact, I used much more than I had. So I want to just finish with one thing. You all probably know the type of advertisement of HSBC with the duality of what is work, what is play. If I'll use Pat Winston from the first day of this meeting, what is..."}, {"start": 5891.98, "end": 5927.22, "text": "And with David Marr's proposition that we'll have adventures and fun, my only question remaining is, which is fun and which is adventure? And with that, I will conclude this presentation. Very nice. So, Haim Hirsch."}, {"start": 5927.22, "end": 5954.48, "text": "We'll take over in running the discussion. Haim, you can see this in the booklet, is a professor of computer science at Rutgers University. He was director of the Division of Information Intelligence System at NSF. He has been one of the engines of our intelligence initiative and a great help. Haim."}, {"start": 5955.31, "end": 5984.43, "text": "This is the last panel, last event of the symposium. So let me first point out, I think, a couple of themes going through the whole event. I think one of the thrills of this event is that we're all here absolutely excited and tantalized about understanding mind-brain intelligence. And specifically, I think we're all tantalized with the idea that the advent of computing and advances over the last few decades in terms of our growth of knowledge"}, {"start": 5984.43, "end": 6014.45, "text": "Instrumentation, growth of data are making possible new understanding that we couldn't have even conceptualized only a few decades ago. So even though we're all here with the shared thrill of conquering one of mankind's greatest challenges, it's really interesting to showcase the difference between the bookends of this event. The first event where we had statements as severe as nothing's been done in 20 years."}, {"start": 6014.45, "end": 6019.28, "text": "To an event where a number of"}, {"start": 6019.28, "end": 6048.24, "text": "People have been able to express things that not only weren't achievable 20 years ago, but many of these things we wouldn't have even thought of 20 years ago. And by the way, so we have seven people here, but there are many people who spoke over the course of the event and who are in this room. I mean, Tommy himself, Patrick, and others who've had commercial successes taking ideas from the computational understanding of intelligence and translating it into the marketplace. So the question I'll try to start the conversation with for the panel is,"}, {"start": 6048.24, "end": 6080.42, "text": "How to resolve that very different sense of perspectives that nothing's been done versus we have these really exciting successes. Are we answering different questions? Is one group wrong and the other one right? I'm curious what you guys think. Yeah. I mean, I think some amazing things have happened. I mentioned speech, but just to mention something in vision. And this is not egomania, because I had nothing to do with this. But I think that"}, {"start": 6080.42, "end": 6109.52, "text": "Object recognition has really, in the last decade, transformed out of all recognition. At one time, we were recognizing pyramids and solids made of straight lines and planar faces and so on. And now people are recognizing not only complex objects, but also in complex contexts. And of course, there's so much."}, {"start": 6109.52, "end": 6136.7, "text": "Still to do. Like I said earlier, it's up to 100 objects. And also, lest anyone may think that it's only engineering, and a lot of it, of course, is engineering, and I think this panel, amongst other things, stands for the power of engineering, there are some beautiful ideas. And I'll cite one beautiful idea in object recognition, which is the idea of visual words, which is stolen from document retrieval."}, {"start": 6136.7, "end": 6167.6, "text": "And who would have thought that there was any meaningful definition of the notion of a word in the visual context? And yet, somehow, this has emerged as a rather important concept. This is good. I think the dichotomy can be framed between, I think, the difference in terms of the advances we've made in specific AI applications and, I guess, the original dream of general AI, one algorithm that all"}, {"start": 6167.6, "end": 6196.54, "text": "A set of algorithms that can be applied to anything, like the brain can. So I think maybe that's where the difference in view in terms of advance comes from. So I was going to say that I think it depends on what your taste is, what you're interested in doing. Or another way to say it is, what's your success criteria? David said very clearly that part of his success was being able to set that success criteria ahead of time and then trying to optimize it. I resonate."}, {"start": 6196.54, "end": 6222.7, "text": "Very much with that. That's what I always try to do. And when you've done it correctly, you're done. Now, yesterday we heard Chomsky tell us that he thought it was absolutely bizarre that you would use the word success in this strange sense, the sense of being successful. He thought that that was totally alien to all senses of science. And what he meant was it didn't count."}, {"start": 6222.7, "end": 6250.26, "text": "The statistical techniques that we use, or David used, were being successful in approaching the goal very closely. What counted was, are they successful in coming up with a succinct theory that's easy to understand? And so I think there's these two different viewpoints of an engineering viewpoint versus a scientific viewpoint. And to me, the theory says the same thing, whether I can understand it or not. And if I don't understand it, that's my fault, not the theory's fault."}, {"start": 6250.26, "end": 6279.31, "text": "But other people can have different points of view as to what's important to them. I think in our mind, we understand what is easy for computers to do and what is difficult. But we have a problem in accepting it emotionally because the AI stuff, the things that are related to intelligence for us is so effortlessly done that we emotionally do not accept that computers have a difficult time doing it. Yesterday at the reception, I was talking with Josh Tenenbaum."}, {"start": 6279.31, "end": 6306.42, "text": "And there were tons of people around us. And he said, OK, your system can recognize, can identify that pedestrian, that person, where you saw only 2% of the body surface because of occlusion. I said, well, no. Can it do it in a year from now? He said, I don't think so. So you have a system that organizes pedestrians, but it's still a situation in which you would expect it to do very well because we humans do it."}, {"start": 6306.42, "end": 6334.4, "text": "Do it very well. And this goes across all the areas of artificial intelligence. Because for us, it is being done so effortlessly, we expect progress to be much, much faster. I don't know why you'd want to do that, because if the guy is being occluded by another guy, he's the one who's going to be knocked over by the car. Automotive area, that's what I told Josh. David, then. So very specific."}, {"start": 6334.4, "end": 6344.93, "text": "I mean, I think things have changed, and I think we are having more success. I mean, I think when I go back to when I was in graduate school, I was fascinated with AI, and at the time it was in the"}, {"start": 6344.93, "end": 6364.74, "text": "And it was all about these expert systems. And I was very motivated to build medical expert systems and built one. And one of the tragedies there was that while you can model these complex reasoning paradigms, deduction, induction, you can do all kinds of interesting things."}, {"start": 6364.74, "end": 6385.41, "text": "It used to break because of what we call a knowledge acquisition bottleneck. I mean, you had to build this detailed axiomatic model of the entire domain. And this was difficult and time consuming and required a tremendous amount of domain expertise. And the knowledge wasn't there and readily accessible to the computer in a way that this can ultimately become effective."}, {"start": 6385.41, "end": 6405.47, "text": "And it's been a long time. But I mean, with the availability of large amounts of text, both for applying heuristics as well as for applying statistics, doing the kind of NLP that you see in Watson and many other applications, we're starting to be able to reason over unstructured content."}, {"start": 6405.47, "end": 6435.42, "text": "That's hugely significant, because now I could build an expert-like model that generates hypotheses or guesses, takes those and gathers information and analyzes them. And maybe they're not following that precise axiomatic deductive proof that it might be able to do if you axiomatize everything. It's still giving you value. And I think that's the key. If I can generate and score and evaluate evidence from all that knowledge out there in its natural form,"}, {"start": 6435.42, "end": 6464.43, "text": "I can bring value to the user, whether it be a doctor or a person doing financial services or whatever. And now I can do it more readily. I can do it over these huge volumes of information that we're all prolific in generating, which is natural language content. And I've seen that change. That's what I'm excited about. Are we doing it perfectly? Do we have a perfect theory of how it works? No, but it really, I think, advances both the science for business and the underlying science if we could start to do that kind of thing and bring value to humans."}, {"start": 6465.2, "end": 6495.2, "text": "Great. So three quick points. The first one is I agree with everything that's been said. I think I want to add that I think it's really striking to me, certainly overseeing the symposium, that latent in the apparent heterogeneity of the applications, there really are common techniques and principles at work. So in that sense, both the dream of improvements in real useful systems and an understanding of what might be happening underneath, I think real progress is being made."}, {"start": 6495.2, "end": 6497.92, "text": "I think the second point I want to make is"}, {"start": 6497.92, "end": 6527.25, "text": "I think there's another dimension to this question of progress, which at least I remember. I went to graduate school, actually, in brain and cognitive sciences. It's often conflated with dimensions in computer science. There's this question of are we making progress at making systems that are better at getting the right answer, or getting good answers to the problems they're posed, versus are we making progress at this other element of the dream of AI, which is maybe understanding ourselves a little bit better, the elements of our minds and brains. Again, I actually think the answer is we are making"}, {"start": 6527.25, "end": 6552.48, "text": "We're making progress at both of those. There's a whole research thread in cognitive science and neuroscience now, which is all about taking models and techniques from the state of the art of AI and extracting predictions out of them for human cognitive error and neural function. The last quick comment about this question of progress. So one of the first guys who invented the compiler, Alan Perlis, had this great quote, which I think it's"}, {"start": 6552.48, "end": 6583.36, "text": "Critical, at least for me to remember, working in AI, which is every program is the special case of a more general program. And all too often, we notice this too quickly. I think it's important for us to keep that in mind in AI, because the problem is maybe much bigger than it's easy for us to keep in our vision. I like that. Last again. From my point of view,"}, {"start": 6583.36, "end": 6611.65, "text": "The expectation that AI contribution into the two opposing direction, one, what it can contribute into brain research and cognitive science, and the other one, in applications, commercial applications and so on, should have similarity is not necessarily a robust expectation. I don't know why we need."}, {"start": 6611.65, "end": 6640.58, "text": "More intelligent systems in the commercial world. We have pretty many of them. There are about 150 of them sitting here. They're self-producing, self-maintaining. And I'm pretty satisfied with taking out of AI those things that can replace humans in what they are not good at and what they are"}, {"start": 6640.58, "end": 6666.58, "text": "I'm bored at doing. In inspecting stents, for example, in tests that I give to inspectors and to the machine, the inspectors find 37% of defects planted there and the machine 99.7 with two employees and three simple machines."}, {"start": 6666.61, "end": 6694.67, "text": "I have the capacity that other competitors use 250 inspectors, and they have to run them three times on each stand because they find only 40%. This is a job for a machine. And if Forever will take out of AI into the commercial world, those things that the machines are very good at, and we are not very good at, and actually don't want to do."}, {"start": 6694.67, "end": 6720.98, "text": "The fact that today in Israel I can make a stand for less than one third the cost that it is made in China and it will bring it into Israel because machines do much of the work is enough for me. The complicated and important contribution of AI into understanding the brain, why? First of all, because it's there."}, {"start": 6720.98, "end": 6750.42, "text": "Secondly, because there may be great benefits to health, learning, and other things, is on a very, very different and much more sophisticated level. And in a way, I understand in this way the choice of word of destruction that Pat used. Why waste talent on simple tasks like commercial machines? That's OK."}, {"start": 6750.42, "end": 6779.92, "text": "People will choose whatever they choose, but they don't have to be on the same level. We can do, on a sophisticated level, the brain research, and on a much more simplified but efficient level, the machine vision, machine understanding, machine whatever. Tommy's behind me, and he's going to say something. So you mentioned the first day, and Marvin, and Noam."}, {"start": 6779.92, "end": 6805.3, "text": "And I'm a good friend of Marvin. And to put it bluntly, I think he's completely wrong and is out of touch with what has happened in the last 20 years. We have made progress. It's all relative. If you read the proposal that started the conference, which was the founding act of AI."}, {"start": 6805.3, "end": 6834.56, "text": "It was the case that people thought then it would take a few years to solve the problem of intelligence, to have a machine that will pass the Turing test. And I think this is very naive. Intelligence is not only a great problem in science. I think it is the greatest problem in science. If you think about it, there are many other problems, time travel."}, {"start": 6834.56, "end": 6864.62, "text": "The origin of the universe, the nature of life. But if you could solve intelligence and enhance your intelligence or create a machine more intelligent than us, you can more easily solve all other problems. So that's the top problem. And if you can even- If you had a time machine, you'd have plenty of time to solve the other problems. Yeah, but you know, you have to play probabilities. So it all depends on which science fiction writers you read most."}, {"start": 6864.62, "end": 6894.21, "text": "I think I was more realistic maybe because I was in the Liceo Classico in Italy. And so I studied very little math and physics, a lot of history and philosophy. And so the problem of intelligence has been around for thousands of years. I think it would be crazy to imagine we can solve it within 10 years, 50 years, one century. I mean, we have not solved physics yet."}, {"start": 6894.21, "end": 6923.7, "text": "These are big problems if you want. Solving means solving it in all details. So for me, I think that the progress that has been made, for instance, in object recognition, what Andrew mentioned, is astonishing. 20 years ago, I would have never said. 10 years ago, I would have never said. Or what Watson has achieved. Incredible. Or what Google does. Equally incredible."}, {"start": 6923.7, "end": 6950.51, "text": "I think we've made astonishing progress. Still, there are basic challenges. And the basic one, which has come up several times in this symposium, is this problem of integration. If you think about a Turing test, I think there are machines now, like for instance Watson, that can solve"}, {"start": 6950.51, "end": 6978.93, "text": "Narrow version of the test. If the test is jeopardy, well, it has been solved. If the test is driving a car, well, it will be solved very soon. But if the test is asking the computer all the kind of questions about sports and emotions and language and vision that you can ask a person, any normal person, then we are still"}, {"start": 6978.96, "end": 7009.25, "text": "Still a long way out. And so that's why I think we need basic research. I think we need neuroscience. The point that was made in a previous session about consciousness, it's an hypothesis, but the defining property of consciousness is the integration, to integrate between different aspects. It may apply to intelligence, to a real Turing test of intelligence. And so the question I think interesting for the panel is,"}, {"start": 7009.25, "end": 7039.23, "text": "How can we go in that direction? How can we integrate? Is it possible to integrate, say, the mobilized system with Watson and with Google? And how can we go in that direction? I'm coming back to Marvin Minsky again. He had this book, which is really a coffee table book, but each chapter is great."}, {"start": 7039.23, "end": 7065.76, "text": "A little set of thoughts. And the title is the main idea in the book, is a society of mind. The mind is a society of pretty dumb agents. The question is, do we have already the agents and maybe easier than we think to connect them together or not? So I'm anxious to just tell, I guess this is kind of a joke, but it's very much related. Someone actually asked me, you know,"}, {"start": 7065.76, "end": 7074.32, "text": "So can Watson. But Watson plays Jeopardy, but it can't play chess. And I said, sure it can. It just calls his brother Deep Blue."}, {"start": 7074.32, "end": 7100.45, "text": "There's an interesting point about integration there, which is, gee, that's kind of the obvious, loosely coupled way to solve this, is that you, depending on what kind of question you ask it, it just engages this entire other system and runs this entire other process, and there you have it. But at the same time, more seriously, I think the notion of integration, and particularly across modalities and across representations, is really so important. And one of the things that"}, {"start": 7100.45, "end": 7122.8, "text": "Underlies the Watson system is a thing called, that we created the five years earlier, is a thing called UIMA, which is the Unstructured Information Management Architecture, which is really plumbing. It's a framework for integrating different kinds of analytics. And one of the underlying premises of that was to enable you to combine analytics across different modalities."}, {"start": 7122.8, "end": 7152.08, "text": "The expectation is that you're going to be able to have to analyze text, you have to analyze speech, and you have to analyze images and video. And you have to do this all in one system and one environment. And I think it would be really cool to see what we learn by building systems that start to combine this. And we're starting to see that text and speech. And if you think about medicine, you want to be able to get cues both from natural language as well as from images as well as from speech. Building systems that do all of that together, I think, is going to help inform us."}, {"start": 7152.08, "end": 7180.26, "text": "I think it's one of the big things going forward. I think it will be very helpful if we accept a certain truth that it's going to be messy. When David explained about the Watson, one of the things that he said that it's not one algorithm. They have dozens of independent systems working together and also that mobilize not one classification algorithm. We have many algorithms working together."}, {"start": 7180.26, "end": 7209.38, "text": "And this is messy. And science doesn't like messiness. It's like a 10,000 page proof for a mathematical theorem. It may be valid, but it's not satisfactory. And we need to accept that in order to reach the goals that Tommy mentioned, it is not going to be a one line simple thing that we're simply not seeing right now and somebody clever will come and see 10 years from now. As you also mentioned, biology is also messy."}, {"start": 7209.38, "end": 7232.37, "text": "Well, in fact, to emphasize that, when we started the Jeopardy project, I had a lot of different responses from my own team about whether or not we should do it or not do it and everything. I had one person who came up and said, well, if we do it, we're going to have to come up with Maxwell's equations for QA. And I said, if that's what we need, we're screwed. And I actually believe that that's not the case."}, {"start": 7232.37, "end": 7254.22, "text": "But I think it's irresistible to look for that elegant theory of anything. And I don't think you should throw that away. But at the same time, you've got to deal with reality. And you will learn if you make things work. You will learn stuff. There's no doubt. So in some ways, this was an unfair group of people to be asking this question to, because we have seated here all the people who favor"}, {"start": 7254.67, "end": 7286.14, "text": "Who have engineered some complex systems to do things that exhibit intelligence. I'm curious. We intentionally have the speakers of the event sit up near the front so they can participate in the conversation as it develops. I'd actually like to turn to maybe some of the other people who participated in other parts of the session and let them respond to do the things that we've seen here have implications for the broader pursuit of intelligence that many of you have focused on? Yeah, wait for the microphone, please. I'd like to ask a question to David about Watson."}, {"start": 7286.14, "end": 7302.69, "text": "So in the news articles about it, and also there was one slide in your talk that suggested that although Watson's performance is really pretty astonishing and wonderful to behold, often the second and third"}, {"start": 7302.69, "end": 7317.58, "text": "The answers, the ones that the system is less confident about, are not only wrong, but from a human point of view, absurdly wrong. They're the wrong category of thing, suggesting that"}, {"start": 7317.74, "end": 7343.81, "text": "In some reasonable sense, it's not doing anything we would call thinking. A thinker wouldn't make a mistake like that just to get something where, say, the answer is a kind of thing x, and then gets a completely different kind of thing that couldn't possibly be the answer to that question. So that's a great, great question. And I think it reveals the approach we ended up taking. Because taxonomies"}, {"start": 7343.81, "end": 7371.02, "text": "We think of taxonomies as very rigid, but there's a couple of examples I'll give you. At one point, questions would be asking about during this month, et cetera, et cetera, something, something. And we'd answer with winter when the answer was February. And the reason is because we're breaking the question down to different elements. We're looking at different kinds of evidence, and we're scoring that evidence independently. And you saw those evidence profiles. So you get different weights for different kind of evidence."}, {"start": 7371.02, "end": 7399.98, "text": "And sometimes that evidence would be skewed in a direction that the computer thought, there's a lot of evidence for this. But a human looks at that and says, wait a second, that other dimension, like the fact that it's in Minnesota, or the fact that it's supposed to be a man versus a woman, is just really overwhelming for me. That's the evidence you need to pay attention to. You could just ignore all that other stuff. And that depends on the ability for humans to sit there and say, I know exactly what's going on given the context."}, {"start": 7399.98, "end": 7429.38, "text": "So a great example was this month one. So I said, you know what? I mean, we're getting winter in first position. We know it's supposed to be a month. The evidence is getting skewed in the wrong direction, right? And we insisted on looking at whether or not something is a month from a probabilistic perspective. We didn't sit there and say, you know, there are 12 months. That's it. We sat there and said, how is language used around the word month? What kinds of things tend to be considered a month given the context in which that word is used?"}, {"start": 7429.38, "end": 7456.08, "text": "But it looked really stupid. And I said, you know what? That's our principle, but we're going to have to violate that. Just tell it what the 12 months are, because this just looks too stupid. And we put the 12 months in. And then there was a question, during this month, Muslims celebrate blah, blah, blah. And the answer was Ramadan. And we got it wrong, because we didn't anticipate that that was one of the months Jeopardy would be asking about. There are countless examples like that. And there was one asking about a direction. You name all the different ways you can express a direction."}, {"start": 7456.08, "end": 7476.38, "text": "And then you get this up, down, right, left, south, degrees around a compass. And then there's the question, this is the direction of fabric. And the answer was grain, which I don't know that you would have anticipated. So we ended up sort of backing up and saying, you know what, we're going to stick with this probabilistic approach. Sometimes it's going to look stupid, but this is what we're going to do."}, {"start": 7476.38, "end": 7503.58, "text": "In that famous question, which I've answered this a million times now, which is Toronto, how did you get Toronto as a US city? I mean, what Watson learned by looking at a lot of these things was that the category in jeopardy was not a strong signal for the answer type. It was a weak signal. And the question said, its largest city. I'm sorry, its largest airport. So just because the category was US city did not mean the answer was this US city."}, {"start": 7503.58, "end": 7532.85, "text": "So it wasn't a strong indicator. And then when you look at Toronto, is Toronto a US city? Well, we had databases that would tell us this. But we also relied a lot on text in order to categorize these things. And Toronto has an American baseball league. Toronto is often talked about as an American-like city. And there was some weak signal that Toronto could be considered a US city. Moreover, it didn't expect it that it had to be. So it gave what you thought was a very stupid answer."}, {"start": 7533.14, "end": 7562.93, "text": "A thinking person with a just slight degree of knowledge will know what the months are and won't make a mistake like that. So my point is, is it really thinking? Not the way you are. Go ahead. Let me take a shot at answering for David. I think one way to think of it is using the language of this morning's panel is that that little readout wasn't looking at the conscious mind. It was looking at the subconscious mind. And a thinker would be reminded."}, {"start": 7562.93, "end": 7573.9, "text": "And then the conscious mind would take over and say, oh, yeah, I was reminded of that, but that can't be right. And the readout wasn't looking at the conscious level. It was looking at the subconscious level."}, {"start": 7574.38, "end": 7603.98, "text": "In that particular case, Watson was very unsure of both Toronto and Chicago. It was way below threshold. But also, when you look at what happened on the game, was often the right answer would have very strong confidence, and the two below would actually have weak confidence. That was a tremendous success for us. It wasn't always like that. That meant for more times than not, we were actually making the right distinction and weighing all that information in the right direction."}, {"start": 7603.98, "end": 7630.11, "text": "But I think the answer was no. If it is the unconscious mind, it's not thinking. It's dreaming. Yeah, I mean, it's not thinking the way you're thinking. That's absolutely true. Often, we would get the same wrong answer that the humans were getting. And this would happen as well. Does that mean we are thinking like the human? I don't think so. I'm inferring that behind your question was"}, {"start": 7630.11, "end": 7659.18, "text": "A suggestion that that's an example of why these aren't really giving us good clues towards the nature of intelligence. I think as the fellow next to you said, maybe we're talking about a kind of unconscious, something that's more like human unconscious intelligence. That might be right. So I agree. I mean, I don't think that aspect was telling us a lot about the way humans think. But I think it's only fair to say that as we as a team are working on this,"}, {"start": 7659.18, "end": 7688.06, "text": "While we didn't approach this as how would the brain do it, we did not approach it that way. But often in our meetings, we would sit there and we'd be working on an algorithm. And we'd think, how would a human solve this problem? And that would inspire some of those techniques. There's no doubt about it. How would a human think about puns? How do we program that? And we did. So it wasn't like a starting point. There was an implication. It informed us often."}, {"start": 7688.06, "end": 7718.96, "text": "But I would like to ask you a question back. Why do you even expect this to contribute to our understanding of intelligence? Isn't it enough what David said, that it contributed a lot into other projects that they'll do in the future that process data, and it helped in the image of the customers? This is a commercial project. Why should it help understand the brain? But look, maybe it does."}, {"start": 7718.96, "end": 7747.06, "text": "Help. Because if it's more like unconscious intelligence, then if we understand that what we are is unconscious intelligence plus an overlay of conscious intelligence, that might really be telling us something about the human mind. You can argue the strategy. We've got lots of questions on the strategy. Why does Watson bet in a crazy way? Watson doesn't bet in a crazy way. It seems crazy to you or to humans often because"}, {"start": 7747.06, "end": 7775.79, "text": "Actually, you're not thinking in as sophisticated a way as Watson is on the betting. Go ahead. I wanted to also try to answer your question from being one of the old timer golden age types. One part I had to cut out of my little talk to try to stay in 10 minutes was a little bit on shift of perspective from when I was a student in the 60s until later. When I was a student, the idea of linguistics as a science"}, {"start": 7775.79, "end": 7805.34, "text": "The model was physics or mathematics. And the evaluative aesthetics was if you didn't have an elegant theory, something like a system of axioms, then you didn't have an explanation. And there was what was called simplicity metric. A shorter grammar was necessarily a better grammar. And that really did change over the years when the kind of model went from physics to biology."}, {"start": 7805.34, "end": 7826.77, "text": "Realizing that natural language didn't, I mean, God maybe invented physics, but we evolved language. So messiness is no longer a proof of incorrectness. On the other hand, there are things that have elegant"}, {"start": 7826.77, "end": 7852.59, "text": "There are elegant, clean sub-theories. I would have talked about generalized conjunction as something that showed that Boole is right, and is still right, because it's universal across all languages. So we don't want to throw away the parts that can be explained in an elegant way. But I think maybe my old teacher doesn't agree about success."}, {"start": 7852.59, "end": 7874.56, "text": "In these changing ways, but I'm certainly ready to that there can be a mix of cleanly describable theories and a lot of messy stuff that can't be described so cleanly, but may very well be right and true. So is it fair to say that you're also kind of conjecturing that working both ends towards the middle has some value?"}, {"start": 7874.56, "end": 7903.62, "text": "If you think of the tidy Maxwell's equations version as one end and the engineering from the other. I don't want to make any predictions. I don't want to make any predictions about what would work. I remember when I was a student, Chomsky thought we'd reach the middle by having mathematical theories of language that got more and more like real language and linguistic theories that got more and more precise. And then after a while, he gave up on the mathematical linguistic side as being irrelevant."}, {"start": 7903.62, "end": 7929.63, "text": "But I would hope so. I would hope so. One comment about that tension between clean principles and messy examples. Actually, Herb Simon, one of the founders of actually both psychology and computer science, I think actually he spoke to this in a way that I think is both helpful and really consonant with the themes of the applications that we've seen, which is that if you have a system, that's the result of some adaptive process."}, {"start": 7929.63, "end": 7949.2, "text": "Then its structure, first, there's two levels you can try to understand it. One is, what is it trying to achieve in the environment? And the second one is, how is it achieving that? And you'll find that the complexity of the system says as much about the complexity of the environment it's acting in as the mechanisms that it's using."}, {"start": 7949.2, "end": 7971.71, "text": "So an organism or a mind evolved for a simple environment might be, the underlying principles might be quite elegant, but it might produce something that on another level appears very complex and on another level appears very simple. At least for me, I think that's how I helped to hold that spectrum of descriptions in mind. So one of the premises of the Intelligence Initiative and this whole event is that"}, {"start": 7971.71, "end": 8001.9, "text": "We need to return to our roots. We need to go back to broadly based interdisciplinary study of intelligence, bring cognition, computing, philosophy, the various stakeholders that we've seen at this event to really be able to tackle fully the nature of intelligence and how mind and brain come together. Do you guys agree with that? Or are you just proofs that we need to return to our roots in order to solve this?"}, {"start": 8003.79, "end": 8032.94, "text": "And are any of you comfortable saying something about that? Well, I have nothing more to say than repeat myself. I don't think. You, the researcher, you, the scientist, should do your work. AI puts understanding in your brain. The best way I can think how we can feedback is if we are successful,"}, {"start": 8033.55, "end": 8061.79, "text": "Generous, and remember where we came from. We'll donate 10 postdoc grants to AI students. But to be successful, do we need to do anything differently? Are there certain kinds of success that we're not achieving? Say, for example, by requiring better interdisciplinary training of our students, or asking new questions, or is staying the course going to? I mean, yes, we're going to be able to do what you just said, but."}, {"start": 8061.79, "end": 8091.36, "text": "Is there a need for a phase transition, and are we heading towards it? I guess my feeling is that we don't have to. I mean, both directions seem justifiable in their own rights. And to the extent that they can inform each other is a bonus. I don't know why they're competitive. I don't see them as competitive. I'm trying to tease out what contributions they can have towards each other, if you have some insights towards that. Is the goal you're talking about trying to understand the brain, or are you talking about?"}, {"start": 8091.36, "end": 8122.82, "text": "Trying to build artificial intelligence systems, because I don't think of the same goal. So I think of it as computationally tractable understandings of intelligence. So it's not just understanding intelligence in some abstract philosophical way, but things that give us sufficient mechanistic understanding that we know how to build things from it. I actually think whichever definition of intelligence you take, kind of the descriptive one for human behavior or the normative one of do the right thing, there's"}, {"start": 8122.82, "end": 8145.49, "text": "I mean, while I agree that they're really different enterprises in terms of sort of their central tendency, I think they absolutely can and should inform each other, primarily because they help fix the tunnel vision problems that can arise when you just work on one alone. And I don't mean that to say that you can't do great things in applications with precisely that tunnel vision. I mean, one of my at least"}, {"start": 8145.49, "end": 8165.82, "text": "One of my favorite families of examples is for a very long time, machine learning was really focused on tuning parameters of simple functions. And that's an enormously powerful technique, absolutely. On the other hand, there was a lot of work coming out of philosophy and psychology and statistics that sort of focused on the idea that maybe human learning"}, {"start": 8165.82, "end": 8194.61, "text": "Can be mathematically modeled in a similar way, but over much more complicated or structured representations. And that work has now fed back into building AI systems that then do statistics in this much more complicated and rich way. I think that's one example. And there are other examples in neuroscience where you really can have productive interactions. I think a phase transition can happen in a situation which it's not clear to me how that situation can"}, {"start": 8194.61, "end": 8222.06, "text": "It can be implemented where the commercial efforts and the academic efforts are combined. What do I mean by that? The commercial efforts, they can deploy resources that are not available to academia. So for example, if mobilized software was available to all the researchers, if the Watson code was available to all researchers,"}, {"start": 8222.06, "end": 8249.7, "text": "If the Microsoft code of detecting gestures was available to all researchers, this would create a phase transition. Coming up in June. That's very good. Mobileye software is not coming out anytime soon. So this would create a phase transition, because then you combine forces that are complementary. I think that one of the paths I see between the two. Oh, I'm sorry. Go ahead. Just very quickly is that"}, {"start": 8249.7, "end": 8277.47, "text": "The direction my research is taking is very much in the area of active learning. And that's very much informed and inspired by how you see people, even children, learn language. I mean, they start with trying to understand. It's amazing how plastic they are with syntax. It's remarkable how they choose just the right questions to ask back to fill in the gaps. So the whole notion of active learning and building systems that are doing understanding, learning, and interacting all together."}, {"start": 8277.47, "end": 8306.91, "text": "Which is very much where I would like to take the Watson stuff. It's very much inspired what I observe and what others have observed about how people learn. So not necessarily a deeper understanding how the brain works. But I mean, that bidirectionality, that informing both sides, I think is very important. I think if we're interested in doing the integrative intelligence that we talked about that can solve a generalized Turing test or something like that, whether we should build something like that, whether that's worthwhile is another question. Maybe we should stick to building AI systems for things we don't want to do."}, {"start": 8306.91, "end": 8310.19, "text": "But if we assume that that's still a goal, then I think"}, {"start": 8310.19, "end": 8339.5, "text": "Having some neuroscience in the mix is a good idea, because for two reasons. One, I think, the obvious one is to inspire some new types of algorithms for things that we don't know how to do at the moment. For example, Matt Wilson talked yesterday about some great work he does with hippocampus and rats and how they deal with spatial cognition and represent space. And maybe, for example, there's something we can learn there about flexible spatial representations for AI. And I think the second one that's often overlooked is if you want an integrated system, and I agree,"}, {"start": 8339.5, "end": 8368.72, "text": "with Amnon that it's going to be messy. It's going to have a lot of different algorithms and a lot of different things going on in there, as the brain has. Then we can use it as validation testing. We can say something like, for example, reinforcement learning seems to be a general algorithm, seems to be potentially a useful thing to have in the mix for an integrated intelligence. And then we can go and look in the brain. And people like Peter Diane Wolfram Schultz have found reinforcement learning is implemented ubiquitously by the dopamine system in the brain. So now we can go, well, OK, so it's not so crazy."}, {"start": 8368.72, "end": 8396.53, "text": "To think of reinforcement learning as maybe one of the key components in that mix for the integrative intelligence. So I think there's two things we can use systems neuroscience. So the kind of algorithmic representation level that Tommy and David Marr talked about can inform, I think, some AI advance. I was going to say, I think another use is if we want to have systems that are communicating with people, they have to learn to speak their language. So we need to understand that at that level."}, {"start": 8396.56, "end": 8419.81, "text": "Let me turn to others in the audience. Go ahead. Draw a single thread through the most successful systems that you've described. It would be data. And Peter gives this fantastic talk called The Unreasonable Effectiveness of Data. And as far as I can tell, many or most of you have worked with enormous, rich, varied data sets to build and train the systems that you've described. However,"}, {"start": 8419.81, "end": 8443.76, "text": "The overwhelming majority of these useful data sets are proprietary, and for good reason, for competitive advantage, for the expense of distributing them, and also to protect individual privacy. And so if we're interested in driving maximal progress, I'd like to ask the panelists how, if at all, we could share these large essential data sets more broadly."}, {"start": 8446.45, "end": 8472.99, "text": "So there's a lot more availability to the data than there was ever before. I mean, even in an open source sense, it's enormous. But I couldn't agree more. I mean, data matters. And the compute power necessary to experiment over it matters too. I would agree with that. And you mentioned my talk. And yes, it's true that I can't give you a trillion web pages and have you fit it on your laptop."}, {"start": 8472.99, "end": 8494.77, "text": "But in my talk, all the examples I gave involved only a few billion words of training data. And that you can fit on your laptop. And we offer a version of it that you can download. And other people have other corporates that you can get. And a lot more is available, certainly in natural language and in many other fields as well. And also, many of the companies that"}, {"start": 8494.77, "end": 8504.7, "text": "Are well endowed with data, take interns as a way to allow people to be able to conduct research. And so that's one way that's happening. Another way is"}, {"start": 8504.7, "end": 8530.64, "text": "Many people look at the search engine companies and say, we would love your data. And of course, it has great value to them. So Jamie Callen came along and said, I'm going to build a crawler. And he consulted the main companies and built a crawl. And it's now being distributed, in that case, partly funded by the National Science Foundation to facilitate people. The data doesn't come for free. And another way to do this is that we roll up our sleeves and generate the data ourselves."}, {"start": 8530.64, "end": 8560.22, "text": "Many of the things that were generated, yeah, they were able to do it. But anyone could have with sufficient resources and with sufficient elbow grease. And I think for much of AI that was data driven in the old days, we kind of assumed a culture where putting in the elbow grease to generate data wasn't something that was core to our effort. Whereas other disciplines, building an instrument for a year before you can conduct your research is much more common. So basically, I agree with you. And the ways to do that are varied. There's others."}, {"start": 8560.22, "end": 8587.22, "text": "Google has this prediction API, which is a way to provide access to data without exposing the data. Microsoft has the Ngram server, which is another way to provide access to a web crawl that doesn't give up the secret sauce and the magic ingredients that they have. So there's a range of ways, but it's untrivial. So if we can compose the services just right, then you don't need the original data. Right."}, {"start": 8587.22, "end": 8595.26, "text": "And that's, I think, part of the big trick is, do we know what services to expose so that you as a researcher can do just what you want to do?"}, {"start": 8595.26, "end": 8621.33, "text": "Or are we missing something that you can't get through the API? We need more experience in getting that right. And some of the most fun stuff about data are people using the data in ways that no one else expected. So in trying to expose the data in a way that allows you to do that is really a challenge. But then once you do that, you can start building services on top of services and middleware to support scholarly, reproducible results, which is another thing that we're not doing particularly well. So say someone from Google publishes a result on spam detection in Gmail."}, {"start": 8621.33, "end": 8654.11, "text": "That's not something that's going to be easily reproducible. And so the standards of science in a data intensive world is also a struggle for us to wrap our brains around, I think. The impact of AI on society. And the question here is, you're putting a lot of people, like you said, that because there's a lot of repetitive jobs in the economy, you're putting a lot of people out of work."}, {"start": 8654.11, "end": 8680.62, "text": "Much more efficient. Let me give you an example. For instance, one of the threats of Jeopardy is that if it's ever applied to the legal profession, 99% of the lawyers will be put out of work because most of them just... That's a good thing or a bad thing? So the question really is, you let loose..."}, {"start": 8680.62, "end": 8714.32, "text": "These amount of lawyers on our society, don't you feel that that is a danger? If they'll start working? I think the way you ask it calls for a trick answer that maybe if we are really freeing people from repetitive job, freeing them to do intelligent jobs, this is our contribution to intelligence."}, {"start": 8716.21, "end": 8747.54, "text": "I enjoyed part of it for three years. And I enjoyed it enormously here in MIT. Clay Shirky has this theory of cognitive surplus that says as people are freed up from doing all the drudgery, they can invent new things and use their creativity. We'll see how well that goes. I mean, I agree with Kobe that we should let computers do what they're good at."}, {"start": 8747.54, "end": 8776.72, "text": "Let people do what they're good at and they have fun with. Figure out how people and computers can work together to do the things that they can only do as a partnership. And figure out where it makes sense to do automation. And I remember Wernher von Braun was asked, what's the best kind of robot we can put into space? And he said, the best computer to put into space is a human being. And furthermore, it's the only one we can mass produce with unskilled labor."}, {"start": 8776.72, "end": 8802.13, "text": "And let me also just expand on the Clay Shirky comment. It wasn't Clay Shirky speculating, starting from today, looking forward. But he painted a picture where you go back decades, and we're seeing. So it's not just about the IT revolution. It's about the technology revolution and how it's changed that equation. So it's a very compelling argument. Why don't we take one last question from the audience?"}, {"start": 8802.13, "end": 8832.14, "text": "Google, Microsoft, and IBM in Jeopardy anytime soon. Everybody already thought Google can do it, so I don't know what the upside is there. Yeah, Jeopardy is a solved problem, so thanks to you. So we need something harder for the next challenge. That's right. We need the next challenge. It's all about trivia, so probably yes. I think the answer is no as a formal competition. Last question, do we have anyone else? Because that was such an easy one. One last question."}, {"start": 8839.92, "end": 8865.7, "text": "Could you comment on whether most of the content on the internet is HTML-based? It's the static content. And I was wondering if you had any thoughts about how the semantic web, quote unquote, going to RDF or L would leverage or make things more model-driven and more intelligent. You want to take it? I can comment."}, {"start": 8865.7, "end": 8892.42, "text": "I guess I feel like we'll try to make sense of whatever data is out there, whatever format it is, whether it's HTML or some markup language or video or photos or anything else. I think that to date, the semantic web content has only been in a couple of cases. One is where there's a database that's being exposed through the web."}, {"start": 8892.42, "end": 8921.92, "text": "And then we want to sort of reverse engineer the structure. And it's great if you can do that with formal markup. We also have some techniques to try to do it without the formal markup by inducing it in case the authors haven't put it out that way. And it tends to work best in a community that is cooperative, that want to work together rather than one that's adversarial. So I think there will always be limited applications of that. And I think for most of the data, we'd rather solve"}, {"start": 8921.92, "end": 8950.75, "text": "And I think ultimately, automated methods allow us to get multiple interpretations on the data, which I think is important. Because you can't always anticipate"}, {"start": 8950.75, "end": 8957.71, "text": "How your user is going to think about the data. OK, why don't we thank our panelists. Thank you so much."}]}