{"url": "https://www.youtube.com/watch?v=_r3yF4lV0wk", "page_url": "https://www.youtube.com/watch?v=_r3yF4lV0wk", "title": "Google AlphaGo and machine learning (Alphago Korean Press Briefing 2 of 2)", "show_or_channel": "\ub370\ube0c\uba58\ud1a0(Devmento : Real IT Portal)", "date": "2016-01-28", "duration_s": 1654, "type": "other", "speakers": ["Demis Hassabis", "David Silver"], "confidence": "medium", "notes": "AlphaGo Korean press briefing (Q&A, part 2 of 2), Jan 2016.", "people": ["demis-hassabis"], "id": "2016-01-28_devmento-real-it-p-google-alphago-and-machine-learning-alp", "audio_duration_s": 1653.690375, "worker": "jc-w12", "transcribe_s": 37.3, "model": "large-v3", "segments": [{"start": 5.68, "end": 34.35, "text": "And I have a question for Mr. Demis Havadis. I think you mentioned that the reason for this competition is to ultimately help solve real problems. And how can meeting a series of Paduk or Go games reach this purpose? Can you be more specific on the future of AlphaGo and the program and link it to the various applications that you've mentioned? Yes, sure. So..."}, {"start": 35.92, "end": 60.05, "text": "Basically, to try and beat a game like Paddock, you have to have quite a few different features and capabilities of your program. So you have to be able to use pattern recognition to understand the board. And then you have to use reinforcement learning for the system to improve through experience."}, {"start": 60.05, "end": 82.18, "text": "And we use a technology called Monte Carlo Tree Search to do the planning. So if you think about it, that's quite a general description of many real-world problems. You have a lot of data that you need to find structure in and find insights in. And then once you find those structure, you need to make long-range plans."}, {"start": 82.18, "end": 112.0, "text": "So we think actually the type of algorithms we're building, they're not specific to playing Go or Haddock. They are general algorithms that can learn from any kind of data. In fact, last year we used similar types of algorithms to learn to play Atari games, so computer games, just from the pixels on the screen, like a human would if they were looking at the screen."}, {"start": 112.0, "end": 141.73, "text": "So these are quite generalizable algorithms. So what we hope, maybe I can give one example of something we might be able to do in the future, is imagine you were planning a trip to Europe, a holiday, and you wanted to have your smartphone book you a holiday that would be very interesting. And so you need two capabilities."}, {"start": 141.73, "end": 165.34, "text": "That the algorithms would have to understand your preferences. So it would have to understand from the way you interact with it and the things you've done before, what kinds of things you like to do when you're on holiday. So whether it's visiting a museum or a concert or what types of restaurants, these kinds of things. So that would be the kind of pattern recognition part. And then you need the planning."}, {"start": 165.34, "end": 194.48, "text": "So then you have to not only book flights and hotels, but also plan an itinerary that would make sense and that you can manage in the time that you have and include as many of the interesting things that you like as possible. So that would require some kind of long-term planning. So that's an example of using the same set of algorithms that we're developing for Paddock, but now in a new setting. Another example might be medical diagnostics."}, {"start": 194.48, "end": 220.82, "text": "So, you know, maybe you get a scan, a CT scan or a brain image, and you have to process that image and maybe recognize something that's wrong in that image, and then help work out what the best kind of treatment schedule should be from, you know, given maybe what disease that you have."}, {"start": 220.82, "end": 250.74, "text": "And sort of recommend and work with the doctor and help assist the doctor in deciding what kind of treatments might be good. So those are two very different examples, but we think that the same kind of underlying technology and the same kind of algorithms will be useful in both those situations. Yes, can you hear me? I'd like to ask you a few questions. The first question is, can you explain and elaborate on treatment in AI?"}, {"start": 250.74, "end": 275.47, "text": "IBM and artificial intelligence of Google. Can you provide and explain the difference between IBM's AI and Google's AI? And the second question is, so if there is no big data, would it be impossible for these machines to learn? For instance, human beings also would need some sort of an input of data in order for humans to learn certain things."}, {"start": 275.47, "end": 305.26, "text": "So with the small amount of data, would machine learning be possible? That's the second question. And third question is, is AlphaGo now fully trained and educated on the patterns of Lee Se-dol, the Go champion? Has AlphaGo fully versed itself and understand the patterns and the way Mr. Lee actually plays? Or is he just coming out to the match without that prior?"}, {"start": 305.26, "end": 332.34, "text": "Hi, so let me take those questions one at a time. So first of all, regarding IBM's AI, I presume the question is specifically about IBM's AI in the game of chess, the deep blue advance, or possibly recently Watson in Jeopardy. So if I take those in turn,"}, {"start": 332.34, "end": 360.66, "text": "There were some very significant differences between what IBM did in Deep Blue when it learned to play chess, when it built. It was an amazing achievement, but it was based primarily on a brute force search technique, which was able to expand the search tree in the game of chess up to an enormous depth, typically 20 up to 40 feet, brute force look ahead. So this was a big achievement, but the evaluation used in that search was actually handcrafted."}, {"start": 360.66, "end": 387.15, "text": "by human grandmasters who created their own handcrafted rules to understand the game of chess. In contrast, what we've done here is to build a program which understands the game of padauk for itself. So it watches, first of all, human games and understands just from watching those human games using machine learning which patterns are effective without any hand crafting whatsoever."}, {"start": 387.34, "end": 410.35, "text": "The second big difference is that instead of using brute force search, it uses something much more human-like, where it does a much more selective path through the tree, where it imagines by, in its own mind, it plans ahead and imagines how the game might be played out. And this provides a much more effective way to search, despite the fact it actually looks at thousands of times fewer positions than DeepMu did in chess."}, {"start": 411.34, "end": 438.02, "text": "So just regarding if perhaps the question was about Jeopardy, so IBM have also done some amazing work in the game of Jeopardy, which was a quiz program shown in American television. And again, there's a big contrast between what was done there and what we've done here with the game of Red Hook. In that case, IBM used something which was based very much on a collection of special case systems."}, {"start": 438.02, "end": 466.7, "text": "that used databases of information collected from the internet in order to answer questions which were focused specifically on different topics that came up in that game of Jeopardy. So it wasn't a general path for the system that was able to learn for itself. And the second difference was it actually was using a somewhat shallower evaluation system based on just a simple layer of processing. What we do here at DeepMind is we build a system that learns for itself in a very general way."}, {"start": 466.7, "end": 496.51, "text": "So the second question, could we have been as effective without big data? It's a very interesting question. The answer is machine learning in itself can be effective"}, {"start": 496.51, "end": 522.72, "text": "But it must be showing some examples of the data which it's accessing. So what we did in this work actually was we built on two types of data. First of all, games that came from watching human experts. And secondly, from games that actually generated for itself. So I think even without human expert games, I think it's very plausible that in the coming years we will see machine learning algorithms based on reinforcement learning and approach that learns for itself."}, {"start": 522.72, "end": 551.41, "text": "which are able to succeed from scratch with no prior knowledge whatsoever and no examples, just by playing for itself and experiencing for itself and doing better and better. I should just add to that one. It is still true, even though the data, it could have generated the data itself, it still requires playing many millions of games. So it still needs a lot more practice, let's say, than a human expert player would."}, {"start": 551.41, "end": 581.2, "text": "So humans are still more efficient at learning, and that's something that we are working on as well in this project is to try and learn from fewer examples. So neuroscience, this would be called one-shot learning, where humans are able to learn even from just one example by using knowledge they have from other domains. So this is something computers still aren't able to do, but it's also something we're working on. And to take the third part of the question,"}, {"start": 581.2, "end": 605.86, "text": "So we haven't special cased our program to play any particular player. Of course, Mr. Lee is an exceptional Go player and we fully expect the challenge to be enormous. But right now we're just focusing on making the strongest possible player in general terms. And indeed it's actually rather hard to special case to a small number of games from any individual."}, {"start": 605.86, "end": 639.25, "text": "Because our machine learning techniques are very general and take as their input millions of human positions. I understand that Google is also deeply involved in autonomous driving vehicles. I wonder how your technology AI will be applied to those types of self-driving motor vehicles. So they have their own systems that are specially optimized for driving."}, {"start": 639.25, "end": 665.65, "text": "And the situations that come up in driving. We've worked with that team a little bit on braking systems. So there are some learnt systems there that act as backups to the special case systems. But mostly they have their own technology that is kind of specialized towards that one purpose. So that's not a particular area of focus for us."}, {"start": 665.65, "end": 684.46, "text": "My first question has to do with how did you come about to name this AlphaGo? What is the meaning of AlphaGo and how did you select that term? Second question is that I understand that the method that was employed for a machine also was a Monte Carlo method."}, {"start": 684.46, "end": 716.0, "text": "for a machine that participated in the worldwide Hadoop tournament. So how is that technology different from this Google's method? Can you elaborate a little more on those aspects? I'll just answer that first question. So AlphaGo has many meanings for us. So we wanted a name that was international. And we like the name Alpha because it can mean best or leader."}, {"start": 716.0, "end": 743.68, "text": "And also Google's parent company is called Alphabet. So it's a reference to that as well. So I'll take the second part of the question. So indeed it's true that AlphaGo builds on recent successes and advances in Baduk based on Monte Carlo tree search. The main difference between what we do in AlphaGo and the previous programs"}, {"start": 743.68, "end": 772.75, "text": "is that we use deep neural networks to guide and steer the search process. So specifically, when we actually run in simulation a particular game, so AlphaGo imagines how that game will be played out all the way to the end. That's the main idea of Monte Carlo search. But in AlphaGo, we actually use two neural networks to guide that simulation. So it uses a policy network to actually"}, {"start": 773.04, "end": 801.95, "text": "Only consider the most effective moves that are given by the neural network trained from human play. So in other words, you can think of this as being almost like a human standing over the searcher's shoulder, suggesting which move to play at each step of the simulation. And secondly, which is a big difference from previous approaches, we actually have something, which is the value network, which is able to evaluate a position in the game of Waduk. So we have a neural network which can look at a position and say, aha,"}, {"start": 801.95, "end": 824.46, "text": "I believe that Black is going to win in this situation. And as a result, what this means is that we don't have to go to the bitter end of the game. So all previous Monte Carlo search programs have evaluated positions solely by playing randomized games against themselves all the way to the end of the game and seeing who wins in these randomized games. In AlphaGo, instead..."}, {"start": 824.46, "end": 850.35, "text": "We actually have a neural network which can estimate who will win the game very accurately. And this is the first time that that's been achieved in the game of Paduk. I also have a couple of questions. First is, do you use TensorFlow? Or if you're not using TensorFlow, what is the dedicated framework that you are using? And also, can you actually train TensorFlow to actually play the game of Paduk?"}, {"start": 850.35, "end": 866.58, "text": "Let's play the game of Paduk. And when AlphaGo plays Paduk, between its moves, is it really quick, or how many minutes or seconds does it take for it to position this stone, the next stone, after the other?"}, {"start": 866.58, "end": 897.74, "text": "And I would think that going forward, we will see more AI machines based on machine learning that will start to play this game. What would be the most effective criteria, I guess, or a yardstick that we could use to evaluate one machine from the other in terms of how good they are in playing this game? Hi, so first of all, the question about TensorFlow. So we make..."}, {"start": 897.74, "end": 926.93, "text": "To train on neural networks, we use a large amount of computation spread across many machines on the Google Cloud network. And what we need to do there is actually we explore several different approaches. TensorFlow has been extremely effective for us, and we make heavy use of TensorFlow when we're training our policy networks and value networks. Secondly, in terms of the amount of time that AlphaGo uses, AlphaGo is a scalable, anytime search program, which means it will think for as long as we tell it to think."}, {"start": 926.93, "end": 956.61, "text": "And so the computation could be 0.1 second, it could be 10 minutes, depending how long you want it to think for and how accurately you would like it to search. So specifically in the game against Fan Pui, which we report in our article, both players were given one hour of time control for the entire game, plus another 30 seconds of Biyomi for three periods of Biyomi. Finally, in terms of the criteria for evaluation,"}, {"start": 956.61, "end": 985.36, "text": "One of the nice things about playing a game such as Badook is that you can evaluate head-to-head in tournament play. And I think this is the only sensible criteria for evaluating which program is the best. And so this is why when we evaluated against the current state-of-the-art programs for playing Badook, we played an extensive evaluation where we played 500 games between all these other programs to establish which player was the best."}, {"start": 985.36, "end": 1001.65, "text": "And so by doing this, we were able to really know for sure that we actually had developed something which was much stronger than what came before because we played head-to-head games against them all and discovered that we won in 99.8% of the games against each of these top programs."}, {"start": 1002.96, "end": 1028.72, "text": "You mentioned that AlphaGo actually recognizes or is aware of the environment or situation that it is in. I looked at a video on YouTube and I just saw a picture of two people sitting across one another playing this Paduk game. I wonder how does AlphaGo become aware of this board of Paduk? And does AlphaGo ever make a mistake?"}, {"start": 1031.09, "end": 1056.46, "text": "So we supply, so we had a human operator during the match and the human operator played the games physically on the board and also entered them into the computer using a standard interface. So actually the program AlphaGo itself is already told in machine readable format what the board position is for each move and what move was actually selected by the opponent."}, {"start": 1056.46, "end": 1085.39, "text": "And so this information is provided directly, so it doesn't have to extract this for itself by interacting with its environment directly. So for the second part of the question, does AlphaGo make mistakes? So AlphaGo, as we said, is based upon a Monte Carlo tree search, which means that it uses randomness at its heart. Now, when you are looking at the pattern of the project, there are two ways you can evaluate this."}, {"start": 1085.39, "end": 1112.38, "text": "The first is you can look at the size of the potential territory, and the second is you can look at the influence aspect. And depending on the actual player, the actual professional player, each person would have a different view in terms of that, when he or she looks at that pattern or that stage. So I'm just wondering how you integrated all these different components and elements. I guess what I'm trying to ask you is,"}, {"start": 1112.38, "end": 1142.64, "text": "How many different evaluation functions did you really look into? What is the diverse array of the evaluation function that you looked into or you used? And second question is that I see here that the objective function that you have selected is winning probability. But what about the potential size of the territory? Why did you not select that and selected winning probability instead? And can you speak a little slower? That would be helpful."}, {"start": 1143.15, "end": 1170.94, "text": "So in the game of Paduk, there is a very interesting balance between territory and influence. However, when we train AlphaGo to play the game of Paduk, we build just one system, one evaluation function, which learns for itself how to trade off territory and influence. We don't special case."}, {"start": 1170.94, "end": 1197.14, "text": "AlphaGo to deal with any specific piece of knowledge that humans think are important about the way to play the game. Instead, AlphaGo learns for itself how to win the games as effectively as possible. First of all, it watches human games to see how humans balance the many aspects of the game to give the most effective patterns and the most effective evaluation of the game."}, {"start": 1197.14, "end": 1223.97, "text": "Secondly, it improves this further, tunes it further by playing games against itself to make sure that it really understands the balance in just the right way. Not just between territory and influence, but every aspect of the game emerges by itself in this fashion. For example, in previous BADUK programs, there is a lot of special case code to deal with the different aspects of the game, such as Jiseki."}, {"start": 1223.97, "end": 1253.04, "text": "Tetsuji opening patterns, tactical patterns, ladders, end game, all these different aspects. The problem of code, typically in previous programs there are special case systems for all of these. In AlphaGo there is just one system which is learning how to win from any given position. We just ask two questions. One, what move should AlphaGo play?"}, {"start": 1253.04, "end": 1282.83, "text": "This is one of the neural networks. And the second question is, will AlphaGo win from this position? This is all it needs. Let's just add one other thing, which is quite exciting, is we're very excited to see what Mr. Lee Sedol says about the style of the play when he plays it, because it's too strong for us. We are only one damn amateur players, so we love playing, but we're not that strong. So we can't really evaluate the style."}, {"start": 1282.83, "end": 1312.83, "text": "Fan Kuei commented on the style being very human-like, but we'll be very interested to see if, you know, what one of the world's best players like Mr. Li will say about the playing style and, you know, things like is it preferring influence over territory. For the second part of the question, regarding winning probability, all Monte Carlo programs to date focus just on winning the game."}, {"start": 1312.83, "end": 1338.62, "text": "And this has been one of the reasons why Monte Carlo Tree Search has been so successful. In fact, when Monte Carlo Tree Search programs started playing the game, many Go experts commented on their cosmic style, by which they meant that it has a style that looks at the whole board and instead of trying to maximize score, tries to just win the game overall in a cosmic way."}, {"start": 1338.62, "end": 1367.42, "text": "The same is true of AlphaGo, but with much greater sophistication. So every step of the search, AlphaGo will pick the move which is most likely to lead to a win. We do not want to compromise chances of winning by maximizing score. For example, there may be situations where locally it is possible to improve the score, but the overall evaluation of winning may become less likely because there may be opportunities for the opponent to start to fight."}, {"start": 1367.42, "end": 1391.04, "text": "So do you have plans to maybe register AlphaGo with the Paduk Association of Korea so that it could play as a professional Paduk player? And if you lose..."}, {"start": 1391.04, "end": 1420.53, "text": "Who do you think will win this match against Mr. Lee? Would you be thinking about maybe re-challenging him at a later date? Or if you win Mr. Lee, who would be your next opponent? Are you thinking about Mr. Chang Ho Lee, who's also a Grand Baduk Champion? And I think the amount of the award is quite significant. With that amount of money, you could actually start and open a Baduk tournament yourself. Do you have plans to do that?"}, {"start": 1422.93, "end": 1449.15, "text": "So just to answer the first question, so we don't have plans yet to register AlphaGo as a professional player. We're just focusing at the moment on the match with Mr. Lee and then we'll see how that goes. If we lose, then yes, we would probably like to have a rematch."}, {"start": 1449.15, "end": 1474.69, "text": "But again, the details of that are still to be decided. And again, we're just focusing on this first match and hopefully trying to win that. If we win, then I think we would have to consider again what the next step would be. I think we would have to see how the games went and then maybe see if..."}, {"start": 1474.69, "end": 1491.54, "text": "The International Go Federation and the other Go Federations would accept a program maybe playing in a bigger tournament. But again, we haven't planned for anything on that front yet. Again, we're just concentrating on the match with Mr. V."}, {"start": 1491.6, "end": 1521.47, "text": "And the final question. Oh, yes. So we will be sponsoring a few tournaments, especially like in Europe and the UK and possibly other countries in the West to see if we can help popularize the game of Paddock around the West more than it is now. Because we would like to see the game become much more popular."}, {"start": 1521.47, "end": 1557.2, "text": "This is the last and final question. How many days was AlphaGo trained? I mean, let's think of AlphaGo as a human being on that basis. And if a human being were to be trained eight hours a day, every day, how many days would it take to receive the amount of training that AlphaGo received? So if we add it up, AlphaGo was trained for around four weeks of nonstop training."}, {"start": 1557.2, "end": 1586.85, "text": "Time and actually observing human positions, one after the other after the other. The way it trains is not, first of all, when it looks at the human games, it looks at them not in sequence, so it's not like a human playing through those games one after the other. But the total amount of time it spends processing all these positions is around one month. If we ask the question, how long would it take a human to actually play the games, equivalent to AlphaGo,"}, {"start": 1586.85, "end": 1616.77, "text": "It's clearly more than one lifetime. So AlphaGo is able to process when it plays itself by self-play and start to play games against itself. It plays on the order of around a million games. I believe top human professionals play something like 1,000 games a year. For the new generation, we play a lot online, which would mean, I guess, that AlphaGo is processing something like 1,000 years' worth of experience"}, {"start": 1616.77, "end": 1648.78, "text": "This is a very important question. What is your personal... Do you think you win or... I mean, Alphago wins or Isador wins? It's a good question. I think it is 50-50. So we'll have to wait and see. We are quietly confident, but I know Mr. Lee is as well. So I would say it's even chances."}, {"start": 1648.78, "end": 1651.28, "text": "Okay, thank you, thank you."}]}