[0:00:02] Hello, I came here to win ugliest slide competition and I intend to do that. So I'm here to talk about a problem which may turn into a project and I guess the way I'll start that is by talking about a situation which happens quite regularly at OpenAI where I work. So we blindfold all of the research scientists and we have them identify mysterious objects in the office and one scientist will be saying oh well this is clearly you know the side of a dirigible
[0:00:32] And another one will be saying, well, clearly I'm standing next to an air vent because there's this hot air blowing over me. And another one will be saying, well, I'm actually putting my arms around the width of a giant tree. And so I guess the riddle is, what is it that the scientists are dealing with here? Does anyone have an answer? Let's make this interactive. It's an elephant. You've got the giant bulging side of it. You've got the trunk and the leg.
[0:01:01] And modern AI development is a bit like that. So on this slide, you've got a bunch of metrics here on different bits in AI which have advanced at different rates. So from top left, you've got the error recognition, the error rate on image recognition for ImageNet. You've got the error rate on speech for the switchboard corpus. To the right of that, you have the bits per subpixel, which is a measure of how well you've understood structure for generative models.
[0:01:28] On the bottom left, you've got perplexity on Penn Treebank, which is sometimes used in language modeling. On the two ones on the right, you've got data from Facebook's Babby, or Baby Corpus, which tells you about how well we're becoming at sort of reasoning about language riddles. And the reason why I presented it in the ugliest way possible is that there's no good way to present this kind of combined information. We know that AI is advancing across different discrete fields, and at OpenAI, we know that we're starting to train systems which are...
[0:01:58] Composites of these. You can train joint image and speech models. You can train models that are taking some sort of action along with their analysis. And really right now we have no good way to visualize this or to sort of make this explicable to people. So the reason why I want to talk to you about it is I think this is a project that actually needs many people in the community. I think it's something that we at OpenAI are going to be doing research into about how we do fruitful meta-analysis. But I also want all of you who work in AI to think about
[0:02:27] What is the capability that your systems have and how do you track it and how do you communicate that? Because if we get better at this as a community, we're going to be able to project forward more. In 2013, people looked at image recognition speeds and they made large acquisitions on the basis that deep neural networks would get better at that. They're also able to predict products and predict future market opportunities. And I think for stuff like...
[0:02:53] We don't have a good mental model for how to make predictions about this. So I guess in closing, I'll say please come and talk to me about it or read my newsletter, Import AI, where I reason about this stuff in the form of text rather than speech. And check out free projects from three big companies which are trying to tackle this. So Facebook has a project called ComAI, which is about language understanding. And there they're trying to evaluate the...
[0:03:23] Increasing capabilities of agents by exposing a relatively low bandwidth communication channel to them and seeing if they can invent language. This gives you a kind of sliced view of their internal representation. At OpenAI we have a project called Universe where we're testing AI capabilities across a huge diversity of a thousand environments. The idea being if it does well at that then we know that we're tracking something that refers to some kind of composite reasoning.
[0:03:47] And at DeepMind, they've started experimenting with multiple ways of visualizing the state of any given reinforcement learning agent. So these are all good sort of areas of food for thought, but I think there's a lot more that needs to be done. Otherwise, we're sort of like pilots in an aircraft without all of the dials, and it's sort of as hard to explain as this. So that's my problem, and please help me solve it. Thank you very much.