[0:00:04] This is your first time here. That's very exciting. We're glad to have you. I didn't write that all down. That's okay. But basically, this is a meetup about discussing computer science and mathematics and specifically excited about the places where the two meet. The intersections between computer science and mathematics. Really, they're the same thing depending on whether you're a constructivist or a non-constructivist. They may be exactly the same thing or may be completely different foundations. Anyways.
[0:00:34] I digress. So first, I want to say some thanks. Oh, I'm Eric. I want to say some thanks. Number one thanks is to BoltMate, who housed us in this space. Thank you, BoltMate. Thank you, BoltMate. I also want to thank Christopher Collotte. Hi, Christopher. Because similar to the last presenter, Russell O'Connor, last week, when Stephen and I started intersections, I thought, who do I want to speak? I emailed Chris and Robert and said, if you're ever in the vicinity,
[0:01:04] Come and speak. And then he said, well, I'm in San Francisco being cool. I said, that's okay. One day you will return to Canada. And now we are here. So that's very exciting. So thank you for coming back and visiting. And to Matt Masuida, who's going to give his talk on vaccine of choice again to a greater audience. So a couple of reminders. One is that we do have a court of conduct. We have this so that we can have a nice, safe, encouraging environment. So I encourage you to read it all.
[0:01:34] But also, just one quick reminder on that is that we try, sometimes the topics we discuss are complex because we have all varying backgrounds, so we really want you to feel that you can ask any question. I know it's really hard and people feel shy, so I might ask some of the dumb questions to lower the barrier, but also, if you feel the urge to correct somebody's response, try to avoid saying, well, actually, and try to avoid correcting someone else's response. And if you really feel the need, try to phrase it in a...
[0:02:03] Very empowering to share with them. Nobody likes to be told that they're wrong even when they're answering a question. I will also be wrong on answering people's questions, so feel free. Do not do that to me. I'll feel bad. Yeah, so we have two speakers today. Christopher, who's a, well, we won't share the semantics, is a Teal Fellow or used to be a Teal Fellow. Peter Teal is the libertarian CEO of PayPal.
[0:02:33] And he's now an intern on Google's deep learning team, which I'm sure is extremely exciting. And he's coming here. I don't know where you're going to be living forever, but he's coming here from San Francisco, apparently, to give this talk. And we'll be giving a talk on neural networks. We have Matt Masuria, who is a durian enthusiast, once brought a durian to one of our talks. We'll never bring a durian again. He's a postdoc at U Waterloo and a Banach algebraist.
[0:03:02] So, thanks everyone. Chris will start. Take a look. So, actually, I'm just curious. Who here has any sense of what's been going on in machine learning and artificial intelligence in the last three or four years? Some. So, what I think a lot of people don't know is that we live in really exciting times for machine learning. In 2012, Alex, Jesse, Elias, Ed, Steve, and Jeff
[0:03:37] And the rest of Toronto went and published this breakthrough paper in computer vision using what we call deep neural networks to go and solve a computer vision problem, classifying images. So you're trying to recognize the object in the image. And they can be in a thousand different categories. This is called the ImageNet. And so over here, we have an image of a mouse, an image of a computer ship, and a motor scooter, and a leopard, and so on. And we've got those ones right in these confidence bars. And then here, somewhere in this talk, I feel surprised.
[0:04:03] So this one I thought was a Dalmatian. I can't imagine why I thought it was a Dalmatian. And this one I thought was Anger because it's top guess. And I'm pretty sure this one I thought was a Convertible instead of the grill on the front of the Convertible. Its top guess is right 63% of the time. And one of its top guesses is right 85% of the time. So a couple months after this paper got published, all three authors mysteriously ended up at Google. I was lucky enough to go and work on the same team as them.
[0:04:30] And they've continued working on this sort of stuff. And just a couple months ago, they went and they published a follow-up result. Well, actually, it was not them, but a very large number of people at Google. And now we're at 90, the top five, one of the top five guesses is right, 93.3% of the time. And human accuracy is 95% of the time. So this is a really hard efficient problem. And we're getting pretty close to approaching human accuracy. And another really neat result, also out of Google.
[0:05:01] You go and you take an image, and you go and you generate a caption. A bunch of other people did similar things, and they all got released around the same time. So here we have a picture of people at the market, and the model comes up with a group of people shopping outdoors at a market, at an outdoor market, excuse me. There are many vegetables in the fruit stand. So, I mean, maybe fruit stands are the best thing to say there, but it's really, I think, quite impressive that a model can go and look at the image and generate an English sentence describing it. That's pretty cool.
[0:05:30] There's been lots of other neat things. So there was a couple years ago a startup called DeepMoney who went and demonstrated that you can go and go into playing Atari, learning automatically from playing an Atari video game over and over again. Their model discovered how to play it really well and got superhuman performance on a bunch of things. And it just sees the image. It doesn't get a process to manage the image. It goes and helps it figure out what things are or something that works directly.
[0:06:00] So there's been lots of other exciting results. Voice recognition, and sorry, I forgot to start my timer. Translation, so we just recently, neural networks have gone and matched state of the art for translation, and it seems really likely that there will be a lot of progress in the next year and they'll exceed it. Learning to interpret Python programs automatically from seeing examples of programs and their output. Going and discovering algorithms automatically.
[0:06:28] This is really new stuff, not super impressive yet, but the fact that they can do it is really cool. Discovering mathematical identities automatically, similar story. So, I don't know, it's pretty exciting stuff. But it's not quite what I want to talk about today. Because I think that probably a lot of the details are, well, we could get bogged down in the details, and a lot of them...
[0:06:50] are kind of arbitrary. We think that probably we've discovered something that works, but it's not fundamentally really the truth of the universe is that this is how you go and you solve these problems. We just found something that happens to it. And so a lot of these details probably aren't fundamental. And the details keep changing. So if I go and tell you all the details of neural networks, probably five years, a lot of it will be different. So what I want to talk to you about...
[0:07:19] It's fundamental because we're here and we're a bunch of math and computer science people. I think most people here are kind of interested in both. And so what I want to get is something that's deeper than just, you know, we have this model that works. I want to go and talk to you about two things. So the first is, I want to talk to you about, I want to ask, what really is the problem that we're trying to solve? Not, I mean, you can say, you know, we're trying to classify images, but what does that really mean? And then secondly, why is it that neural networks
[0:07:47] are so successful at solving these problems. So obviously those are both really ambitious questions and we have 30 minutes to discuss them. I guess it's fortunate that we're not doing something even more ambitious because that's already scary. Oh, sorry, I was supposed to say what I think. I think that the problem fundamentally comes down to untangling something in really high numbers of dimensions.
[0:08:14] And we'll explain why we might think that's the case in a little bit. And I think that the reason that neural networks are successful fundamentally comes down to function composition, which is kind of an interesting thing. So when I talk about pattern recognition, I typically mean things like going and looking at images and going and predicting what's in the image, meaning where it is, things like that, or going and looking at...
[0:08:44] You know, having a little segment of somebody, audio of somebody speaking and trying to recognize what word it is, or what the phenomes are, or things like this. And what I want to try and do is get into, what does it really mean to solve these problems? So, I guess, the first question is, can we write a program to go and tell these images apart? We have a 5, 0, a 4, and a 1. And it might seem, initially, if you haven't tried to solve a problem like this, like, this is an easy problem.
[0:09:12] You know, a four has a vertical line and a horizontal line and another vertical line, and one's just a single line. But it turns out that you have all these really high-level intuitions about what digits look like, and it's really hard to go and translate them into something concrete that you can act on. They're kind of, the more you try to go and implement them, the fuzzier they turn out to be. And there's lots of weird exceptions and all sorts of things like this that start popping up.
[0:09:36] So it seems like an easy problem because your brain has these really extremely sophisticated tools for trying to process image data. It happens before you become consciously aware of your vision. But it's actually a really hard problem. So we need to classify those. We can think, if you give me an image, we can go and think about it as a matrix. So we just have an array of numbers. You know, it's zero in a lot of places because there's nothing there. You know, in the middle where we have the actual image, we...
[0:10:10] Or the actual one, we start to get higher numbers. It's gray, so we have some things that are around the five. And then at the very center, we go up to one. We can take that matrix if we want, and we can flatten it into a vector. So it's a 28 by 28 matrix. I actually didn't include the entire matrix in some sample, but it would be a 28 matrix. And we can go and flatten it into a 784-dimensional vector. Yes? Hey, so dumb question time. What's the approach to flattening it?
[0:10:42] It doesn't matter. We do not care how it is indexed. I just want to go and have every entry in the matrix be an entry in the vector. So there are some cases where we care about the structure of the matrix. For modern neural nets, there's a lot of tricks that you use to go and do a better job. But for the general idea, we're just going to forget about any structure we have except that there's all these different dimensions which correspond to the intensity of some pixel.
[0:11:13] Typical images. Sorry, actually, there was a second question. I should double check. Oh, I just raised my hand. Excellent. So we can go and look at images. And we can take a look at, sort of ask, what would a typical vector in the space look like? And, well, they look really different from our actual images of digits. And so what that tells us immediately is that the images that we care about, they form a very small set.
[0:11:44] They're embedded within the space of images. But most images aren't images. Most images, 20 by 20 grayscale images, are not images of digits or even really sensible images at all. And we tend to think, if you sort of think about how you would go and transform these images in one way or another or deform them, there's a fairly limited number of ways you could do that that would preserve them being an image. Certainly there are 704 different ways you could go in.
[0:12:15] Distinct ways, orthogonal ways that you could go in and transform it. Like if you think, just, you know, starting to go in and drop in a pixel somewhere, that will go in and continue making something a digit. So, we think it's something lower dimensional. So, there are manifolds in mathematics are sort of a generalization of surfaces. So, if you go and you have a, say, a sphere, well, a sphere is an example of a two-dimensional manifold, it's a surface. And it's locally, if you zoom in,
[0:12:46] You know, it's perfect, but if you look close enough, it looks flat. And you can have sort of three-dimensional things embedded in five dimensions or things like this. So we have something embedded in another space, and it's kind of, it's locally linear. So locally, if you zoom in closely, it looks like it's flat. And so the novel hypothesis is that, very generally, when you have data,
[0:13:11] From different sources. It tends to form some sort of manifold embedded in this really high dimensional space. And we don't know if that's true. It's probably, I think, it's probably kind of true. So I think people usually who hold these sort of views, they usually don't quite literally mean it. They don't necessarily mean that it's locally linear everywhere. Maybe it has a sharp corner here.
[0:13:38] Maybe over here the dimensionality changes, but people sort of think that something kind of in that similar flavor is true, if not the strong one. So my question is, can we see it in that sense? So recall that because we went to mean, we took our matrix, every entry was the intensity of a particular pixel, and then we flattened it. In the 784-dimensional space we have, every direction is the intensity of a particular pixel. So it goes from...
[0:14:11] And you can sort of imagine that to be a little bit like a snow globe. So the general story here is that if you go and you stare into this, you sort of imagine we have a 784-dimensional cube, and you stare into it. And when you stare into it, it turns out that you don't see anything interesting. The structure just...
[0:14:42] One thing is there's lots of angles we can look at this cube from, and we're, in that particular visualization, we're just going to look at it straight off. So we're going to try and look at it from a nice angle, and I guess the next question is, is there some way we can find the perfect angle to visualize it from? And there is. There's a technique called PCA that I'm not going to go into, but it gives us a technique for going, and if you have some really high-dimensional data, we can sort of find the perfect angle that goes and preserves, in some sense, the most structured look at it from. Over here, you can see we have blunts over here.
[0:15:15] And we have zeros over here, and nines over here, and sevens, and twos, and stuff like that. And so we can see a bit of structure. The ones came over here, and the zeros are over here. But we're not really understanding. So I feel like that's not very good. So maybe there's some way that we could try to explicitly grab this manifold if it exists. Is there some way that we could try to extract it? And so one thing we could do.
[0:15:45] As we can go on thinking about every single point, we can ask, what are its nearest neighbors? So maybe let's look at its three nearest neighbors. There's three points that are really close by. Well, what do those ones connect to? What are closest to those? And so we get this graph. It's a nearest neighbor graph. And it's sort of this manifold, and you try to follow it. And as we add more and more points, you try to follow it better and better, because we'll start to fill in the gaps, and we'll be able to sort of follow all of them. And there's techniques for visualizing graphs.
[0:16:14] One next one is we go and pretend that every one of the connections between these, the edges, is a spring. And we go and pretend that every node, every point, is just a charged particle. And so all the particles propel, the springs hold them back together, and let's watch this happen. So this one, I know, this one I was worried about. Here is the points slowly going around to try and find the good way to be. They have good space, springs, and they're doing their job.
[0:16:56] So each color corresponds to a particular type of image. So the reds are the zeros, for instance. See? Over here, all of these are zeros. Those are sixes. Those are twos, and so on. So are they moving towards some kind of equilibrium state, where they've minimized the energy? They're moving. They're going to try and minimize the energy. And they'll get stuck in the local minima, probably. I don't have any guarantees that it's going to be the perfect solution that they'll get to.
[0:17:25] Because we're just trying to use this visualization, if it just comes to something good, then that corresponds in some sense. In some sense, we've turned visualization into an optimization problem. We've described, so now I'm starting to use some, this doesn't particularly matter for the purposes of this talk, but we've described, it's kind of interesting, we can go and turn a visualization problem into an optimization problem and optimize it to try and get sort of the closest possible thing or the closest thing we can find to the ideal visualization.
[0:17:53] So the colors on the nodes, are those based on an algorithm's categorization? Those are the answers. So right now we're not trying to understand any algorithm, we're just trying to understand what the actual structure of the data is. So you can see that the zeros are separated substantially from other things. There's a few things that are connected to it, so let's have a look at that one over there. It's a 6, and it's a movie, a 6 that has a really big tail on it, I guess. So it looks a little like a zero.
[0:18:22] So there's stuff going on. And as we add more points, this would get a little bit cleaner. But it's very tangled. So as we'll see in a minute, actually, it sort of gives us a bit of intuition for how easy and difficult it is to unclassify this. It's pretty easy to go and get 90, 95% accuracy. You can just sort of block off these growths that are sticking out of it. So you can, if we have over here, zeros for instance, it's pretty easy to go and just...
[0:18:49] So there seems to be some sort of structure here, but it's sort of difficult to understand because it's very tangled. Another way we can think about this is we can think about it as what happens if we translate an image. So we imagine translating an image or even just, you know,
[0:19:32] One dimensional in which we just have a bunch of pixels in a row. And we translate, and we only have one pixel lit up. What happens? Well, initially we start with that one pixel. It's got its full intensity. But as we translate it, the intensity shifts into the next pixel, and then into the next pixel, and the next pixel. So it's kind of like a wave. But it's a really strange wave, because every half period, the wave oscillates into a different dimension, and it never returns to the same dimension. So it's a...
[0:20:02] I don't know, a multidimensional wave or something where it goes in and oscillates every step into another dimension. And of course, an actual image is just a single pixel. It's a lot of pixels, so we'd have to go and superimpose a bunch of these waves to get the actual manifold. But this tells us a little bit that there is at least some transformation we can do that preserves the image being some sort of valid image. If you translate a 1, it's still a 1.
[0:20:31] And it goes and it scoops out some crazy, just that translation. Because that's just one way we can transform it. That creates some crazy curve. And I say crazy because it's going in, every step it's going in switching dimensions and ending up just a very small transformation. It goes and causes it to be naively in a different position in space. So we have this wave that, it's sort of, it's almost like a corkscrew. It sort of goes and circles or spirals around going into a new dimension each time. So it's kind of...
[0:21:02] Very, very complicated structure. We can think about other things, so we translate on two axes, we can scale, we can go and rotate a little bit, we can go and, you know, perform different elastic distortions and things. All of these, you know, it's not quite as elegant as translation, but they do something similar to this. So this manifold, if it exists, it's going and flowing, like, wildly through the space. It's not just sort of gently going through the space. It's flying around in all sorts of crazy ways.
[0:21:32] It also suggests some kind of repetition. So this is kind of a silly data set. I just have these two classes. There's a red class and a blue class. And, you know, a point here is in the red class, a point here is in the blue class, and a point here is in the blue class. And my only point with this is just that it has a lot of repetition. It's the same thing over and over again in lots of places. And I think that probably a lot of real-world data has something similar to this, where there's the same thing going on in lots of different places.
[0:22:08] But it's more complicated than this. They're in different dimensions, and maybe they're scaled and transformed. Something of this flavor seems really likely to be true, and I think people generally have the intuition that it's true. So if we're trying to actually solve this problem, we have a bunch of points where we know when the answer is. We know that that particular point, that vector corresponds to a 1. That vector corresponds to a 0. That vector corresponds to a 3.
[0:22:41] And maybe we think about that if we want this to be continuous. That vector corresponds to a 100% confidence that it's a 1. And this one corresponds to a 100% confidence that it's a 3. And we get probabilities in between, so it's continuous. And we want to interpolate that function. And so mathematicians have lots of techniques for interpolating things. But it doesn't quite work. And the problem is the reason why it's not easy.
[0:23:08] So there's a few basic techniques we can do.
[0:23:36] One is called nearest neighbors, and the answer is go and look at the closest points that we have and go and have them vote. So, you know, we saw a 1 over there, we saw a 1 over here, and I thought it was close to me, I'm probably a 1. Or maybe, you know, if there's a 2 nearby as well, well, I have 60% confidence it's a 1, that I'm a 1, and a third of my confidence is that it's a 3, or something like that. So, nearest neighbors is kind of interesting. There's also the idea of a linear SEM, and we just try to sort of...
[0:24:07] The farther you are on one side of the line or the other, the more confident I am that you're more... Nearest neighbor sort of feels like a fundamentally right thing, in a sense, right? How can you do better than guessing whatever the closest thing that I've seen is, or maybe some sort of vote between the closest things, or some weighted vote or something like that? That seems like the right thing to do. And if I had an infinite amount of data...
[0:24:35] It would simply be the right thing to do. If I had all of the data, it's basically doing a lookup table, and I could just answer my question that way. But when we're trying to interpolate, sometimes that's not a very good thing to do. So here, I have two curves. A red curve and a blue curve. And we can see that they're both cosine waves. Of course, I guess they're sine waves. And if we look at them, we think that humans would say, well, that's very important news, and thanks.
[0:25:04] We just look at things locally. The answer that we come to is not that. And there's all of the, well, me and Ed were something very different. And so what we wanted to do was extract something from the global structure about the data and then go and finance it. You could imagine going through and learning some sort of change of basis. So if we could discover the right change of basis to go and sort of flatten out this way.
[0:25:31] Then we could use a linear SVM because they'd both just be flat lines with a bunch of points on them. We'd get the perfect answer. So in some sense, the thing that's missing here is the change of basis. If we get the right change of basis, a linear SVM would be a great approach, just separating with a line. So neural networks are basically that. We go and we try to go in, we have multiple layers. And what the early layers do is they go in the end of the data. So here we only have one layer that is in the upper layer.
[0:26:07] And you can see that it's trying to classify these two curves. We have a blue curve and a red curve. If I give you a point on the red curve, you're supposed to say it's the red bus. If I give you a point on the blue curve, you're supposed to say it's the blue bus. And I'm filling in other places so we can just sort of see what the algorithm is doing. And so what the neural network does is it bends the data and then divides them with a straight line like an SEM, and it answers the question. And so here's it going and classifying two curves. What it does is it goes in and just bends the data.
[0:26:44] These are what I call low-dimensional rail lines. So because they're operating just on a low number of dimensions, they're not operating on 724 dimensions, or most data that we use for real problems is even a higher number of dimensions, we're able to go and explicitly see what they do and try to explicitly understand what it is that they're doing. Unfortunately, real problems are high-dimensional, and it becomes harder.
[0:27:14] Dimensionality reduction, like the graph technique I showed you, to try to understand what's going on in these bidimensional spaces.
[0:27:46] And then we try to answer the question. Here it's really helpful. So one reason, if you win the Manipal hypothesis, is that they can go and contract some sections of the Manipal to bring data closer together, and they can stretch the space between them, and they can sort of bring out the important differences and sort of get rid of some of the less important differences, and they can bend things so that they can be cut apart more easily, and stuff like this.
[0:28:16] And if you think about it, you can interpret that as sort of building on these high-level features, these high-level sort of concepts in the data. But another reason, you might know that I gave this example of the data set before. And you can sort of imagine looking at it, just a bunch of, we sample the data sets, we get a bunch of points from the red glass and the blue glass. It looks something like this. I'm very, very faintly including the actual answers behind there, but it's hard to understand.
[0:28:49] And if you just looked at this and you didn't have the answer behind it, when you invert, there's a plus sign in it, there's a red circle in it, there's no reasonable way you can do that. But what I'm not allowed to do is if you want to recognize that there's a repeating pattern, and it can fold. If I fold this on the x-axis and the y-axis once, it would be more plausible that we could start to understand the real structure. And we could fold again. Maybe you only fold on this axis here, so we get this over here, then this one here, and then you can go and fold it, you know,
[0:29:25] And that allows us to have a lot more data to go and understand the local structure. And so this is called non-local generalization of neural networks. And it's sort of, it's been demonstrated that it works. I don't have any explicit visualizations, but there's a nice paper by Yoshua Bengio. So how do we train the network?
[0:29:55] I said that this is sort of a nice way to solve the problem, but how do we figure out, you know, if we have a bunch of these layers and they start off doing random things, how do we figure out what they should actually be for? And so the nice thing about these models is that they're differentiable. We can take derivatives through the entire thing. And so I can, given a whole bunch of examples, I can try to go and ask, how can I tweak the model to go and make it slightly better? And I use an algorithm called Gradient Descent, where I just try to go, every step I go on, I try to go a little bit better and move in a direction that makes things a little bit better.
[0:30:25] And eventually, I hopefully end up doing pretty well. Maybe I don't get perfect, but I end up with something pretty good. So what's important here? Well, so the study of neural networks is pretty young. I mean, the idea's been around for a couple decades, but these modern deep neural networks have only been around since the late 2000s. And so we have a very young field here.
[0:30:55] I think something that's interesting about young fields, or at least something I see in neural networks, is that there's competing narratives. It's not decided what the answers to these questions are. There isn't some theory that everyone accepts about why things work. There's lots of different stories going around, and people have different views.
[0:31:13] I've sort of, I've entirely avoided going and talking about this in terms of neurons, but there's people who like to think about things in terms of neurons and it being like the brain, and historically these models sort of came from trying to go and, originally, many, a long time ago, going and trying to create neuroscience models, and it turns out that's not very real. These aren't very realistic models of the brain or neurons or anything like this.
[0:31:41] But that narrative still persists a little bit, especially in the popular media. There's this new narrative, this manifold hypothesis one, where we sort of think about things geometrically. You might guess that I am quite biased towards this one. There's more probabilistic interpretations where you think of, you can think of each dimension, so the way that you get this neuron interpretation is we think of each dimension as being the amount that a neuron fires. So if you have two neurons, this one fired a bunch of this.
[0:32:16] And then over here, this one fired a bunch, and this one fired a quarter, and then this one fired a ton, and this one fired all, and both fired, and stuff like this. And I'm still thinking about the neuron story. But you can also go and think of these as sort of the variable, the sort of probabilistic beliefs throughout the world. So each one of these independently represents some sort of proposition that we have some confidence in, or something like this.
[0:32:44] And so on. So we have all these competing narratives. So what I want to give you is my own twist on this narrative that we have about the manifold hypothesis. And it's something that I've been developing in my head for a while, and it's not very mature idea yet. I hope in the next couple of months to go and publish something about it. I'd like to talk about it here as ideas.
[0:33:14] It's something that was kind of relevant. We're interested both in computer science and against functional programming in this area, which is also an interest in this area. So what are these layers except a chain of composed functions, of really simple composed functions? I mean, I think that's what they are. And what exactly the functions that we have in each section has changed a bit. We used to have what are called sigmoid layers. Now we have volume layers.
[0:33:45] Using a linear transformation to stay consistent, but a lot of things have changed and sort of the advice about exactly how to set them up has changed and stuff like this. But the things that's remained consistent and that we think is, I think, sort of everyone thinks is really important is having this chain of layers, having a deep model that seems really important. And it seems like there's something really interesting that happens when you optimize, you apply gradient descent or you try to use some sort of other optimization process to optimize a chain of functions.
[0:34:15] And I don't have any sort of deep theorems about that. I just have empirical evidence that these models, when we try to go and use these chains of composed functions and try to optimize them, we get these really impressive results. And part of it might be because they can go and fold the data or stretch the data, and that's really all they can do. And most of the model's capacity is only dedicated to going and stretching the data, and only a tiny bit of the model is dedicated to actually answering the question. And so maybe something interesting happens because of that.
[0:34:44] I don't really know. It just seems that when we have these chains of proposed functions and we optimize them, something really interesting happens. It also happens that actually we're starting to see neural networks where there's other kinds of higher order functions being posed. So in translation, they basically do is they go on, you have a sentence as your input and you have a sentence as your output. And the model of Google that's been released, that's matched the person's skin with other techniques, basically does a fold.
[0:35:14] So we have a small neural network that folds, going and gobbling up the sentence. And then another one that doesn't unfold to go and produce a new sentence. So that's kind of interesting. In other areas of people going and trying to do natural language processing, so trying to answer questions like, is this a happy review or a negative review? We've seen what people call tree structure networks. And what these are doing is they gobble up a tree in a fairly natural way.
[0:35:43] And so what they do is they go and they have these nodes, and they just go and they, you know, they have a parse tree of the sentence, and the first thing goes and transforms each word into a vector representing the word, and then they go and combine this together to get a vector representing, you know, two words, and then larger fragments, and they go up a tree, and they get a vector representing the whole sentence, and then they classify that, and it works really well. So there's something, you start to see these examples of higher order functions.
[0:36:13] With differential components embedded inside them, being optimized. It's sort of a recurring theme that we're seeing in different places. And then if you try to go and think, well, if you try to go and describe this, you can't just go and impose any two together. They have to go and agree on how the data is embedded. They have to agree on their representations. So, sorry, let me introduce the term representation. By representation, I just mean how the data is embedded here. So, this is one representation of the data. This is another representation.
[0:36:42] And the layer assumes that the data is going to be in a particular representation. It's a lot like, you know, I'm putting the data in an n-dimensional space. But just like when I go and I encode normal discrete data on a computer in n bits, I have to choose some way to go and embed them in n bits. And so here I have to choose some way to go and embed my data in n dimensions. It seems like the way I do it is really important. And I can only compose functions together when they agree on how I'm going to do that.
[0:37:09] They learn together, they sort of come to a consensus about how to do it by being composed together and learning together. But these representations, they have a field of types. And then when you start to go and talk about some of the more sophisticated models and exactly how the representations relate, you start to see dependent types. You start to see examples where there's a variable that you can go and have different kinds of input and it results in different kinds of output and stuff like this. So, I'm...
[0:37:39] It's just an interesting sort of flavor to it, right? And I don't have any evidence that this is the right way. But you can sort of imagine that maybe in 30 years, looking back at it, people are going to say, well, what were they doing with those deep learning people? They were getting all those really good results they thought they were going and dealing with neurons. But really what they were doing was that they were going and studying the interaction between optimization and composition, or studying differential functional programs, or something like this. I don't know if that's going to be the case.
[0:38:08] I'm not very confident that it would be. I think it probably won't. But I think it's plausible. And I think it's a really interesting direction. It suggests a number of research directions, at least. So it's an interesting idea. I didn't talk very much about actual neural networks in this talk. We had a limited amount of time. But if people are interested, I want to refer you to two books. First of all, Michael Nielsen, who is a friend of mine and a really excellent writer.
[0:38:44] You might know him for having written the standard textbook on quantitative giving. He has been writing a book on that. And it's very introductory and approachable. And he is an excellent writer. Also, Yoshua Bengio, Aaron Kerrall, and Ian Goodfellow from the Labrador de Informatique de Sistema Dept. in Montreal, the University of Montreal, have been writing a book. And it's more comprehensive. Yoshua is one of the leading authorities on neural networks.
[0:39:15] I also go and write a blog about neural networks and other things. I also go and write about neural networks. Some of the writing is about, some of the ideas in this talk are there. The functional programming thing is something I'm working on.
[0:39:45] They'll get published at some point. But there's lots of visualizing data and exploring the structure of high dimensional data and some posts going and reviewing the directions of research and stuff like this. So that might also be of interest. That's all. So I am happy to answer any questions.