[0:47:12] Thanks for coming. And sorry, I had some technical difficulties. We really need AI for this. So what we want to show today is that Tesla is much more than an electric car company, that we have deep AI activity in hardware, on the inference level, on the training level. And basically, I think we're...
[0:47:41] Arguably, the leaders in real world AI, as it applies to the real world. And those of you who have seen the full self-driving beta, I can appreciate the rate at which the Tesla neural net is learning to drive. And this is a particular application of AI, but I think there are more applications down the road that will make sense. And we'll talk about that later in the presentation.
[0:48:09] Yeah, we basically want to encourage anyone who is interested in solving real-world AI problems at either the hardware or the software level to join Tesla or consider joining Tesla. So let's see, we'll start off with Andre. It doesn't sound like it. Okay, hello. Great. Okay, hi, everyone. Welcome. My name is Andre, and I lead the vision team here at Tesla Autopilot, and I'm incredibly excited to be here to kick off this section, giving you a...
[0:48:56] So we're going to start off with the vision component here. Now, in the vision component, what we're trying to do is we're trying to design a neural network that processes the raw information, which in our case is the eight cameras that are positioned around the vehicle. And they send us images, and we need to process that in real time into what we call the vector space. And this is a three-dimensional representation of everything you need for driving.
[0:49:25] So this is the three-dimensional positions of lines, edges, curbs, traffic signs, traffic lights, cars, their positions, orientations, depths, velocities, and so on. So here I'm showing a video of, actually hold on, apologies. So here I'm showing the video of the raw inputs that come into the stack and then neural processes that into the vector space. And you are seeing parts of that vector space rendered in the instrument cluster on the car. Now,
[0:50:11] What I find kind of fascinating about this is that we are effectively building a synthetic animal from the ground up. So the car can be thought of as an animal. It moves around. It senses the environment and acts autonomously and intelligently. And we are building all of the components from scratch in-house. So we are building, of course, all of the mechanical components of the body, the nervous system, which is all of the electrical components, and for our purposes, the brain of the autopilot, and specifically for this section, the synthetic visual cortex.
[0:50:38] Now, the biological visual cortex actually has quite intricate structure and a number of areas that organize the information flow of this brain. And so in particular, in your visual cortices, the information hits the light, hits the retina, goes through the LGN all the way to the back of your visual cortex, goes through areas V1, V2, V4, the IT, the ventral and the dorsal streams. And the information is organized in a certain layout. And so when we are designing the visual cortex of the car,
[0:51:07] We also want to design the neural network architecture of how the information flows in the system. So the processing starts in the beginning when light hits our artificial retina. And we are going to process this information with neural networks. Now, I'm going to roughly organize this section chronologically. So starting off with some of the neural networks and what they looked like roughly four years ago when I joined the team and how they have developed over time. So roughly four years ago, the car was mostly driving in a single lane.
[0:51:36] And so it had to keep lane, and it had to keep distance away from the car in front of us. And at that time, all of processing was only on individual image level. So a single image has to be analyzed by a neural net and make little pieces of the vector space, process that into little pieces of the vector space. So this processing took the following shape. We take a 1280 by 960 input, and this is 12-bit integers streaming in at roughly 36 hertz. Now we're going to process that with a neural network.
[0:52:04] So we instantiate a feature extractor backbone. In this case, we use residual neural networks. So we have a stem and a number of residual blocks connected in series. Now, the specific class of resnets that we use are regnets, because regnets offer a very nice design space for neural networks, because they allow you to very nicely trade off latency and accuracy. Now, these regnets give us, as an output, a number of features at different resolutions and different scales.
[0:52:32] So in particular, on the very bottom of this feature hierarchy, we have very high resolution information with very low channel counts. And all the way at the top, we have low resolution spatially, but high channel counts. So on the bottom, we have a lot of neurons that are really scrutinizing the detail of the image. And on the top, we have neurons that can see most of the image and a lot of that context, have a lot of that same context. We then like to process this with feature pyramid networks. In our case, we like to use bi-FPNs.
[0:53:00] They get to multiple scales to talk to each other effectively and share a lot of information. So for example, if you're a neuron all the way down in the network and you're looking at a small patch and you're not sure if this is a car or not, it definitely helps to know from the top layers that, hey, you are actually in the vanishing point of this highway. And so that helps you disambiguate that this is probably a car. After a BI-FPN and a feature fusion across scales, we then go into task-specific heads. So for example, if you are doing object detection, we have a one-stage YOLO-like object detector here.
[0:53:30] Where we initialize a raster and there's a binary bit per position telling you whether or not there's a car there. And then in addition to that, if there is, here's a bunch of other attributes you might be interested in. So the XY with height offset or any of the other attributes like what type of a car is this and so on. So this is for the detection by itself. Now, very quickly we discovered that we don't just want to detect cars. We want to do a large number of tasks. So for example, we want to do traffic light recognition and detection, a lane prediction, and so on.
[0:53:57] So very quickly, we converged in this kind of architectural layout where there's a common shared backbone and then branches off into a number of heads. So we call these, therefore, HydraNets, and these are the heads of the Hydra. Now, this architectural layout has a number of benefits. So number one, because of the feature sharing, we can amortize the forward pass inference in the car at test time. And so this is very efficient to run because if we had to have a backbone for every single task,
[0:54:26] That would be a lot of backbones in the car. Number two, this decouples all of the tasks. So we can individually work on every one task in isolation. And for example, we can uprev any of the data sets or change some of the architecture of the head and so on. And you are not impacting any of the other tasks. And so we don't have to revalidate all the other tasks, which can be expensive. And number three, because there's this bottleneck here in features, what we do fairly often is that we actually cache these features to disk.
[0:54:52] And when we are doing these fine-tuning workflows, we only fine-tune from the cached features up and only fine-tune the heads. So most often, in terms of our training workflows, we will do an end-to-end training run once in a while where we train everything jointly. Then we cache the features at the multi-scale feature level. And then we fine-tune off of that for a while and then end-to-end train once again and so on. So here's the kinds of predictions that we were obtaining, I would say, several years ago now.
[0:55:21] From one of these Hydra nets. So again, we are processing individual images. There we go. We are processing just individual image and we're making a large number of predictions about these images. So for example, here you can see predictions of the stop signs, the stop lines, the lines, the edges, the cars, the traffic lights, the curbs here, whether or not the car is parked, all of the static objects like trash cans, cones, and so on. And everything here is coming out of the net. Here in this case, out of the Hydra net.
[0:55:50] So that was all fine and great, but as we worked towards FSD, we quickly found that this is not enough. So where this first started to break was when we started to work on Smart Summon. Here I am showing some of the predictions of only the curb detection task, and I'm showing it now for every one of the cameras. So we'd like to wind our way around the parking lot to find the person who is summoning the car. Now the problem is that you can't just directly drive on image space predictions. You actually need to cast them out and form some kind of a vector space around you.
[0:56:18] So we attempted to do this using C++ and developed what we call the occupancy tracker at the time. So here we see that the curb detections from the images are being stitched up across camera scenes, camera boundaries, and over time. Now, there were two major problems, I would say, with the setup. Number one, we very quickly discovered that tuning the occupancy tracker and all of its hyperparameters was extremely complicated. You don't want to do this explicitly by hand in C++. You want this to be inside a neural network and train that end to end.
[0:56:49] Number two, we very quickly discovered that the image space is not the correct output space. You don't want to make predictions in image space. You really want to make it directly in the vector space. So here's a way of illustrating the issue. So here I'm showing on the first row the predictions of our curves and our lines in red and blue. And they look great in the image. But once you cast them out into the vector space, things start to look really terrible. And we are not going to be able to drive on this. So you see how the predictions are quite bad?
[0:57:18] And the reason for this fundamentally is because you need to have an extremely accurate depth per pixel in order to actually do this projection. And so you can imagine just how high of a bar it is to predict that depth so accurately in these tiny, in every single pixel of the image. And also, if there's any occluded area where you'd like to make predictions, you will not be able to because it's not an image space concept in that case.
[0:57:46] The other problems with this, by the way, is also for object detection. If you are only making predictions per camera, then sometimes you will encounter cases like this where a single car actually spans five of the eight cameras. And so if you are making individual predictions, then no single camera sees all of the car. And so obviously, you're not going to be able to do a very good job of predicting that whole car. And it's going to be incredibly difficult to fuse these measurements.
[0:58:10] So we have this intuition that what we'd like to do instead is we'd like to take all of the images and simultaneously feed them into a single neural net and directly output in vector space. Now, this is very easily said, much more difficult to actually achieve. But roughly, we want to lay out a neural net in this way where we process every single image with a backbone. And then we want to somehow fuse them. And we want to re-represent the features from image space features to directly some kind of a vector space features and then go into the decoding of the head.
[0:58:40] So there are two problems with this. Problem number one, how do you actually create the neural network components that do this transformation? And you have to make it differentiable so that end-to-end training is possible. And number two, if you want vector space predictions from your neural net, you need vector space data sets. So just labeling images and so on is not going to get you there. You need vector space labels. We're going to talk a lot more about problem number two later in the talk.
[0:59:09] For now, I want to focus on the neural network architecture. So I'm going to deep dive into problem number one. So here's the rough problem, right? We're trying to have this bird's eye view prediction instead of image space predictions. So for example, let's focus on the single pixel in the output space in yellow. And this pixel is trying to decide, am I part of a curb or not, as an example? And now, where should the support for this kind of a prediction come from in the image space?
[0:59:33] Well, we know roughly how the cameras are positioned, and they're extrinsics and intrinsics, so we can roughly project this point into the camera images, and the evidence for whether or not this is a curb may come from somewhere here in the images. The problem is that this projection is really hard to actually get correct because it is a function of the road surface. The road surface could be sloping up or sloping down, or also there could be other data-dependent issues. For example, there could be occlusion due to a car. So if there's a car occluding this viewport,
[0:59:59] So, in order to solve this issue, we use a transformer to represent this space. And this transformer, it uses multi-headed self-attention and blocks off it. In this case, actually, we can get away with even a single block doing a lot of this work. And effectively, what this does is
[1:00:29] You initialize a raster of the size of the output space that you would like, and you tile it with positional encodings with signs and codes in the output space. And then these get encoded with an MLP into a set of query vectors. And then all of the images and their features also emit their own keys and values. And then the queries, keys, and values feed into the multi-headed self-attention. And so effectively, what's happening is that every single image piece is broadcasting in its key what is it a part of.
[1:00:55] So, hey, I'm part of a pillar in roughly this location, and I'm seeing this kind of stuff, and that's in the key. And then every query is something along the lines of, hey, I'm a pixel in the output space at this position, and I'm looking for features of this type. Then the keys and the queries interact multiplicatively, and then the values get pooled accordingly. And so this re-represents the space, and we found this to be very effective for this transformation. So if you do all of the engineering correctly, this, again, is very easily said, difficult to do, you do all of the engineering correctly,
[1:01:25] There's one more problem, actually, before. I'm not sure what's up with the slides. So one more thing. You have to be careful with some of the details here when you are trying to get this to work. So in particular, all of our cars are slightly cockeyed in a slightly different way. And so if you're doing this transformation from image space to the output space, you really need to know what your camera calibration is. And you need to feed that in somehow into the neural net. And so you could definitely just concatenate the camera calibrations of all of the
[1:01:54] We found that we can do much better by transforming all of the images into a synthetic virtual camera using a special rectification transform. So this is what that would look like. We insert a new layer right above the image, rectification layer. It's a function of camera calibration, and it translates all of the images into a virtual common camera. So if you were to average up a lot of repeater images, for example, which face to the back, without doing this, you would get kind of a blur.
[1:02:23] But after doing the rectification transformation, you see that the back mirror gets really crisp. So once you do this, this improves the performance quite a bit. So here are some of the results. So on the left, we are seeing what we had before. And on the right, we're now seeing significantly improved predictions coming directly out of the neural net. This is a multi-camera network predicting directly in vector space. And you can see that it's basically night and day. You can actually drive on this.
[1:02:51] And this took some time and some engineering and incredible work from the AI team to actually get this to work and deploy and make it efficient in the car. This also improved a lot of our object detection. So for example, here in this video, I'm showing single camera predictions in orange and multi-camera predictions in blue. And basically, you can't predict these cars if you are only seeing a tiny sliver of a car. So your detections are not going to be very good, and their positions are not going to be good. But a multi-camera network does not have an issue.
[1:03:20] Here's another video from a more nominal sort of situation. And we see that as these cars in this tight space cross camera boundaries, there's a lot of jank that enters into the predictions. And basically, the whole setup just doesn't make sense, especially for very large vehicles like this one. And we can see that the multi-camera networks struggle significantly less with these kinds of predictions. OK, so at this point, we have multi-camera networks, and they're giving predictions directly in vector space. But we are still operating at every single instant in time completely independently.
[1:03:49] So very quickly, we discovered that there's a large number of predictions we want to make that actually require the video context. And we need to somehow figure out how to feed this into the net. So in particular, is this car parked or not? Is it moving? How fast is it moving? Is it still there, even though it's temporarily occluded? Or for example, if I'm trying to predict the road geometry ahead, it's very helpful to know if the signs or the road markings that I saw 50 meters ago. So we tried to develop
[1:04:15] We tried to insert video modules into our neural network architecture. And this is kind of one of the solutions that we've converged on. So we have the multi-scale features, as we had them from before. And what we are going to now insert is a feature queue module that is going to cache some of these features over time, and then a video module that is going to fuse this information temporally. And then we're going to continue into the heads that do the decoding. Now, I'm going to go into both of these blocks one by one.
[1:04:40] Also, in addition, notice here that we are also feeding in the kinematics. This is basically the velocity and the acceleration that's telling us about how the car is moving. So not only are we going to keep track of what we're seeing from all the cameras, but also how the car has traveled. So here's the feature queue and the rough layout of it. We are basically concatenating these features over time and the kinematics of how the car has moved and the positional encodings. And that's being concatenated, encoded, and stored in a feature queue.
[1:05:08] And that's going to be consumed by a video module. Now, there's a few details here, again, to get right. So in particular, with respect to the pop and push mechanisms and when do you push and especially when do you push, basically. So here's a cartoon diagram illustrating some of the challenges here. The ego car is coming from the bottom and coming up to this intersection here. And then traffic is going to start crossing in front of us. And it's going to temporarily start occluding some of the cars ahead. And then we're going to be stuck at this intersection for a while and just waiting our turn.
[1:05:37] This is something that happens all the time and is a cartoon representation of some of the challenges here. So number one, with respect to the feature queue and when we want to push into a queue, obviously we'd like to have some kind of a time-based queue where, for example, we enter features into the queue, say, every 27 milliseconds. And so if a car gets temporarily occluded, then the neural network now has the power to look and reference the memory in time.
[1:06:01] And learn the association that, hey, even though this thing looks occluded right now, there's a record of it in my previous features. And I can use this to still make a detection. So that's kind of like the more obvious one. But the one that we also discovered is necessary in our case is, for example, suppose you're trying to make predictions about the road surface and the road geometry ahead. And you're trying to predict that I'm in a turning lane. And the lane next to us is going straight.
[1:06:27] It's really necessary to know about the line markings and the signs. And sometimes they occur a long time ago. And so if you only have a time-based queue, you may forget the features while you're waiting at your red light. So in addition to a time-based queue, we also have a space-based queue. So we push every time the car travels a certain fixed distance. So some of these details actually can matter quite a bit. And so in this case, we have a time-based queue and a space-based queue to cache our features. And that continues into the video module.
[1:06:54] Now, for the video module, we looked at a number of possibilities of how to fuse this information temporally. So we looked at three-dimensional convolutions, transformers, axial transformers in an effort to try to make them more efficient, recurrent neural networks of a large number of flavors. But the one that we actually like quite a bit as well and I want to spend some time on is a spatial recurrent neural network video module. And so what we're doing here is
[1:07:17] Because of the structure of the problem, we're driving on two-dimensional surfaces. We can actually organize the hidden state into a two-dimensional lattice. And then as the car is driving around, we update only the parts that are near the car and where the car has visibility. So as the car is driving around, we are using the kinematics to integrate the position of the car and the hidden features grid. And we are only updating the RNN at the points where we have that are nearby us, sort of.
[1:07:46] Here's an example of what that looks like. Here, what I'm going to show you is the car driving around. And we're looking at the hidden state of this RNN. And these are different channels in the hidden state. So you can see that this is after optimization and training this neural net. You can see that some of the channels are keeping track of different aspects of the road, like for example, the centers of the road, the edges, the lines, the road surface, and so on. Here's another cool video of this.
[1:08:15] So this is looking at the mean of the first 10 channels in the hidden state for different traversals of different intersections.
[1:08:23] And all I want you to see basically is that there's cool activity as the recurrent neural network is keeping track of what's happening at any point in time. And you can imagine that we've now given the power to the neural network to actually selectively read and write to this memory. So for example, if there's a car right next to us and is occluding some parts of the road, then now the network has the ability to not write to those locations. But when the car goes away and we have a really good view, then the recurrent neural net can say, OK, we have very clear visibility. We definitely want to write information about what's in that part of space.
[1:08:52] Here's a few predictions that show what this looks like. So here we are making predictions about the road boundaries in red, intersection areas in blue, road centers, and so on. So we're only showing a few of the predictions here just to keep the visualization clean. And yeah, this is done by the spatial RNN. And this is only showing a single clip, a single traversal, but you can imagine.
[1:09:17] There could be multiple trips through here. And basically, a number of cars, a number of clips could be collaborating to build this map, basically and effectively an HD map, except it's not in the space of explicit items. It's in the space of features of a recurring neural network, which is kind of cool. I haven't seen that before. The video networks also improved our object detection quite a bit. So in this example, I want to show you a case where there are two cars over there.
[1:09:43] And one car is going to drive by and occlude them briefly. So look at what's happening with the single frame and the video predictions as the cars pass in front of us. So that makes a lot of sense. So a quick playthrough through what's happening. When both of them are in view, the predictions are roughly equivalent. And you are seeing multiple orange boxes because they're coming from different cameras. When they are occluded, the single frame networks drop the detection, but the video module remembers it, and we can persist the cars.
[1:10:16] And then when they are only partially occluded, the single frame network is forced to make its best guess about what it's seeing and it's forced to make a prediction and it makes a really terrible prediction. But the video module knows that there's only a partial, that it has the information and knows that this is not a very easily visible part right now and doesn't actually take that into account. We also saw significant improvements in our ability to estimate depth and of course, especially velocity. So here I'm showing a clip from our remove the radar push.
[1:10:45] Where you are seeing the radar depth and velocity in green. And we were trying to match or even surpass, of course, the signal just from video networks alone. And what you're seeing here is in orange, we are seeing a single frame performance. And in blue, we are seeing, again, video modules. And so you see that the quality of depth is much higher. And for velocity, the orange signal, of course, you can't get velocity out of a single frame network. So we just differentiate depth to get that. But the video module actually is basically right on top.
[1:11:14] And so we found that this worked extremely well for us. So here's putting everything together. This is what our architectural roughly looks like today. So we have raw images feeding on the bottom. They go through a rectification layer to correct for camera calibration and put everything into a common virtual camera. We pass them through regnets, residual networks, to process them into a number of features at different scales. We fuse the multi-scale information with BIFBN.
[1:11:42] This goes through a transformer module to re-represent it into the vector space and the output space. This feeds into a feature queue in time or space that gets processed by a video module, like the spatial RNN, and then continues into the branching structure of the HydroNet with trunks and heads for all the different tasks. And so that's the architecture roughly what it looks like today. And on the right, you are seeing some of its predictions sort of visualized both in a top-down vector space and also in images. So definitely,
[1:12:09] This architecture has definitely complexified from just a very simple image-based single network about three or four years ago and continues to evolve. It's definitely quite impressive now. There's still opportunities for improvements that the team is actively working on. For example, you'll notice that our fusion of time and space is fairly late in neural network terms. So maybe we can actually do earlier fusion of space or time and do, for example, cost volumes or optical flow-like networks on the bottom.
[1:12:33] Or for example, our outputs are dense rasters. And it's actually pretty expensive to post-process some of these dense rasters in the car. And of course, we are under very strict latency requirements. So this is not ideal. We actually are looking into all kinds of ways of predicting just the sparse structure of the road, maybe like point by point or in some other fashion that doesn't require expensive post-processing. But this basically is how you achieve a very nice vector space. And now I believe Ashok is going to talk about how we can run plane control on top of it.
[1:13:11] Thank you, Andre. Hi, everyone. My name is Ashok. I lead the planning and controls, auto labeling, and simulation teams. Like Andre mentioned, the vision networks take dense video data and then compress it down into a 3D vector space. The role of the planner now is to consume this vector space and get the car to the destination while maximizing the safety, comfort, and efficiency of the car. Even back in 2019, our planner was a pretty capable driver. It was able to stay in the lanes, make lane changes as necessary, and take exits of the highway.
[1:13:42] But CDC driving is much more complicated. Rarely there are structured lane lines. Vehicles do much more free-form driving than the car has to respond to all of curtains and crossing vehicles and pedestrians doing funny things. What is the key problem in planning? Number one, the action space is very non-convex. And number two, it is high dimensional. What I mean by non-convex is
[1:14:13] There can be multiple possible solutions that can be independently good. But getting a globally consistent solution is pretty tricky. So there can be pockets of local minima that the planning can get stuck into. And secondly, the high dimensionality comes because the car needs to plan for the next 10 to 15 seconds. It needs to produce the position, velocities, and acceleration over this entire window. This is a lot of parameters to produce at runtime.
[1:14:38] Discrete search methods are really great at solving non-convex problems because they are discrete. They don't get stuck in local minima. Whereas continuous function optimization can easily get stuck in local minima and produce poor solutions that are not great. On the other hand, for high-dimensional problems, discrete search sucks because it is discrete. It does not use any graded information. So it literally has to go and explore each point to know how good it is. Whereas continuous optimization uses gradient-based methods to very quickly go to a good solution.
[1:15:10] Our solution to this entire problem is to break it down hierarchically. First, use a code search method to crunch down the non-convexity and come up with a convex corridor, and then use continuous optimization techniques to make the final smooth trajectory. Let's see an example of how the search operates. So here, we're trying to do a lane change. In this case, the car needs to do two back-to-back lane changes to make the left turn up ahead. For this, the car
[1:15:41] Searches over different maneuvers. So the first one it searches is a lane change that's close by. But the car brakes pretty harshly, so it's pretty uncomfortable. The next maneuver it tries does the lane change a bit late. So it speeds up, goes in front of the other cars, and finally does the lane change. But now it risks missing the left turn. We do thousands of such searches in a very short time span. Because these are all physics-based models, these features are very easy to simulate.
[1:16:15] And in the end, we have a set of candidates. And we finally choose one based on the optimality conditions of safety, comfort, and easily making the turn. So now the car has chosen this path. And you can see that as the car executes this trajectory, it pretty much matches what we had planned. The cyan plot on the right side here, that one is the actual velocity of the car. And the white line underneath it was the plan. So we are able to plan for 10 seconds here and able to match that when you see in hindsight. So this is a well-made plan.
[1:16:48] When driving alongside other agents, it's important to not just plan for ourselves. But instead, we have to plan for everyone jointly and optimize for the overall scene's traffic flow. In order to do this, what we do is we literally run the autopilot planner on every single relevant object in the scene. Here's an example of why that's necessary. This is an auto corridor. I'll let you watch the video for a second. Yeah, that was autopilot driving in an auto corridor going around parked cars, cones, and poles. Here, there's a 3D view of the same thing. The oncoming car arrives now.
[1:17:34] And Autopilot slows down a little bit, but then realizes that we cannot yield to them because we don't have any space to our side. But the other car can yield to us instead. So instead of just blindly braking here, Autopilot reasons about that car has low enough velocity that they can pull over and should yield to us because we cannot yield to them. And assertively makes progress. A second oncoming car arrives now. This vehicle has higher velocity. And like I said earlier, we literally run the Autopilot planner for the other object. So in this case, we run the planner for them.
[1:18:05] That object's plan now goes around their side's parked cars. And then after they pass the parked cars, goes back to the right side of the road for them. Since we don't know what's in the mind of the driver, we actually have multiple possible futures for this car. Here, one future is shown in red. The other one is shown in green. The green one is a plan that yields to us. But since this object's velocity and acceleration are pretty high, we don't think that this person is going to yield to us. And they are actually going to go around these parked cars. So Autopilot decides that, OK, I have space here.
[1:18:35] This person is definitely going to come, so I'm going to pull over. So as Autopilot is pulling over, we notice that that car has chosen to yield to us based on their yard rate and their acceleration. And Autopilot immediately changes its mind and continues to make progress. This is why we need to plan for everyone, because otherwise we wouldn't know that this person is going to go around the other parked cars and come back to their site. If we didn't do this, Autopilot would be too timid and would not be a practical self-driving car. So now we saw how the search and planning for other people set up a convex valley.
[1:19:07] Finally, we do a continuous optimization to produce the final trajectory that the planner needs to take. Here, the gray thing is the convex corridor, and we initialize the spline in heading and acceleration, parametrized over the arc length of the plan. And you can see that the optimization continuously makes fine-grained changes to reduce all of its costs. Some of the costs, for example, are distance from obstacles, traversal time, and comfort. For comfort, you can see that the lateral acceleration plots on the right.
[1:19:36] I have nice trapezoidal shapes. Here on the right side, the green plot. That's a nice trapezoidal shape. And if you record a human trajectory, this is pretty much how it would look like. The lateral jerk is also minimized. So in summary, we do a search for both us and everyone else in the scene. We set up a convex corridor and then optimize for a smooth path. Together, this can do some really neat things like shown above. But driving looks a bit different in other places, like where I grew up from.
[1:20:06] It's very much more unstructured. Cars and pedestrians cutting each other, arch breaking, honking. It's a crazy world. We can try to scale up these methods, but it's going to be really difficult to efficiently solve this at runtime. What we instead want to do is use learning-based methods to efficiently solve them. And I want to show why this is true. So we're going to go from this complicated problem to a much simpler toy parking problem, but still illustrates the core of the issue.
[1:20:35] Here, this is a parking lot. The ego car is in blue, and it needs to park in the green parking spot here. So it needs to go around the curbs, the parked cars, and the cones shown in orange here. This is a simple baseline. It's an A-star standard algorithm that uses a lattice-based search. And the heuristic here is the distance, the Euclidean distance to the goal. So you can see that it directly shoots towards the goal, but very quickly gets trapped in a local minima, and it backtracks from there, and then searches a different path to try to go around this parked car.
[1:21:06] Eventually, it makes progress and gets to the goal, but ends up using 400,000 nodes for making this. Obviously, this is a terrible heuristic. We want to do better than this, so if you added a navigation route to it and has the car to follow the navigation route while being close to the goal, this is what happens. The navigation route helps immediately, but still, when it encounters cones or other obstacles, it basically does the same thing as before, backtracks and then searches an old new path,
[1:21:38] The poor search has no idea that these obstacles exist. It literally has to go there, check if it's in collision, and if it's in collision, back up. The navigation heuristic helped, but still this took 22,000 nodes. We can design more and more of these heuristics to help the search go faster and faster, but it's really tedious and hard to design a globally optimal heuristic. Even if you had a distance function from the cones that guided the search, this would only be effective for the single cone.
[1:22:08] But what we need is a global value function. So instead, what we want to use is neural networks to give this heuristic for us. The vision networks produces vector space. And we have cars moving around in it. This basically looks like an Atari game. And it's a multiplayer version. So we can use techniques such as mu0, alpha0, et cetera, that was used to solve Go and other Atari games to solve the same problem. So we are working on neural networks that can produce state and action distributions that can then be plugged into multicolored research with various cost functions.
[1:22:38] Some of the cost functions can be explicit cost functions, like collisions, comfort, traversal time, et cetera. But they can also be interventions from the actual manual driving events. We trained such a network for the simple parking problem. So here, again, the same problem. Let's see how MCTS researched us. So here, you notice that the planner is basically able to, in one shot, make progress towards the goal. To note is that this is not even using a navigation heuristic. Just given the scene, the planner is able to go directly towards the goal.
[1:23:14] All the other options you're seeing are possible options. It's not choosing any of them, just choosing the option that directly takes it towards the goal. The reason is that the neural network is able to absorb the global context of the scene and then produce a value function that effectively guides it towards the global minima as opposed to getting stuck in any local minima. So this only takes 288 nodes and several orders of magnitude less than what was done in the A star with the Euclidean distance heuristic. So this is what a final architecture is going to look like.
[1:23:43] The vision system is going to crash down the dense video data into a vector space. It's going to be consumed by both an explicit planner and a neural network planner. In addition to this, the network planner can also consume intermediate features of the network. Together, this produces a trajectory distribution. And it can be optimized end-to-end, both with explicit cost functions and human intervention and other limitation data. This then goes into explicit planning function that does whatever is easy for that and produces the final steering and acceleration commands for the car.
[1:24:15] With that, we need to now explain how we train these networks. And for training these networks, we need large data sets. I invite Andre to speak briefly about manual labeling. Yeah, so the story of data sets is critical, of course. So far, we've talked only about neural networks. But neural networks only establish an upper bound on your performance. Many of these neural networks, they have hundreds of millions of parameters. And these hundreds of millions of parameters, they have to be set
[1:24:49] If you have a bad setting of parameters, it's not going to work. So neural networks are just an upper bound. You also need massive data sets to actually train the correct algorithms inside them. Now, in particular, I mentioned we want data sets directly in the vector space. And so really the question becomes, how can you accumulate, because our networks have hundreds of millions of parameters, how do you accumulate millions and millions of vector space examples that are clean and diverse to actually train these neural networks effectively?
[1:25:16] So there's a story of datasets and how they've evolved on the side of all of the models and developments that we've achieved. Now, in particular, when I joined roughly four years ago, we were working with a third party to obtain a lot of our datasets. Now, unfortunately, we found very quickly that working with a third party to get datasets for something this critical was just not going to cut it. The latency of working with a third party was extremely high. And honestly, the quality was not amazing.
[1:25:41] And so in the spirit of full vertical integration at Tesla, we brought all of the labeling in-house. And so over time, we've grown more than 1,000 person data labeling org that is full of professional labelers who are working very closely with the engineers. So actually, they're here in the US and co-located with the engineers here in Bay Area as well. And so we work very closely with them. And we also build all of the infrastructure for them from scratch ourselves. So we have a team we are going to meet later today.
[1:26:09] That develops and maintains all of this infrastructure for data labeling. And so here, for example, I'm showing some of the screenshots of some of the latency throughput and quality statistics that we maintain about all of the labeling workflows and the individual people involved and all the tasks and how the numbers of labels are growing over time. So we found this to be quite critical and we're very proud of this. Now, in the beginning, roughly three or four years ago, most of our labeling was in image space. And so you can...
[1:26:37] Imagine that this is taking quite some time to annotate an image like this, and this is what it looked like, where we are sort of drawing polygons and polylines on top of these single individual images. As I mentioned, we need millions of vector space labels, so this is not going to cut it. So very quickly, we graduated to three-dimensional or four-dimensional labeling, where we are directly labeling in vector space, not in individual images. So here what I'm showing is a clip, and you are seeing a very small reconstruction.
[1:27:07] You're about to see a lot more reconstructions soon. But it's a very small reconstruction of the ground plane on which the car drove and a little bit of the point cloud here that was reconstructed. And what you're seeing here is that the labeler is changing the labels directly in vector space. And then we are re-projecting those changes into camera images. So we're labeling directly in vector space. And this gave us a massive increase in throughput for a lot of our labels, because you label once in 3D, and then you get to re-project. But even this.
[1:27:35] We realized was actually not going to cut it. So because people and computers have different pros and cons. So people are extremely good at things like semantics, but computers are very good at geometry, reconstruction, triangulation, tracking. And so really for us, it's much more becoming a story of how do humans and computers collaborate to actually create these vector space data sets. And so we're going to now talk about auto labeling, which is some of the infrastructure we've developed for labeling these clips at scale. Hi again.
[1:28:11] So even though we have lots of human labelers, the amount of training that are needed for training the network significantly outnumbers them. So we try to invest in a massive auto labeling pipeline. Here's an example of how we label a single clip. A clip is an entity that has dense sensor data, like videos, IMU data, GPS, odometry, et cetera. This can be 45 seconds to a minute long. These can be uploaded by our own engineering cars or from customer cars. We collect these clips.
[1:28:37] And then send them to our servers, where we run a lot of neural networks offline to produce intermediate results, like segmentation masks, depth, point matching, et cetera. This then goes through a lot of robotics and AI algorithms to produce a final set of labels that can be used to train the networks. One of the first tasks we want to label is the road surface. Typically, we can use splines or meshes to represent the road surface, but those are, because of the topology restrictions, are not differentiable and not amenable to producing this.
[1:29:07] So what we do instead is in the style of neural radiance fields work from last year, which is quite popular. So we use an implicit representation to represent the road surface. Here, we are querying xy points on the ground and asking for the network to predict the height of the ground surface along with various semantics such as curves, lane boundaries, road surface, drivable space, et cetera. So given a single xy, we get a z. Together, these make a 3D point, and they can be reprojected into all the camera views.
[1:29:36] So we make millions of such queries and get lots of points. These points are re-projected into all the camera views. We are showing on the top right here one such camera image with all these points re-projected. Now we can compare this point, a re-projected point, with the image space prediction of the segmentations. And jointly optimizing this over all the camera views across space and time produces an excellent reconstruction. Here's an example of how that looks like.
[1:30:05] So here, this is an optimized road surface that reproduces into the eight cameras that the car has and across all of time. And you can see how it's consistent across both space and time. So a single car driving through some location can sweep out some patch around the trajectory using this technique. But we don't have to stop there. So here, we collected different clips from the same location, from different cars maybe, and each of them sweeps out some part of their road. Cool thing is we can bring them all together
[1:30:39] This is an effective way to label the road surface, not just where the car drove, but also in other locations that it hasn't driven yet. Again, the point of this is not to just build HD maps or anything like that.
[1:31:04] It's only to label the clips through these intersections. So we don't have to maintain them forever, as long as the labels are consistent with the videos that they were collected at. Optionally, then humans can come on top of this and clean up any noise or add additional metadata to make it even richer. We don't have to stop at just the road surface. We can also arbitrarily reconstruct 3D static obstacles. Here, this is a reconstructed 3D point cloud from our cameras. The main innovation here is the density of the point cloud.
[1:31:35] Typically, these points require texture to form associations from one frame to the next frame. But here, we are able to produce these points even on textureless surfaces like the road surface or walls. And this is really useful to annotate arbitrary obstacles that we can see in the world. One more cool advantage of doing all of this on the servers offline is that we have the benefit of hindsight. This is a super useful hack because, say, in the car, the network needs to produce the velocity.
[1:32:07] It just has to use the historical information and guess what the velocity is. But here, we can look at both the history, but also the future, and basically cheat and get the correct answer of the kinematics, like velocity, acceleration, et cetera. One more advantage is that we can have different tracks, but we can stitch them together even through occlusions. Because we know the future, we have future tracks, we can match them and then associate them. So here you can see the pedestrians on the other side of the road are persisted even through multiple occlusions by these cars.
[1:32:37] This is really important for the planner, because the planner needs to know if it saw someone, it still needs to account for them, even when they're occluded. So this is a massive advantage. Combining everything together, we can produce these amazing data sets that annotate all of the road texture, all the static objects, and all of the moving objects, even through occlusions, producing excellent kinematic labels. You can see how the cars turn smoothly, produce really smooth labels, all the pedestrians are consistently tracked.
[1:33:08] The parked cars have zero velocity, so we can also know that they are parked. So this is huge for us. This is one more example of the same thing. You can see how everything is consistent. We want to produce a million such labeled clips and train our multicam video networks with such large data set and really crush this problem. We want to get the same view that's consistent that we're seeing here in the car.
[1:33:36] We started our first exploration of this with the Remove the Radar project. We removed it in a very short time span, I think, within three months. In the early days of the network, we noticed, for example, in low-visibility conditions, the network can suffer, understandably, because, obviously, this truck just dumped a bunch of snow on us, and it's really hard to see. But we should still remember that this car was in front of us. But our networks early on did not do this because of the lack of data in such conditions. So what we did, we added the fleet to produce lots of similar clips.
[1:34:07] And the feed responded it did. So it produces lots of video clips where shit's falling out of all other vehicles. And we send this through our auto-labeling pipeline that was able to label 10k clips within a week. This would have taken several months with humans labeling every single clip here. So we did this for 200 different conditions. And we were able to very quickly create large data sets.
[1:34:38] And that's how we were able to remove this. So once we train the networks with this data, you can see that it's totally working and keeps the memory that this object was there and provides this. So finally, we wanted to actually get a Cybertruck into our data set for the remote radar. Can you all guess where we got this clip from? I'll give you a moment. Someone said it. Yes. Yes, it's rendered. It's our simulation. It was hard for me to tell initially. And if I may say so myself, it looks pretty.
[1:35:11] It looks really pretty. So yeah. In addition to auto labeling, we also invest heavily in using simulation for labeling our data. So this is the same scene as seen before, but from a different camera angle. So a few things that I wanted to point out. For example, the ground surface. It's not plain asphalt. There are lots of cars and cracks and tar seams. There's some patchwork done on top of it.
[1:35:40] Vehicles move realistically. The truck is articulated, even goes over the curb, and makes a wide turn. The other cars behave smartly. They avoid collisions, go around cars, and also brake and accelerate smoothly. The car here with the logo on the top is Autopilot. Actually, Autopilot is driving the car, and it's making an unproductive left turn. And since this is a simulation, it starts from the vector space. So it has perfect labels. Here we show a few of the labels that we produce.
[1:36:10] These are vehicle cuboids with kinematics, depth, surface normals, segmentation. But Andre can name a new task that he wants next week and we can very quickly produce this because we already have the vector space and we can write the code to produce these labels very, very quickly. So when does simulation help? It helps, number one, when the data is difficult to source. As large as our fleet is, it can still be hard to get some crazy scenes like this couple and their dog running on the highway while there are other high-speed cars around.
[1:36:41] This is a pretty rare scene, I'd say. But still, it can happen. And Autopilot still needs to handle it when it happens, when data is difficult to label. There are hundreds of pedestrians crossing the road. This could be Manhattan, downtown, people crossing the road. It's going to take several hours for humans to label this clip. And even for automatic labeling algorithms, this is really hard to get the association right, and it can produce bad velocities. But in simulation, this is trivial, because you already have the objects. You just have to spit out the cuboids and the velocities.
[1:37:10] And also, finally, when we introduce closed-loop behavior where the car needs to be in a determined situation or the data depends on the actions, this is pretty much the only way to get it reliably. All this is great. What's needed to make this happen? Number one, accurate sensor simulation. Again, the point of the simulation is not to just produce pretty pictures. It needs to produce what the camera in the car would see and other sensors would see. So here, we are stepping through different exposure settings of the real camera on the left side.
[1:37:43] And the simulation on the right side. We're able to pretty much match what the real cameras do. In order to do this, we had to model a lot of the properties of the camera in our sensor simulation, starting from sensor noise, motion blur, optical distortions, even headlight transmissions, even like diffraction patterns of the windshield, et cetera. We don't use this just for the AutoPile software. We also use it to make hardware decisions such as lens design.
[1:38:11] Camera design, sensor placement, even headlight transmission properties. Second, we need to render the visuals in a realistic manner. You cannot have what in the game industry called jaggies. These are aliasing artifacts that are a dead giveaway that this is simulation. We don't want them. So we go through a lot of paints to produce a nice spatiotemporal anti-aliasing. We also are working on neural rendering techniques to make this even more realistic.
[1:38:47] Yeah. In addition, we also use ray tracing to produce ray tracing lighting and global illumination. OK, that's the last of the cop cars, I think. We obviously cannot have really just four or five cars, because the network will easily overfit because it knows the sizes. So we need to have realistic assets, like the moose on the road here. We have thousands of assets in our library. And they can wear different shirts and actually can move realistically. So this is really cool.
[1:39:13] We also have a lot of different locations mapped and created to create these environments. We have actually 2,000 miles of road built. And this is almost the length of the roadway from the east coast to the west coast of the United States, which I think is pretty cool. In addition, we have built efficient tooling to build several miles more on a single day for a single artist. But this is just the tip of the iceberg. Actually, most of the data that we use to train is created
[1:39:42] Procedurally using algorithms as opposed to artists making these simulation scenarios. So these are all procedurally created roads with lots of parameters such as curvature, various varying trees, cones, poles, cars with different velocities. And the interaction produce an endless stream of data for the network. But a lot of this data can be boring because the network may not already get it correct. So what we do is we use also ML-based techniques to basically put up the network to see where it's failing at and create more data around the failure points of the network.
[1:40:11] So this is in closed loop trying to make the network performance be better. We don't want to stop there. So actually, we want to recreate any failures that happens to the autopilot in simulation so that we can hold autopilot to the same bar from then on. So here on the left side, you're seeing a real clip that was collected from a car. It then goes through our auto labeling pipeline to produce a 3D reconstruction of the scene along with all the moving objects. With this combined with the original visual information, we recreate the same scene.
[1:40:44] Synthetically and create a simulation scenario entirely out of it. And then when we replay autopilot on it, autopilot can do entirely new things and we can form new worlds, new outcomes from the original failure. This is amazing because we really don't want autopilot to fail and when it fails, we want to capture it and keep it to that bar. Not just that, we can actually take the same approach that we said earlier and take it one step further. We can use neural rendering techniques.
[1:41:14] To make it look even more realistic, we take the original video clip, recreate a synthetic simulation from it, and then apply neural rendering techniques on top of it. And it produces this, which looks amazing in my opinion, because this one is very realistic and looks almost like it was captured by the actual cameras. This is a result from last night, because it was cool and we wanted to present it. But yeah, I'm very excited for what Sim can achieve.
[1:41:39] But this is not all bullshit, because networks trained in the car already use simulation data. We used 300 million images with almost half a billion labels. And we want to crush down all the tasks that are going to come up for the next several months. With that, I invite Milan to explain how we scale these operations and really build a label factory and spit out millions of labels. MILAN SINGH- Thanks, Ashok. Hey, everyone. I'm Milan. I'm responsible for the integration of our neural networks in the car.
[1:42:16] And for most of our neural network training and evaluation infrastructure. And so tonight, I'd just like to start by giving you some perspective into the amount of compute that's needed to power this type of data generation factory. And so in the specific context of the push we went through as a team here a few months ago to get rid of the dependency on the radar sensor for the pilot, we generated over 10 billion labels across 2 and 1 half million clips.
[1:42:41] And so to do that, we had to scale our huge offline neural networks and our simulation engine across thousands of GPUs and just a little bit shy of 20,000 CPU cores. On top of that, we also included over 2,000 actual autopilot full self-driving computers in the loop with our simulation engine. And that's our smallest compute cluster. So I'd like to give you some idea of what it takes to take our neural networks and
[1:43:12] Move them in the car. And so the two main constraints that we're working under here are mostly latency and frame rate, which are very important for safety, but also to get proper estimates of acceleration and velocity of our surroundings. And so the meat of the problem really is around
[1:43:31] The AI compiler that we write and extend here within the group that essentially maps the compute operations for my Power Torch model to a set of dedicated and accelerated pieces of hardware. And we do that while figuring out a schedule that's optimized for throughput while working under severe SRAM constraints.
[1:43:53] And so by the way, we're not doing that just on one engine, but across two engines on the autopilot computer. And the way we use those engines here at Tesla is such that at any given time, only one of them will actually output control commands to the vehicle, while the other one is used as an extension of compute. But those roles are interchangeable, both on the hardware and software level. So how do we iterate quickly together as a group through these AI development cycles?
[1:44:19] Well, first, we have been scaling our capacity to evaluate our software and neural network dramatically over the past few years. And today, we are running over a million evaluations per week on any code change that the team is producing. And those evaluations run on over 3,000 actual full-cell driving computers that are hooked up together in a dedicated cluster.
[1:44:42] And so on top of this, we've been developing really cool debugging tools. And so here is a video of one of our tools, which is helping developers iterate through the development of neural networks and comparing live the outputs from different revisions of a same neural network model as reiterating life through video clips. And so last but not least, we've been scaling our neural network training compute dramatically over the past few years. And today, we're barely shy of
[1:45:11] 10,000 GPUs, which just to give you some sense, in terms of number of GPU, is more than the top five publicly known supercomputers in the world. But that's not enough, and so I'd like to invite Ganesh to talk about the next steps. Thank you, Milan. My name is Ganesh, and I lead Project Dojo. It's an honor to present this project on behalf of the multidisciplinary Tesla team that is working on this project. As you saw from Milan,
[1:45:55] There's an insatiable demand for speed as well as capacity for neural network training. And Elon prefetched this. And a few years back, he asked us to design a super fast training computer. And that's how we started Project Dojo. Our goal is to achieve best AI training performance and support all these larger, more complex models that Andre's team is dreaming of and be power efficient and cost effective at the same time.
[1:46:26] So we thought about how to build this, and we came up with a distributed compute architecture. After all, all the training computers out there are distributed computers in one form or the other. They have compute elements in the box out here connected with some kind of network. In this case, it's a two-dimensional network, but it could be any different network. CPU, GPU, accelerators, all of them have compute, little memory, and...
[1:46:56] But one thing which is common trend amongst this is it's easy to scale the compute. It's very difficult to scale up bandwidth and extremely difficult to reduce latencies. And you'll see how our design point catered to that, how our philosophy addressed these aspects of traditional limits. For Dojo, we envisioned a large compute plane.
[1:47:23] Filled with very robust compute elements backed with large pool of memory and interconnected with very high bandwidth and low latency fabric and in a 2D mesh format. And on to this, for extreme scale, big neural networks will be partitioned and mapped to extract different parallelism, model, graph, data parallelism. And then a neural compiler of ours will
[1:47:52] We wanted to exploit spatial and temporal locality such that it can reduce communication footprint to local zones and reduce global communication. And if we do that, our bandwidth utilization can keep scaling with the plane of compute that we desire out here. We wanted to attack this all the way top to bottom of the stack and remove any bottlenecks at any of these levels.
[1:48:22] And let's start this journey in an inside-out fashion, starting with the chip. As I described, chips have compute elements. Our smallest entity of scale is called a training node. And the choice of this node is very important to ensure seamless scaling. If you go too small, it will run fast, but the overheads of synchronization and software will dominate if you pick it too big.
[1:48:51] It will have complexities in implementation in the real hardware and ultimately run into memory bottleneck issues. Because we wanted to address, we wanted to address latency and bandwidth as our primary optimization point. Let's see how we went about doing this. What we did was we picked the farthest distance a signal could traverse in a very high clock cycle. In this case,
[1:49:19] 2 gigahertz plus. And we drew a box around it. This is the smallest latency that a signal can traverse, one cycle at a very high frequency. And then we filled up the box with wires to the brink. This is the highest bandwidth you can feed the box with. And then we added machine learning compute underneath, and then a large pool of SRAM. And last but not the least, a programmable core to control. And this.
[1:49:48] gave us our high performance training node. What this is, is a 64-bit superscalar CPU optimized around matrix multiply units and vector SIMD. It supports floating point 32, bfloat16, and a new format, CFP8, configurable FP8. And it is backed by one and a quarter megabyte of fast ECC-protected SRAM.
[1:50:15] And the low latency, high bandwidth fabric that we designed. This might be our smallest entity of scale, but it packs a big punch. More than one teraflop of compute in our smallest entity of scale. So let's look at the architecture of this. The computer architects out here may recognize this. This is a pretty capable architecture as soon as you see this. It is a superscalar in-order CPU.
[1:50:44] We call it in order, although the vector and the scalar pipes can go out of order. But for the purists out there, we still call it in order.
[1:50:59] And it also has four-way multi-threading. This increases utilization because we can do compute and data transfers simultaneously. And our custom ISA, which is the instruction set architecture, is fully optimized for machine learning workloads. It has features like transpose, gather, link traversals, broadcast, just to name a few. And even in the physical realm, we made it extremely modular, such that we could start abutting these
[1:51:30] training nodes in any direction and start forming the compute plane that we envisioned. When we click together 354 of these training nodes, we get our compute array. It's capable of delivering 362 teraflops of machine learning compute. And of course, the high bandwidth fabric that interconnects these. And around this compute array, we surrounded it with
[1:52:00] High-speed, low-power services, 576 of them, to enable us to have extreme IO bandwidth coming out of this chip. Just to give you a comparison point, this is more than two times the bandwidth coming out of the state-of-the-art networking switch chips which are out there today. And network switch chips are supposed to be the gold standards for IO bandwidth.
[1:52:29] If we put all of it together, we get training optimized chip, our D1 chip. This chip is manufactured in seven nanometer technology. It packs 50 billion transistors in a miserly 645 millimeter square. One thing you'll notice, 100% of the area out here is going towards machine learning training and bandwidth.
[1:52:54] There is no dark silicon. There is no legacy support. This is a pure machine learning machine. And this is the D1 chip in a flip chip BGA package. This was entirely designed by Tesla team internally, all the way from the architecture to GDS out and package. This chip is like a
[1:53:28] GPU level compute with a CPU level flexibility and twice the network chip level IO bandwidth. If I were to plot the IO bandwidth on the vertical scale versus teraflops of compute that is available in the state of the art machine learning chips out there, including some of the startups, you can easily see why our design point excels beyond par.
[1:53:56] Now that we had this fundamental physical building block, how to design the system around it? Let's see. Since D1 chips can seamlessly connect without any glue to each other, we just started putting them together. We just put 500,000 training nodes together to form our compute plane. This is 1,500 D1 chips.
[1:54:33] seamlessly connected to each other. And then we add Dojo interface processors on each end. This is the host bridge to typical hosts in the data centers. It's connected with PCIe Gen 4 on one side with a high bandwidth fabric to our compute plane. The interface processors provide not only the host bridge, but high bandwidth.
[1:55:01] DRAM shared memory for the compute plane. In addition, the interface processors can also allow us to have a higher Radix network connection. In order to achieve this compute plane, we had to come up with a new way of integrating these chips together. And this is what we call as a training tile. This is the unit of scale for our system. This is
[1:55:32] A groundbreaking integration of 25 known good D1 dies onto a fan out wafer process, tightly integrated such that it preserves the bandwidth between them. The maximum bandwidth is preserved there. And in addition, we generated a connector, a high bandwidth, high density connector that preserves the bandwidth coming out of this training tile. And this tile,
[1:56:01] It gives us nine petaflops of compute with a massive IO bandwidth coming out of it. This, perhaps, is the biggest organic MCM in the chip industry, multi-chip module. It was not easy to design this. There were no tools that existed. All the tools were croaking. Even our compute cluster couldn't handle it. Our engineers came up with different ways of solving this. They created new methods.
[1:56:33] To make this a reality. Now that we had our compute plane tile with high bandwidth IOs, we had to feed it with power. And here, we came up with a new way of feeding power vertically. We created a custom voltage regulator module that could be reflowed directly, directly onto this fan-out wafer.
[1:57:00] So what we did out here is we got chip, package, and we brought PCB-level technology of reflow onto this fan-out wafer technology. This is a lot of integration already out here, but we didn't stop here. We integrated the entire electrical, thermal, and mechanical pieces out here to form our training tile, fully integrated, interfacing.
[1:57:31] with a 52 volt DC input. It's unprecedented. This is an amazing piece of engineering. Our compute plane is completely orthogonal to power supply and cooling. That makes high bandwidth compute planes possible. What it is, is a nine petaflop training tile. This becomes our unit of scale for our system. And this,
[1:58:12] It's real. I can't believe I'm holding nine petaflops out here. And in fact, last week, we got our first functional training tile. And on a limited, limited cooling benchtop setup, we got some networks running. And I was told Andre doesn't believe that we could run networks till we could run one of his creations. Andre, this is MinGPT2 running on Dojo. Do you believe it?
[1:59:12] Next step, how to form a compute cluster out of it. By now, you must have realized our modularity story is pretty strong. We just put together some tiles. We just tiled together tiles. A two by three tile in a tray makes our training matrix, and two trays in a cabinet give 100 petaflops of compute. Did we stop here? No.
[1:59:47] We just integrated seamlessly. We broke the cabinet walls. We integrated these tiles seamlessly all the way through, preserving the bandwidth. There is no bandwidth divot out here. There is no bandwidth cliffs. All the tiles are seamlessly connected with the same bandwidth. And with this, we have an exapod. This is one exaflop of compute.
[2:00:20] It's more than a million training nodes that you saw. We paid meticulous attention to that training node, and there are one million nodes out here with uniform bandwidth. Not just the hardware, the software aspects are so important to ensure scaling. And not every job requires a huge cluster, so we plan for it right from the get-go. Our compute plane
[2:00:56] Can be subdivided, can be partitioned into units called Dojo processing unit. A DPU consists of one or more D1 chips. It also has our interface processor and one or more hosts. And this can be scaled up or down as per the needs of any algorithm, any network running on it. What does the user have to do?
[2:01:28] They have to change their scripts minimally. And this is because of our strong compiler suite. It takes care of fine-grained parallelism and mapping the neural networks very efficiently onto our compute plane. Our compiler uses multiple techniques to extract parallelism. It can transform the networks to achieve
[2:01:58] Not only fine-grained parallelism using data model graph parallelism techniques, it also can do optimizations to reduce memory footprints. One thing because of our high bandwidth nature of the fabric is enabled out here is model parallelism could not have been extended to the same level as what we can. It was limited to chip boundaries. Now we can
[2:02:27] Because of our high bandwidth, we can extend it to training tiles and beyond. Thus, large networks can be efficiently mapped here at low batch sizes and extract utilization and new levels of performance. In addition, our compiler is capable of handling high-level dynamic control flows like loops, if-then-else, et cetera. And our compiler engine is
[2:02:55] Just part of our entire software suite. The stack consists of an extension to PyTorch that ensures the same user level interfaces that ML scientists are used to. And our compiler generates code on the fly, such that it could be reused for subsequent execution. It has a LLVM backend that generates the binary for the hardware. And this ensures we can create optimized code
[2:03:25] For the hardware without relying on even single line of handwritten kernel. Our driver stack takes care of the multi-host, multi-partitioning that you saw a few slides back. And then we also have profilers and debuggers in our software stack. So with all this, we integrated
[2:03:53] In a vertical fashion, we broke the traditional barriers to scaling. And that's how we got modularity up and down the stack to add to new levels of performance. To sum it all, this is what it will be. It'll be a fastest AI training computer, 4x the performance at the same cost, 1.3x
[2:04:19] Better performance per watt, that is energy saving, and 5x smaller footprint. This will be Dojo Computer. And we are not done. We are assembling our first cabinets pretty soon, and we have a whole next generation plan already. We are thinking about 10x more with different aspects that we can do, all the way from silicon to the system again.
[2:04:56] We will have this journey again. We're recruiting heavily for all of these areas. Thank you very much. And next up, Elon will update us on what's beyond our vehicle fleet for AI. Unlike Dojo, obviously, that was not real. So Dojo is real. The Tesla bot will be real. But basically, if you think about...
[2:07:11] What we're doing right now with the cars, Tesla is arguably the world's biggest robotics company because our cars are like semi-sentient robots on wheels. And with the full self-driving computer, essentially the inference engine on the car, which will keep evolving obviously, and Dojo, and all the neural nets recognizing the world, understanding how to navigate through the world, it kind of makes sense to put that onto a humanoid form.
[2:07:41] And we're also quite good at sensors and batteries and actuators. So we think we'll probably have a prototype sometime next year that basically looks like this. And it's intended to be friendly, of course, and navigate through a world built for humans.
[2:08:11] Eliminate dangerous, repetitive, and boring tasks. We're setting it such that it is, at a mechanical level, at a physical level, you can run away from it. And most likely overpower it. So hopefully that doesn't ever happen, but you never know. So it'll be a light, five miles an hour.
[2:08:43] If you can run past in that, you'll be fine. So yes, it's around 5 foot 8, has sort of a screen where the head is for useful information. But it's otherwise basically got the autopilot system in it. So it's got cameras, got eight cameras, and yeah, full self-driving computer, and making use of all of the same tools that we use in the car.
[2:09:19] So, I mean, things I think that are really hard about having a useful humanoid robot is, can it navigate through the world without being explicitly trained? I mean, without explicit, like, line-by-line instructions. Can you talk to it and say, you know, please pick up that bolt and attach it to a car with that wrench, and it should be able to do that.
[2:09:49] It should be able to, you know, please go to the store and get me the following groceries. That kind of thing. So, yeah, I think we can do that. And, yeah, this I think will be quite profound because if you say, like, what is the economy? It is, at the foundation, it is labor. So what happens when there is, you know...
[2:10:19] No shortage of labor. That's why I think long term that there will need to be a universal basic income. But not right now because this robot doesn't work. We still need a minute. But I think essentially in the future, physical work will be a choice. If you want to do it, you can, but you won't need to do it. I think it obviously has profound implications for the economy because...
[2:10:50] Given that the economy at its foundational level is labor, I mean, capital equipment is just distilled labor, then is there any actual limit to the economy? Maybe not. So, yeah, join our team and help build this. All right, so I think we'll have everyone come back on the stage, and you guys can ask questions if you'd like. I'd just like to maybe try to get the camera angle from there, but not from the side.
[2:12:15] All right, cool. So we'll probably turn the lights back on. And... Yeah, great. So we're happy to answer any questions you have about anything on the software, hardware side, where things are going. And, yeah, fire away. Because the lights are like interrogation lights, we actually cannot see... Ah, there we go. Great. All right, cool.
[2:12:57] I can just, okay, there we go. First off, I mean, thanks to all the presenters. That was just super cool to see everything. I'm just curious at a high level, and this is kind of a question for really anyone who wants to take it. To what extent are you interested in publishing or open sourcing anything that you do for the future? Well, I mean, it is fundamentally extremely expensive to create the system.
[2:13:36] Somehow that has to be paid for. I'm not sure how to pay for it if it's fully open sourced. Yeah, unless people want to work for free. But I should say that if other car companies want to license it and use it in their cars, that would be cool. This is not intended to be just limited to Tesla cars. This is for the Dojo supercomputer. So did you solve the compiler problem of scaling to these many nodes?
[2:14:13] If it is solved, is it only applicable to Dojo? Because I'm doing research in deep learning accelerators and getting the correct scalability or the distribution, even in one chip, is extremely difficult from the research project's perspective. So I was just curious. Excuse me. Have we solved the problem? Not yet. Are we confident we will solve the problem?
[2:14:45] Yes. We have demonstrated networks on prototype hardware now. We have models, performance models showing the scaling. The difficulty is, as you said, how do we keep the localities? If we can do enough model parallel, enough data parallel to keep most of the things local, we just keep scaling. We have to fit the parameters in our working set in our SRAM that we have, and we flow through the pipe. There's plenty of opportunities. Sorry. As we get further scale for further processor nodes, have more local memory.
[2:15:17] Memory tradeoffs with bandwidth, we can do more things. But as we see it now, the applications that Tesla has, we see a clear path. And our modularity story means we can have different ratios, different aspects created out of it. I mean, this is something that we chose for our applications internally. Sure. The locality portion of it, given that training is such a soft-scaling application, even though you have all this compute and have a high bandwidth,
[2:15:54] High bandwidth interconnect. It could not give you that performance because you are doing computations on limited memory at different locations. So that's very curious to me when you said it's solved because I just jumped onto the opportunity and would love to know more given that how much you can open source. Yeah. Yeah, I guess the proof's in the pudding. So we should have Dojo operational next year.
[2:16:25] I think we'll obviously use it for video training. The primary application initially is we've got vast amounts of video, and how do we train vast amounts of video as efficiently as possible? And also shorten the amount of time. If you're trying to train to a task,
[2:16:51] Just in general, innovation is how many iterations and what is the average progress between each iteration. And so if you can reduce the time between iterations, the rate of improvement is much better. So if it takes sometimes a couple days for a model to train versus a couple hours, that's a big deal. But the acid test here and what I've told the
[2:17:18] Dojo team is, it's successful if the software team wants to turn off the GPU cluster. But if they want to keep the GPU cluster on, it's not successful. Hi. Brad over here. Love the presentation. Thank you for getting us out here. Love everything, especially the simulation part of the presentation. I was wondering, it looked very, very, very realistic.
[2:17:51] Are there any plans to maybe expand simulation to other parts of the company in any way? Hi, I'm Ian Glow. I manage the autopilot simulation team. So as we go down the path to full self-driving, we're going to have to simulate more and more of the vehicle. Currently, we're simulating...
[2:18:10] Vehicle dynamics, but we're going to need BMS. We're going to need the MCU. We're going to need every single part of the vehicle integrated. And that actually makes the autopilot simulator really useful for places outside of autopilot. So I want to expand, or we want to expand eventually to being a universal simulation platform. But I think before that, we're going to be spinning up a lot of Optimus support. And then a little bit further down the line, we have some rough ideas on potentially how to get the simulation infrastructure and some of the cool things we've built into the hands of people outside of the company.
[2:18:40] Optimus is the code name for the Tesla bot. Oops. Optimus O'Brien. Hi, this is Ali Jahanian. Thank you for the great presentation and putting all of these cool things together. Yeah, for a while I have been thinking that the car is already a robot. So why not a human robot? And I'm so happy that today...
[2:19:14] You mentioned that you are going to build such thing. Especially, I think that this can give opportunity for raise of putting multi-modality together. For instance, we know that in the example that you showed that there was a dog and with some passengers were running together. The language and symbolic processing can really help for visualizing that.
[2:19:44] So I was wondering if I could hear a little more about these type of putting modalities together, including language and vision. Because I have been working with, for instance, many GPT and Android put out there. And yeah, I didn't hear much about other modalities that's going into the car, or at least in the simulation. Is there any comment?
[2:20:14] Yes, you could tell us. Well, driving is fundamentally basically almost entirely vision neural nets. Basically, it's running on a biological vision neural net. And what we're doing here is a silicon camera neural net. So there is some amount of audio. You want to hear if there's like emergency vehicles
[2:20:44] I guess converse with the people in the car. If somebody's yelling something at the car, the car needs to understand what that is. So all the things that are necessary for it to be fully autonomous. Thank you.
[2:21:11] Hi. Thank you for all the great work that you've shown. My question is for the team, because the data that was shown seems to be predominantly from the United States that the FST computer is being trained on. But as it gets rolled out to different countries, which have their own road systems and challenges that come with it, how do you think that it's going to scale? I'm assuming ground up is not a very viable solution. So how does that transfer to different countries?
[2:21:41] Well, we actually do train on using data from probably like 50 different countries. But we have to pick, you know, as we're trying to advance full self-driving, we need to pick one country. And since we're located here, we pick the U.S. And then we get a lot of questions like, why not even Canada? I'm like, well, because the roads are a little different in Canada, different enough. And so when you're trying to solve a hard problem, you want to...
[2:22:10] Let's not add additional complexity right now. Let's just solve it for the US, and then we'll extrapolate to the rest of the world. But we do use video from all around the world. I think a lot of what we are building is very country agnostic. Fundamentally, all the computer vision components and so on don't care too much about country-specific features. Different countries have roads, and they have curbs, and they have cars, and everything we're building is fairly general for that.
[2:22:40] Yeah, and the prime directive is don't crash. Right, and that's true for every country. Yes, this is the prime directive. And even right now, the car is pretty good at not crashing. And so just basically, whatever it is, don't hit it. Even if it's a UFO that crash-landed on the highway, still don't hit it. You should not need to recognize it in order to not hit it. That's very important.
[2:23:19] And I want to ask that when you do the photometric process, multi-view geometry, how much of an error do you see? Is it like one millimeter, one centimeter? So I'm just, if it's not confidential. Sorry, the question, what is the resolution? What's the difference between the synthetic? What is the difference between the synthetically created geometry to the actual geometry? Yeah, it's usually within a couple centimeters, three or four centimeters.
[2:23:52] That's the standard deviation. But different kind of modalities to bring down that error. We primarily try to find scalable ways to label. In some occasions, we use other sensors to help benchmark. But we primarily try to use cameras for this system. OK, thanks. Yeah, I mean, I think we want to aim for the car to be positioned accurately to the sort of centimeter level, something on that order.
[2:24:23] Obviously, it will depend on distance. Closed by things can be much more accurate than farther away things. And they would matter less because the car doesn't have to make decisions much farther away. And as it comes close, it will become more and more accurate. Exactly. A lot of questions. Thanks, everybody. My question has to do with AI and manufacturing. It's been a while since we've heard about the alien dreadnought concept. Is the humanoid that's behind you guys, is that kind of brought out of the production health timeline and saying that humans are underrated in that process?
[2:24:57] Well, something that I say is taken to too much of an extreme. There are parts of the Tesla system that are almost completely automated. And then there are some parts that are almost completely manual. And if you were to walk through the whole production system, you would see a very wide range from, like I said, fully automatic to almost completely manual.
[2:25:25] But the vast majority, most of it is already automated. And then with some of the design architecture changes, like going to large aluminum high pressure die cast components, we can take the entire rear third of the car and cast it as a single piece. And now we're going to do the front third of the car as a single piece. So the body line,
[2:25:55] Drops by like 60% to 70% in size. But yeah, the robot is not prompted specifically by manufacturing needs. It's just that we're just obviously making the pieces that are needed for a useful humanoid robot. So I guess we probably should make it. And if we don't, someone else will. So I guess we should make it and make sure it's safe.
[2:26:32] I should say, also, volume manufacturing is extremely difficult and underrated. And we've gotten pretty good at that. It's also important for that humanoid robot. How do you make the humanoid robot not be super expensive? Hi. Thank you for the presentation. And my question will be about scaling of Dojo.
[2:26:58] And in particular, how do you scale the compute nodes in terms of thermals and power delivery? Because there is only so much heat that you can dispense, and only so much power that you can bring to, like, cluster rack. And how do you plan to scale it, and how do you plan to scale it in multiple data centers?
[2:27:33] Hi, I'm Bill. I'm one of Dojo engineers. So from a thermal standpoint and power standpoint, we've designed it very modular. So what you saw on the compute tile, that will cool the entire tile. So once we hook it up to, it is liquid cooled on both the top and the bottom side. It doesn't need anything else. And so when we talk about clicking these together,
[2:28:00] Once we click it to power and once we click it to cooling, it will be fully powered and fully cooled. And all of that is less than a cubic foot. Yeah, so Tesla has a lot of expertise in power electronics and in cooling. So we took the power electronics expertise from the vehicle powertrain and the sort of advanced cooling that we developed for the power electronics and for the vehicle and applied that to the supercomputer.
[2:28:31] Because as you point out, getting heat out is extremely important. It's just really heat limited. So it's funny that at the compute level, it's operating at less than a volt, which is a very low voltage. There's a lot of amps. So therefore, a lot of heat. I squared R really bites you on the ass. Hi. My question's also really a question of scaling.
[2:29:02] So it seems like a natural consequence of using significantly faster training hardware is that you'd be either training models over a lot more data, or you'd be training a lot more complex models, which would be potentially significantly more expensive to run at inference time on the cars. I guess I was wondering if there was a plan to also apply Dojo as something that you'd be using on the self-driving cars, and if so,
[2:29:31] Do you foresee additional challenges there? So as you could see, Andre's models are not just for cars. There are auto labeling models. There are other models that are beyond car application. But they feed into the car stack. So Dojo will be used for all of those too, not just the car inference part of the training. Yeah, I mean, Dojo's first application
[2:30:04] Consuming video data for training that would then be run in the inference engine on the car. And that, I think, is an important test to see if it actually is good. Is it actually better than GPU cluster or not? But then beyond that, it's basically a generalized neural net training computer. But it's very much optimized to be a neural net. So CPUs and GPUs
[2:30:33] They're not designed specifically for training neural nets. We've been able to make GPUs especially very efficient for training neural nets, but that was never their design intent. So GPUs are still essentially running neural net training in emulation mode. So with Dojo, we're saying, OK, let's just ASIC the whole thing.
[2:30:59] Let's just have this thing that's built for one purpose, and that is neural net training. And just generally, any system that is designed for a specific purpose will be better than one that is designed for a general purpose. Hey, I have a question here. Hi. So you described two separate systems, one was for vision, one was for planner and control. Does Dojo allow you to train networks that cross that boundary? And second thing is...
[2:31:28] If you were able to train such networks, would you have the onboard compute capability in the FSD system to be able to run that under your tight latency constraints? Thanks. Yeah, I think we should be able to train panel networks on Dojo or any GPUs. It's really
[2:31:45] And I think, if anything, once we make this entire thing end-to-end, it will be more efficient than decoding all of these intermediate states. So you should be able to run faster if you make the entire thing end-to-end neural networks. We can avoid a lot of decoding of the intermediate states and only decode the essential things required for driving the car. Yep. Certainly, end-to-endness is the guiding principle behind a lot of the network developments. And over time, in the stack, neural networks have taken on more and more functionality.
[2:32:12] We want everything to be trained end-to-end because we see that that works best. But we are building it incrementally. So right now, the interface, there is vector space, and we are consuming it in the planner. But nothing really fundamentally prevents you from actually taking features and eventually fine-tuning end-to-end. So I think that's definitely where this is headed. And the discovery really is, what are the right architectures that we need to place in the network blocks to make it amenable to the task? So like Ana described, we can place spatial RNNs to help with the perception problem.
[2:32:40] And now it's just neural networks. So similarly for planning, we need to bake in search optimization into the network architecture. And once we do that, you should be able to do planning very quickly, similar to C++ algorithms. OK. I think I had a question very similar to what he was asking about. It seems like a lot of neural nets are around computer vision. And kind of traditional planning, you had model predictive control and solving convex optimization problems very quickly.
[2:33:14] And I'd wonder if there's a compute architecture that's more suited for convex optimization or the model predictive control solutions very quickly. Yeah, 100%. We want to bake in, like I said earlier, we want to bake in these architectures that do, say, model predictive control, but just replace some of the blocks with neural networks. Or if we know the physics of it, we can also use physics-based models, part of the neural networks, for our path itself. So we are going to go towards a hybrid system where we will have neural network blocks
[2:33:43] I mean, I do think that as long as you've got surround video neural nets for understanding what's going on and can convert
[2:34:13] There's surround video into vector space. Then you basically have a video game. And if you're in Grand Theft Auto or whatever, you can make the cars drive around and pedestrians walk around without crashing. So you don't have to have a neural net for control and planning. But it's probably ultimately better. But I think you can probably get to
[2:34:41] In fact, I'm sure you can get to much safer than human with control and planning primarily in C++ with perception, vision in neural nets. Hi. My question is we've seen other companies, for example, use reinforcement learning and machine learning to optimize.
[2:35:04] Power consumption and data centers and all kinds of other internal processes. My question is, is Tesla using machine learning within its manufacturing design or other engineering processes? I discourage use of machine learning because it's really difficult. Basically, unless you have to use machine learning, don't do it.
[2:35:34] It's usually a red flag when somebody says, we want to use machine learning to solve this task. I'm like, that sounds like bullshit. So 99.9% of the time, you do not need it. So it's kind of like you reach for machine learning when you need to, but I've not found it to be a convenient, easy thing to do. It's a super hard thing to do.
[2:36:03] That may change if you've got a humanoid robot that can understand normal instructions. But yeah, generally minimize use of machine learning in the factory. Hi. Based on your videos from the simulator, it looked like a combination of graphical and neural approaches. I'm curious what the set of underlying techniques that are used for your simulator and specifically for neural rendering, if you can share.
[2:36:38] Yeah. So we're doing, at the bottom of the stack, it's just traditional game techniques, just rasterization real time, very similar to what you'd see in like GTA. On top of that, we're doing real time ray tracing. And then those results were really hot off the press. I mean, we had that little asterisk at the bottom that that was from last night. We're going into the neural rendering space. We're trying out a bunch of different things. We want to get to the point where the neural rendering
[2:37:07] We have a lot of people who can help us make this run at real time.
[2:37:37] But we're going to try whatever we can do to get to the point where we can train everything just with the simulator if we had to, but we will never have to because we have so much real-world data that no one else has. It's just to fill in the little gaps in the real world. Yeah, I mean, the simulator is very helpful when there's, like, these rare cases, like, you know, like collision avoidance right before an accident.
[2:38:04] And then ironically, the better our cars become at avoiding accidents, the fewer accidents there are. So then our training set is small. So then we have to make them crash in the simulation. So it's like, OK, minimize potential injury to pedestrians and people in the car. You have five meters. You're traveling at 20 meters per second. What actions would minimize probability of injury?
[2:38:33] We can run that in some cars driving down the wrong side of the highway, that kind of thing. Happens occasionally, but not that often. For your humanoid context, I'm wondering if you've decided on what use cases you're going to start with and what the grand challenges are in that context to make this viable. Well, I think for the Tesla bot, Optimus, it's basically going to start with
[2:39:09] Just dealing with work that is boring, repetitive, and dangerous. Basically, what is the work that people would least like to do? Hi. So quick question about your simulations. Obviously, they're not perfect right now. So are you using any sort of domain adaptation techniques to basically bridge the gap between your simulated data and your actual real-world data? Because I imagine it's kind of dangerous to...
[2:39:46] Just deploy models which are solely trained on simulated data. So maybe some sort of explicit domain adaptation or something. Is that going on anywhere in your pipeline? So currently, I mean, we're producing the videos straight out of the simulator, the full clips of kinematics and everything. And then we're just immediately training on them. But it's not the entire data set. It's just a small targeted segment. And we only are evaluating based on real world video.
[2:40:13] We're paying a lot of attention to make sure we don't overfit. And if we have to start doing fancier things, we will. But currently, we're not having an issue with it overfitting on the simulator. We will as we scale up the data. And that's what we're hoping to use neural rendering to bridge that gap, to push that even further out. We've already done things where we're using the same network as in the car, but retrain it to detect sim versus real to drive our art decisions. And that's actually helped prevent some of these things as well.
[2:40:42] Yeah, just to emphasize that overwhelmingly the data set is the real video from the cars on the actual roads. Nothing's weirder or has more corner cases than reality. It gets really strange out there. But then if we find, say, a few examples of something very odd, and there's some very odd pictures we've seen, then in order to train it effectively, we want to create simulations
[2:41:11] Say, a thousand simulations that are variants of that quirky thing that we saw. To fill in some important gaps and make the system better. And really, all of this is about, over time, just reducing the probability of a crash or an injury. And it's called the March of Nines. Like, how do you get to 99.99999% safe, you know? And, yeah.
[2:41:41] Each nine is an order of magnitude difficulty increase. Thanks so much for the presentation. I was curious about the Tesla bot. Specifically, I'm wondering if there are any specific applications that you think the humanoid form factor lends itself to. And then secondary, because of its human form factor, is emotion or companionship at all thought about on the product roadmap at all?
[2:42:16] Well, we certainly hope this does not feature in a dystopian sci-fi movie. But, you know, like, really, at this point, we're saying, like, maybe this robot can... We're trying to be as literal as possible. Can it do boring, dangerous, repetitive jobs that people don't want to do? And, you know, once we can have it do that, then maybe it can do other things, too, but...
[2:42:49] That's the thing that would be really great to have. It could be your buddy, too. If I wanted to have it be your friend and whatever. I'm sure that people will think of some very creative uses.
[2:43:12] So firstly, thanks for the really incredible presentation. My question is on the AI side. So one thing we've been seeing is that with some of these language modeling AIs, we've seen that scaling has just had incredible impacts in their capabilities and what they're able to do. So I was wondering whether you're seeing similar kinds of effects of scaling in your neural networks and your applications. Absolutely. A bigger network, typically we see it performs better, provided you have the data.
[2:43:42] to also train it with. And this is also what we see for ourselves. Definitely in the car, we have some latency consideration to be mindful of. And so there we have to get creative to actually deploy much, much larger networks. But as we mentioned, we don't only train neural networks for what goes in the car. We have these auto labeling pipelines that can utilize models of arbitrary size. So in fact, we've trained a number of models that are not deployable that are significantly larger and work much better because we want much higher accuracy for the auto labeling.
[2:44:09] And so we've done a lot of that, and there we definitely see this trend. The order labeling is an extremely important part of this whole situation. Without the order labeling, I think we would not be able to solve the self-driving problem. It's kind of a funny form of distillation, where you're using these very massive models plus the structure of the problem to do this reconstruction, and then you distill that into neural networks that you deploy to the car.
[2:44:35] But we basically have a lot of neural networks and a lot of tasks that are never intended to go into the car. And also, as time goes on, you get new frames of information. So you really want to make sure your computer is distributed across all the information as opposed to just taking a single frame and hogging on it for, say, 200 milliseconds. You actually will have newer frames coming in. So you want to use all of the information and not just use that one frame. I think one of the things we're seeing is that the car's predictive ability is eerily good.
[2:45:07] It's really getting better than human in terms of predicting. Like you said, predict what this road will look like when it's out of sight, like it's around the bend. And it predicts the road with very high accuracy. And predict pedestrians or cyclists where it just sees a little corner of the bicycle and a little bit through the windows of the bus. Its ability to predict things is
[2:45:35] It's going to be much better than humans, like really way beyond. Right. Yeah, we see this often where we have something that is not visible, but the neural network is making up stuff that actually is very sensible. Sometimes it's eerily good, and you're wondering this isn't a training set. And actually, in the limit, you can imagine the neural net has enough parameters to potentially remember Earth. So in the limit, it could actually give you the correct answer, and it's kind of like an HD map baked into the weights of the neural net. Yeah.
[2:46:08] I have a question about the design of the Tesla bot. Specifically, in order, how important is it to maintain that humanoid form? To build hands with five fingers that also respects the weight limits could be quite challenging. You might have to use cable-driven, and then that also causes all kinds of issues. I mean, this is going to be bot version one. I mean, we'll see. So it's...
[2:46:38] It needs to be able to do things that people do and be a generalized humanoid robot. You could potentially give it two fingers and a thumb or something like that. For now, we'll give it five fingers and see if that works out OK. It probably will. It doesn't need to have incredible grip strength, but it needs to be able to
[2:47:09] Work with tools, carry a bag, that kind of thing. Hi, thanks a lot for the presentation. So an old professor of mine told me that the thing he disliked a lot about his Tesla was that the autopilot UX didn't really inspire much confidence in the system, especially when objects are spinning, classifications are flickering. I was wondering, even if you have a good self-driving system, how are you working on convincing Tesla owners, other road users?
[2:47:40] Are there other road users and just the general public that your system is safe and reliable? Well, I think that's the cars. A while back, cars used to spend. They don't spend any more. If you've seen the FSD beta videos, they're pretty solid. And they will be getting more solid. Yeah, as we add more and more data and train these multicamber networks, these are pretty recent, actually, just a few months old. And it's still improving. It's not a done product.
[2:48:09] No minds, we can clearly see how this is just going to be perfect because why not? All the information is there in the videos. It should produce a given lots of data and good architectures. And this is just an intermediate point in the timeline. It's clearly headed to way better than human, without question. Hi, here. I was wondering if you could talk a little bit about the short- to medium-term economics of the bot, I guess. I understand the long-term vision of replacing
[2:48:44] Hello. Hi. I was curious to know how the car AI prioritizes
[2:49:21] Occupant safety versus pedestrian safety. And what thought process goes into deciding how to make this into the AI? Well, the thing to appreciate is that from the computer standpoint, everything is moving slowly. So to a human, things are moving fast. To the computer, they are not moving fast. So I think this is, in reality, somewhat of a false dichotomy. Not that it will never happen, but it will be very rare.
[2:49:54] You think it was like going the other direction, like rendering with full ray tracing, neural net enhanced graphics on something like Cyberpunk or any advanced video game, doing 60 frames a second, perfectly rendered. How long would it take a person to even render one frame without any mistakes? Can't be done.
[2:50:24] I mean, it would take like a month just to render one frame out of 60 in a second in a video game. Computers are fast and humans are slow. I mean, for example, on the rocket side, you cannot steer the rocket to orbit. We actually hooked up a joystick to see if anyone could steer the rocket to orbit, but you need to react at roughly
[2:51:01] 6, 7 hertz, people can't do it. That's pretty low. We're talking aiming for 30 hertz type of thing. Hi. Over here. With hardware 3, there's been lots of speculation that with larger nets, it's hitting the limits of what it can provide. How much headroom has the extended compute modes provided, and at what point would hardware 4 be required at all?
[2:51:34] Well, I'm confident that hardware 3 or the full self-driving computer 1 will be able to achieve full self-driving at a safety level much greater than a human, probably, I don't know, at least 200 or 300% better than a human. Then, obviously, there will be a future hardware 4 or full self-driving computer 2, which we'll probably introduce with the Cybertruck, so maybe in about a year or so.
[2:52:02] That will be about four times more capable, roughly. But it's really just going to be, like, can we take it from, say, for argument's sake, 300% safer than a person to 1,000% safer? Just like there are people on the road with varying driving abilities, but we still let people drive. You don't have to be the world's best driver to be on the road. So, as we see.
[2:52:33] So are you worried at all, since you don't have any depth sensors on the car, that people might try adversarial attacks, like printed out photos or something to try to trick the RGB neural network? Yeah, like what? Pull some Wiley Cody stuff? Paint the tunnel on the wall? It's like, oops.
[2:53:07] We haven't really seen much of that. I mean, for sure, like right now, most likely, if you had a t-shirt with a stop sign on it, which I actually have a t-shirt with a stop sign on it, and then you flash the car, it will stop. I've proved that. But we can obviously, as we see these adversarial attacks, then we can train.
[2:53:40] And then, you know, I think, you know, I think, you know, I think, you know, I think,
[2:54:12] Do you assume that they're all going to follow those rules, or are you accounting for the possibility that, well, they might be bad drivers, for example? Yeah, we do account for multimodal futures. It's not that we just choose one. We account for, okay, this person can actually do many things, and we use their actual physics and kinematics to make sure that they're not doing a thing that would interfere with us before we act. So if there's any uncertainty, we are conservative, and then we yield to them. Of course, there's a limit to this, because if you're too conservative, then...
[2:54:42] It's probably not practical. So at some point, we have to assert. And even then, we make sure that the other person can yield to us and act sensibly. I should say, before we introduce something into the fleet, we will run it in shadow mode. And we'll see what would this neural net, for example, have done in this particular situation. And then effectively, the drivers are training it.
[2:55:15] Training the net. So if the neural net would have controlled and, say, veered right, but the person actually went left, it's like, oh, there was a difference. Why was there that difference? All the human drivers are essentially training the neural net as to what is the correct course of action, assuming it doesn't then end up in a crash. It doesn't count in that case.
[2:55:40] Yeah, and secondly, we have various estimates of uncertainty, like Flickr. And when we observe this, we actually say we are not able to see something. We actually slow down the car to be, again, safe and get more information before acting. Yeah, we don't want to be brazen and just go into something that we don't know about. We only go into places where we know about. Yeah. Yeah. It should be like, aspirationally, the car should be the less it knows, the slower.
[2:56:06] Thanks for the presentation. So I am curious, appreciate the fact that the FSD is improving, but if you have the ability to improve one component along the AI stack presented today.
[2:56:36] Whether it is simulation, data collections, planning and control, et cetera. Which one, in your opinion, is going to have the biggest impact for the performance of the full self-driving system? It's really the area under the curve of multiple points. And if you improve anything, it should improve the area. In the short term, it's arguably, we need all of the nets to be surround video.
[2:57:05] And so we still have some legacy. This is very short term. Obviously, we're fixing it fast. But there's still some nets that are not using surround video. And I think, ideally, they'll all use surround video. Yeah. I think a lot of puzzle pieces are there for success. We just need more strong people to also just help us make it work. That is the actual bottleneck, I would say, and really one of the reasons that we are putting on this event. Exactly. Well said, Andre.
[2:57:34] There's just a tremendous amount of work to do to make it work. So that's why we need talented people to join and solve the problem. Thank you for the great presentation. Lots of my questions answered. But one thing is, imagine that now you have a large amount of data, even unnecessary. How do you consider that there is a forgetting problem in neural networks?
[2:58:07] Are you considering those aspects? And also, another one, are you considering online learning or continuous learning so that maybe each driver can have their version of self-driving? I think I know the literature that you're referring to. That's not some of the problems that we've seen. And we haven't done too much continuous learning. We trained the system once. We fine-tuned it a few times.
[2:58:34] That sort of goes into the car. We need something stable that we can evaluate extensively, and then we think that that's good, and that goes into cars. So we don't do too much learning on spot or continuous learning and don't face the forgetting problem. But there will be settings that you can say, like, do you want, are you typically a conservative driver, or do you want to drive fast or slow? You know, it's like, I'm late for the airport. Could you go faster than, you know, basically the kind of instructions you'd give to your Uber driver. It's like, I'm late for the flight, please hurry.
[2:59:03] Or take it easy, or whatever your style is. Let's take a few more questions here, and then we'll call it a day. All right. So as our models have become more and more capable, and I guess you're deploying these models into the real world, one thing I guess that's possible is for AI to become more, I guess, misaligned with what humans desire. So I guess, is this something that you guys are worried about as you guys deploy more and more robots? Or do you guys, like, we'll solve that problem when we get there?
[2:59:40] Yeah, I think that we should be worried about AI. What we're trying to do here is narrow AI, pretty narrow, like just make the car drive better than a human, and then have the humanoid robot be able to do basic stuff. So at the point at which you sort of start to get to superhuman intelligence, I don't know. All bets are off.
[3:00:10] That'll probably happen, but what we're trying to do here at Tesla is make useful AI that people love and is unequivocally good. That's our, you know, we'll try to aim for that. Okay, maybe one more question. Hi, my question is about the camera sensor. In the beginning of the talk, you had mentioned about building a synthetic animal. And if you think about it,
[3:00:44] A camera is a very poor approximation of a human eye. And a human eye does a lot more than take a sequence of frames. Have you looked into, like, there are, like, these days are, like, cameras, like, event cameras. Have you looked into them? Or are you looking into a more flexible camera design or building your own camera, for example? Well, with hardware 4, we will have a next-generation camera. But I have to say that the current cameras, we have not reached the limit of the current cameras. So...
[3:01:15] And I'm confident we can achieve full self-driving with much higher safety than humans with the current cameras and current compute hardware. But are we good to be 1,000% better rather than 300% better? So we'll see continued evolution on all levels in pursuit of that goal. And I think in the future, people will look back and say, wow, I can't believe we had to drive these cars ourselves.
[3:01:45] You know, like self-driving cars will just be just a normal, like self-driving elevators. You know, elevators used to have elevator operators. And there's someone there with like, you know, big relay switch operating the elevator. And then every now and then they'd get tired or, you know, make a mistake and shear somebody in half. So now we, you know, we made elevators automatic. And you just go and you press the button and you can be in a 100-story skyscraper.
[3:02:14] Don't really worry about it. Just go and press a button, and the elevator takes you where you want to go. But it used to be that all elevators were operated manually. It'll be the same thing for cars. All cars will be automatic. And electric, obviously. So there will still be some gasoline cars and some manual cars, just like there are still some horses. All right. Well, thanks, everyone, for coming. And I hope you enjoyed the presentation. And thank you for the great questions. All right.