{"url": "https://www.youtube.com/watch?v=5NGYe-EpO1g", "page_url": "https://www.youtube.com/watch?v=5NGYe-EpO1g", "title": "ai.bythebay.io: Tom Brown, Measuring AI Capabilities with OpenAI Gym & Universe", "show_or_channel": "FunctionalTV", "date": "2017-03-08", "duration_s": 2238, "type": "talk", "speakers": ["Tom Brown"], "confidence": "high", "notes": "AI By the Bay 2017 talk while at OpenAI (sched page confirms speaker bio: Grouper/MoPub).", "people": ["tom-brown"], "id": "2017-03-08_functionaltv-ai-bythebay-io-tom-brown-measuring-ai", "audio_duration_s": 2237.952, "worker": "jc-w06", "transcribe_s": 25.0, "model": "large-v3", "segments": [{"start": 15.73, "end": 44.98, "text": "Today we're going to talk about measuring AI progress with OpenAI Gym and Universe. And so those are two different products that OpenAI has built out. I'm going to explain why they are, kind of how to use them, why we should care about them, and we're going to do a little collaborative coding adventure together where we're going to implement a superhuman agent in about 10 minutes."}, {"start": 45.14, "end": 74.26, "text": "So we got a lot to cover. So first, what is OpenAI? OpenAI is a nonprofit research company that is focused on building out AI in a way that's good for everybody. So it's about 80% researchers right now and about 20% engineers. And our mission, and also plus Jack, who runs our comms."}, {"start": 74.26, "end": 104.34, "text": "And has helped me put together this talk, so thanks, Jack. And OpenAI's mission is to build safe AI and to ensure that the benefits of AI are as widely and evenly distributed as possible. And since we're about 80% research and 20% engineering, that basically reflects how we tackle this problem, which is by doing fundamental research ourselves to push on the capabilities of AI."}, {"start": 104.34, "end": 134.38, "text": "As well as building out tools to make everybody in the global research community more capable. And I'm Tom Brown and similarly I have a personal mission which is to find allies to push on safe AI and to make those allies as capable as possible. I'm one of the 20% of engineers at OpenAI so most of my time is spent trying to figure out how we can build tools and products that will make"}, {"start": 134.38, "end": 163.86, "text": "The global research community more capable, more unified, and more knowledgeable about what's going on. And so today, we're going to talk about some of those different products that the team has built out. So first, I'm going to talk about the power of metrics. You all are here at an AI conference, so you are very analytical. So I assume that you're already sold on metrics. I'm going to argue that you should be even more sold on metrics. And I'm going to talk about that."}, {"start": 163.86, "end": 190.14, "text": "In three different contexts, for individual tasks, for organizations, and for entire research fields. So first, tasks. I'm gonna talk, basically, GIM and Universe are ways of defining tasks. I'm gonna talk through them more later. But you can look at this learning curve here, which is."}, {"start": 190.14, "end": 220.05, "text": "Some task that you have a reinforcement learning agent trying to achieve. And you can see it sucks at the beginning and then gets better. So here's a video and you can see that it wants to walk. So the metric that it's trying to get to is how far to the right in this infinite plane it has walked. And you can see it can walk like two steps. And it isn't getting better very fast. But if we wait for another"}, {"start": 220.05, "end": 251.98, "text": "40,000 steps. It gets pretty good. It could work on its running form a little bit, but it's moving very quickly. Let's look at another environment. This is a racing game. Similarly, you have this boat, and what you're trying to do is race faster than these other boats. You can't see here because it's cropped, but instead of having a distance,"}, {"start": 251.98, "end": 281.34, "text": "Metric, we only have score. So what we measured here was score. And you can see, if you look at this graph, it seems like our score goes up over time, right? And it seems like something interesting happens at around 80K. And so whenever something like that happens, we'll get really excited, and we'll go and look at what actually is happening by watching a video that's been made. So here's the video that we found. All right, so it's driving."}, {"start": 281.34, "end": 308.8, "text": "You can see here it's got 1,000 scores, so it's happy. It's still going. And it's not driving great, but somehow it's getting this high score. And so it hit those buoys, and now it scores 3,500, 60,000. And so what it has learned is that trying to win at this race isn't the way to get a score at all. What you should really do is spin around and hit these buoys a lot."}, {"start": 308.8, "end": 335.71, "text": "But the bummer here is that this boat's on fire, and it's spinning around. This is definitely not safe AI. I would not want to be in this boat. So something has gone wrong here. And the particular thing that's gone wrong is that we were measuring a proxy for reward, and we've misspecified what the actual metric is that is supposed to be optimized. So this is the type of things that you discover when you end up having a bunch of different environments that you're training agents on."}, {"start": 335.71, "end": 364.14, "text": "And I'm also going to argue that this same type of thing can happen not just for individual tasks, but also for organizations. So I came from a startup background. I went through Y Combinator a few years back. And one of the main mottos that they drilled into us was that you make what you measure, which basically means that anything that you put up as your number one metric"}, {"start": 364.14, "end": 393.17, "text": "If you take that metric, put it up on the wall, it will tend to go up. And you'll just constantly be thinking of ways to make it go up. And that you'll also be vulnerable to misspecified rewards there. And that actually was something that I found in my company. We basically ended up optimizing for getting the most users, but didn't measure how happy we were making those users. And that ended up as you would predict."}, {"start": 393.17, "end": 417.89, "text": "So this affects organizations. It's incredibly important to figure out what it is that you're measuring. And it also affects entire fields. So you're all familiar with ImageNet. ImageNet was launched by Fei-Fei Li in 2010. And when it was launched, it basically seemed like it was too difficult for people."}, {"start": 417.89, "end": 440.05, "text": "There were already good benchmarks that were out there. And you can see that over from 2010 to 2011, there was only a few percentage points of dropping. But what mattered was that it was something that was already out there and people were all working together to try to improve it because Fei-Fei knew that this would be an important thing."}, {"start": 440.05, "end": 465.55, "text": "That eventually would happen. She didn't know how long it would be. But it seemed like we would be able to solve or to work to get much more progress on vision. So in 2012, AlexNet came out. And because we already had this benchmark, it was something that people couldn't ignore. This was a night and day shift and basically launched the entire deep learning revolution that we're seeing now."}, {"start": 466.13, "end": 488.67, "text": "Plus, as we've continued over the years, this has just been pulled down bit by bit by people collaborating and working to make this number decrease. And now you can see as of 2015 that the human who actually is, we know because of OpenAI's very own Andrej Karpathy,"}, {"start": 488.67, "end": 515.33, "text": "Who spent about a hundred hours training his own brain on how to label all these images that you have to label for ImageNet. We know that now our image classification is superhuman for ImageNet and basically supervised learning is basically solved if you can get enough labeled data."}, {"start": 515.33, "end": 540.48, "text": "You can be confident that you'll be able to train an agent that will perform as well as a human, which is huge. So I think the next question then is what metric comes next if we want to look as a field at where the most important changes are going to come from."}, {"start": 540.48, "end": 570.8, "text": "I would argue that reinforcement learning is a good candidate for that. And so I'm going to talk a little bit about reinforcement learning, and then I'm going to code up an agent that works in reinforcement learning environments afterwards. So RL, many of you are familiar with this. Basically, the idea is you have an agent that's interacting with an environment."}, {"start": 570.8, "end": 599.02, "text": "The actor that you're coding up, the environment is everything that's not the agent. And the agent can take some actions on the environment. The environment will have stuff happen. If you imagine this being a robot, the robot can move its arms, it can change its actuators, and then the environment will give it back some sort of observations to its sensors, like if it has cameras or if it has ways of"}, {"start": 599.02, "end": 626.21, "text": "Feeling the distance that it's moved, those will be the observations. And you also need a reward, which lets the agent know how well it's performing the task that it's trying to do. So this is the standard agent environment setup. Basically, GIM just implements the environment. So you call GIM.make with a string that defines a single environment."}, {"start": 626.21, "end": 654.34, "text": "You'll always get back exactly the same environment when you call gym.make. And if you go to gym.openai.com, you can see a list of all the different environments that have been added by the community. So, for example, you could do Ms. Pac-Man, and when you call gym.make, you get this little environment. And so next, when you call step, you'll get your observation and reward back. And that action will come from the agent that you write."}, {"start": 654.34, "end": 684.11, "text": "So you'll have to figure out some action to give it. The other thing it needs to have is reset, which brings you back to the state. So, Jim, make, step, reset. Very tiny API, but you can fit a lot of stuff into it. So the community has added all the different Atari games. We've added a bunch of different flash racing games, and I'll talk more about these later. These are interesting because"}, {"start": 684.11, "end": 715.26, "text": "Basically, all racing games are kind of the same. You're trying to drive, and you want to drive fast, and if you drive fast, you'll basically do well, excluding that boat mix-up. We've got a bunch of different robot control tasks, which basically, instead of having... So, sorry, for all of those previous ones, the observation will always be screen pixels. For this, instead, you have different sensors on these robots' joints."}, {"start": 715.26, "end": 744.61, "text": "That become your observations. And we've even added some safety things like this predict action carpool. Carpool is a classic reinforcement learning task where you have to balance the stick. And predict action carpool tries to balance the stick while at the same time letting you know what actions it's going to take over the next few seconds, which would be a useful thing to do if you have a car. You'd want to be able to know what the car is planning to do."}, {"start": 744.61, "end": 773.58, "text": "So there's a bunch of different types of safety problems. This is just one example. And then most recently, we've added a bunch of browser tasks. So this is very fast, but if you pay attention to what's going on here, this is an agent choosing a bunch of different things to do, like browsing a file system, moving, filling in this form, trying to stay in this black box, basically anything that you can do on a browser."}, {"start": 773.78, "end": 804.22, "text": "Which copy pasting, selecting from lists, and I guess like banning people on Twitter or something. So anyway, so a question then is like, what doesn't fit in this API? And it's basically just environments that can't be reset. So you can fit almost anything in there, except for things where you have like an actual robot interacting with the world. The robot then you would want to be able to have it practice a bunch of times."}, {"start": 804.22, "end": 834.45, "text": "But you probably won't be able to set up the environment exactly the same way that you had it originally. Unless you had like a time machine. So I think that that's okay. It seems like most tasks actually fit within this. So next I'm going to tell you about Universe, which is our giant collection of different gym environments. And I'll tell you why that's cool. But first, as promised, we are going to implement this agent."}, {"start": 834.9, "end": 870.58, "text": "So let me grab a drink of water. OK, 10 minutes. So we are in Python world. We're going to write a simple agent. We're going to do our env gym.make. Let's do breakout. And again, if you're looking for a list of environments, you can find that on the gym website. We're going to import gym. We need to get our first observation."}, {"start": 870.58, "end": 899.57, "text": "Which is m.reset. And then we're going to take a bunch of steps. So let's do, like, for we don't care in range 500. We're going to take a step. And our action here is just going to be one. And I know that the one is the action that I want here. Because I've, like..."}, {"start": 899.66, "end": 932.62, "text": "Done a lot of these Atari things. But you can look it up here in the GIM repo. And generally, the GIM code is quite well documented. So here, one is fire, which is also like the OK button. So we've got to hit OK to get started. So we'll call end.step. And then just so that we can see it here, we'll end render. So let's see if this works. There we go. So this is our wonderful agent. It plays Breakout."}, {"start": 932.62, "end": 963.42, "text": "It didn't do great, but all right, let's make it better. So problems with this agent. One thing is that we don't do anything with m.step. So m.step returns obs reward done info. And what we want is to, if it's done, then we're going to restart the environment. And let's say if it has a reward,"}, {"start": 963.42, "end": 994.9, "text": "Let's do, let's print the reward. And right now our agent's not getting any reward. So let's fix that. Let's do good old random choice of one, two, three. So now our agent is going to move randomly and it should get some reward. Come on, there it goes, reward one. All right, so still not superhuman. Let's see what we can do."}, {"start": 995.28, "end": 1029.52, "text": "So, one thing we can do here is let's measure the total reward, too. So, here we go. Total reward is going to be zero. And then when we print the reward, we're also going to add it to the total reward. Very interesting. And then at the end of this, we're going to print the total reward. Great."}, {"start": 1029.52, "end": 1057.3, "text": "Let's actually look at some other environments too. So what's cool about this is that this works for Breakout. It will also work for, let's do like Gravatar. Basically any of the different Atari games have this same input. So here it is. It's doing about as well as I would do at Gravatar. Let's see."}, {"start": 1057.3, "end": 1086.98, "text": "They all have the same input, which random choice of 1, 2, 3 just returns a single number, either 1, 2, or 3. So any of the different Atari games, like Ms. Pac-Man, et cetera, will work there. Let's do some of the robotic control games. So we could do Ant. For Ant, this 1, 2, 3 action won't work, because in Ant, instead of having a little controller that you're using, you have these robotic limbs."}, {"start": 1086.98, "end": 1112.32, "text": "So instead, we need a bunch of zeros. 1, 2, 3, 4, 5, 6, 7, 8. This isn't what my code looks like in production, but if we give this agent this ant v1, if we give it this input, here we have our ant, and it sits there with no strength in its limbs."}, {"start": 1112.32, "end": 1141.71, "text": "And you can see here it's getting a reward of 0.99 each second. This is basically just the reward that it gets for just being alive. The ant actually wants to get to the right side, but this ant just sitting there is OK. OK, so this is our ant. Let's upgrade our ant. Let's say action equals. We have a helper for doing the sample thing. The random choice would be a."}, {"start": 1141.71, "end": 1169.82, "text": "A little bit more annoying with this eight-dimensional space. So we can do env.ActionSpace.sample, which will just give us a random action. And here we go. OK. This is not the best. Still not superhuman. I think that we've actually had people in the office try to play as Ant. It's incredibly difficult. Have you guys ever played QWOP? It's basically like this game here."}, {"start": 1169.82, "end": 1201.58, "text": "Where you're trying to move all of your muscles, but your brain doesn't know how to move them at all. So it's this. So we can do slightly better than that random agent as a human. But the score that we end up with is maybe we've gone three steps to the side before our time runs out. So let's do better."}, {"start": 1201.58, "end": 1228.88, "text": "All right, I said we'd do this in 10 minutes. I wanted to give you guys a quick overview of policy gradients, but Jack said that we did not have time for that. So I'm going to import my trained agent. I have a MuJoCo agent, and we are going to cheat and use that. So I've got my MuJoCo agent. I'll also need TensorFlow."}, {"start": 1228.88, "end": 1258.96, "text": "So we're going to start up a little TensorFlow model that has been trained. So we're going to have with TensorFlow session, take all of these things, import TensorFlow, and now we want our policy, which is our MuJoCo agent, which basically has just a very tiny neural net in it. Let's do..."}, {"start": 1258.96, "end": 1288.45, "text": "So basically here in my directory that we're looking at, I've got this trained models folder. And inside of it, we've got a bunch of different dot pkl is a dot pickle file. So it's basically just a dumped Python object that contains all the weights for an agent that we trained last night that knows how to do these various different environments. So since we want ant, we can use this little tiny little 148k ant pickle file."}, {"start": 1288.45, "end": 1316.78, "text": "And pull those weights out and put them in our TensorFlow object. So that's what policy does. So let's do it. Our new action is going to be this policy. And the policy takes an observation. And it gives us the best action that this model has learned that will, in the long run, give it a high score. So if all works well,"}, {"start": 1316.78, "end": 1354.56, "text": "We should start this, and we should see an agent that really knows how to get a high score in this environment going and getting a high score. Let's see. Or it'll give us a bug. All right, so I have to find where this thing lives. It's this pickle file. All right, take two. Oh, wow. All right, so this is our trained agent. It knows how to run really fast. And so this is how you get an agent working in Jim."}, {"start": 1354.56, "end": 1383.41, "text": "And total reward is four, which was much higher than the zero that we got before. Actually, I did a typo. Total reward should be like 4,000. This should be total reward. But that's OK. Typos. All right, cool. So this is like a basic primer on GIM. What you can do is after you run this, you can upload your trained results. I'll show you a picture of that here. And I'll explain what universe is."}, {"start": 1384.21, "end": 1414.72, "text": "All right. Universe. It's an entire universe of gym environments. It's got lots of gym environments in it. So we're going to talk about why that's cool. To start with, imagine if you have, yeah, so we have about 100 racing environments, each represented by a little pink dot here. Here's some agents playing them. They're clustered together because they're similar."}, {"start": 1414.72, "end": 1444.53, "text": "So these racing environments are like, yeah, like I said, you do basically the same thing. I played a lot of racing environments now, and there is not a ton of diversity. So these dots are, I just like drew them in this PowerPoint, but the idea is that they are in a space of all possible environments. And they're grouped together in location here by how similar they are."}, {"start": 1444.53, "end": 1473.94, "text": "And we don't have a great idea of what that actually means beyond the intuitive idea that racing environments seem similar, the robot environments seem much different. And what we expect is that as we get this large collection of different environments and see what agents work on different ones, we'll have a better idea of what similarity of environments actually means and what the different axes are that are collapsed into this 2D space."}, {"start": 1473.94, "end": 1503.65, "text": "And one way that we want to look at that is through transfer learning. And so the basic setup for transfer is we have 100 different racing environments. Let's pick 20 of them and train on the first 80. Set the 20 aside. And then after training on those 80 environments, see how much better we are at working on these 20 holdout environments, these 20 test environments."}, {"start": 1503.65, "end": 1525.12, "text": "Compared to an agent that has never had any experience with racing environments before. And if you can show that having had experience with all these other racing environments made you better equipped to deal with these test environments, then you've illustrated transfer. And there's many people who think that that is..."}, {"start": 1525.12, "end": 1553.12, "text": "One of the big differences now between our AI systems and humans is just that humans have tons and tons of experience going into any new task, whereas these agents are taking their very first steps when they are placed into it. So you can imagine we have this cluster of racing environments that we think are close together. You can look at all the web browser environments that we showed before."}, {"start": 1553.12, "end": 1578.48, "text": "And to me, they feel like they're probably more spread out, that having played 30 racing games would make me more prepared for another racing game than having done 30 tasks on the internet would have prepared me for doing the 31st. So that's why these are kind of spread out. But even more, those two groups are very spread out in terms of space."}, {"start": 1578.48, "end": 1608.83, "text": "Where having done racing games won't prepare you for filling out your taxes at all, basically. So universe is basically that space. And what we're trying to do is to first show transfer within these individual clusters, and then expand that to have a single cluster that encompasses every task that"}, {"start": 1608.83, "end": 1636.43, "text": "A human can do on a computer. So the idea here is that basically you can measure intelligence or think of intelligence in general as your ability to achieve your goals in a wide variety of tasks. That's the best definition that I've heard of it. And universe fills up a large portion of the total tasks that a human being can do on a computer."}, {"start": 1636.43, "end": 1665.76, "text": "This is a little bit around the edge, which it would be difficult to have that reset constraint. But almost any task a human can do would fit in this. And what's exciting is that we already have some solved environments in this. For example, playing that little ant game, doing that as a human is much harder than QWOP, whereas an agent can learn how to do it without having any experience with anything else in about 20 minutes."}, {"start": 1665.76, "end": 1693.06, "text": "So that's an example of something that's already solved. It's like a tiny fraction of the total possible space. But we think it's growing, or it clearly is growing, and we expect this to be something that will eventually fill up this entire circle. And you can see some of this progress now on gym.openai.com, where people can upload their results."}, {"start": 1693.06, "end": 1721.76, "text": "And track the progress of various different agents on various different tasks. So things that I'd like from you all is if this seems interesting to you, I'd love to get you using GIMP, telling us where it's great, telling us where it sucks, and submitting your agents to the scoreboard so that we can see what the state of the art is with AI."}, {"start": 1721.76, "end": 1761.39, "text": "And if you think that building these types of products or pushing on this type of research is interesting, please check out openai.com slash jobs or come talk with me after this or send me an email. Thanks. So raise your hand and I'll bring the mic back to you. Anyone have a question? In the back. Always in the back. I was just interested in what you found at OpenAI in terms of using transfer learning to like..."}, {"start": 1761.39, "end": 1792.67, "text": "Solve all of the racing games. What are some of the techniques that you found successful or that you're excited about that kind of let you learn from one area and apply it to others without forgetting what you've already learned? Yeah, so the question I think was in terms of transfer, this is a new, yeah, in terms of transfer, what has been effective at OpenAI, particularly in racing games."}, {"start": 1792.67, "end": 1816.9, "text": "So we have tried basically the things that you would guess for transfer within racing. So we've done behavioral cloning, so looking at human demonstrations of people doing racing games as a quick way of getting an agent that is fairly capable."}, {"start": 1816.9, "end": 1848.82, "text": "Trying that on very similar environments. So one thing we did is we just unlocked a ton of different levels of the same game to make kind of like a baby step version of transfer from one racing game to another. And then another thing you can do is then after you've trained with that, kind of use that agent that you have bootstrapped off a human as the starting weights for doing more reinforcement learning. Cool."}, {"start": 1848.98, "end": 1887.6, "text": "Any other questions? Oh, are you allowed to ask questions, Alexey? Thank you, Tom. This is great for developers to play. Can you mention what does it take for an average developer to start, right? So you have GitHub, you have a readme file. What is the process like of engagement with the gym? Cool. So the question is, for an average developer, what is the process to get involved with gym?"}, {"start": 1887.6, "end": 1915.62, "text": "Yeah, so I actually got into AI only like a year and a half ago. So I was asking that exact same question then. And basically, Greg Brockman has a post on Quora that kind of I followed the instructions for, which were basically first taking Andrew Ng's course online to give you a basic background in ML."}, {"start": 1915.62, "end": 1944.5, "text": "Then taking Karpathy's course on deep neural nets. And then basically like playing around with some of the simple gym environments. I basically followed John Shulman's deep RL course and started out making like a kind of like throw everything in a dictionary Q learner without any deep learning. And there's simple enough tasks where if you're doing an environment that only has a few thousand states that that will actually work fine."}, {"start": 1944.5, "end": 1977.78, "text": "And then when you scale up, you can take out the part that's looking stuff up in a dictionary out and replace it with a deep neural net. So I think just, like, basically some combination of getting your hands dirty and seeing where you're stuck, plus, like, online courses can get you there quite quickly. Cool. Oh, one more over here. So for the games, the notion of a reward is pretty straightforward. It's, you know, it's like the score or whether you get first place and so forth. But I'm curious to know, like,"}, {"start": 1977.78, "end": 2007.28, "text": "For the web browser tasks, what do you define as a reward? And how do you see that extrapolating across other tasks that are more complex that people are doing with computers? So the question there is with a racing game, for example, the score is very straightforward. There's just like a score on it. Whereas if you're asking someone to book a flight or something on a web browser, that's a much harder task."}, {"start": 2007.28, "end": 2037.15, "text": "Task to get a reward for. I think that's definitely true. So I think one thing that you can look at there is the density of your rewards. So with a racing game, you have a very dense reward where every second or less you get feedback on how you're doing. Whereas if I need someone to book a flight, you only know whether or not they've booked the flight at the end."}, {"start": 2037.15, "end": 2063.28, "text": "So that's one problem. Many of these tasks that we've built out are kind of simple form filling tasks. So we can, I guess two things there is one thing is that the just denseness of rewards thing is something that there's various different techniques that you can get around. It's definitely easier to have"}, {"start": 2063.38, "end": 2092.16, "text": "To quickly train an agent that is in an environment with dense rewards. We've also had success with training agents in environments that have less reward. Where you basically can evaluate how well it's done, but it's harder to know how well it's doing second to second. And then for these particular web browser tasks, you can check them out if you Google for mini-WOB, mini-WOB. Andre Karpathy has a post about"}, {"start": 2092.16, "end": 2123.71, "text": "How they work? They were hand-built, so we wrote a bunch of JavaScript to make these different environments where they generally have clearer rewards than just a random do this thing on the internet task that you'd find in the wild. Can I also ask a question? Yeah. Would it be possible to use this exciting thing, Jim, in the real world for racing cars? Yeah. Building an environment ourselves?"}, {"start": 2123.71, "end": 2154.34, "text": "Observing the world with a camera and then implementing the algorithm. Right, so the question is, can we have gym environments that are interacting with the real world? And I think broadly the problem there is it's difficult to reset the real world. So if you can set up an environment in the real world where you're capable of resetting it, then that's fine. There'd be a human who would reset it."}, {"start": 2154.34, "end": 2185.66, "text": "Yeah. So that's actually something that I don't think is that crazy. The particular thing, what's that? Yeah. Well, so the particular way that people do that is, let's imagine that you're trying to train an agent to drive a car from point A to point B. Or let's say you're trying to get an agent to stack a block on top of the other one. If you run it in the simulator,"}, {"start": 2185.66, "end": 2220.27, "text": "You can make a gym environment that has a very close to reality simulation. And if you just run it in the simulator, you can train it to be quite good at this. And if you then have something that works with transfer, you can basically have it learn in the simulation. And then if all goes well, hopefully it has learned how to act in the real world as well. Sure. OK. Thank you. Awesome. Thanks so much."}]}