{"url": "https://cdn.simplecast.com/audio/240106/24010659-6efc-4b99-8a40-b6a4a3197753/b979e67a-714f-48d4-b970-3b12d059e675/356673596-the-rsa-rsa-presidents-lecture-the-technologists-dilemma_tc.mp3?aid=rss_feed&feed=oFb7TFSC", "page_url": "https://www.thersa.org/events/", "title": "RSA President's Lecture: The Technologist\u2019s Dilemma", "show_or_channel": "RSA Events", "date": "2017-11-16", "duration_s": 3370, "type": "talk", "speakers": ["Mustafa Suleyman"], "confidence": "high", "notes": "RSA President's Lecture 'The Technologist's Dilemma' (full event audio). The 7-min RSA clip 'Has Technology Lost Society's Trust?' is gASockPXtSE; full video xqYqWFksBeQ (embedding blocked, not verified).", "people": ["mustafa-suleyman"], "id": "2017-11-16_rsa-events-rsa-president-s-lecture-the-technologis", "audio_duration_s": 3370.344, "worker": "jc-w15", "transcribe_s": 38.2, "model": "large-v3", "segments": [{"start": 0.82, "end": 29.62, "text": "Welcome to the RSA Events Podcast, the home of world-changing ideas and debate. Your Royal Highness, ladies and gentlemen, it's my very great pleasure to welcome you all this evening on the occasion of the RSA President's Lecture, one of the highlights events in the Society's annual calendar. I'd also like to welcome those fellows and others who are joining us on the event via the live webcast and invite you to take part in the discussion on Twitter using the hashtag"}, {"start": 29.62, "end": 56.82, "text": "RSA Tech. This year, we are delighted that Mustafa Suleiman will take as his president's lecture theme one of our most pressing current challenges. How can we ensure that the rapidly advancing power of technology is channeled towards the common good? Matthew Taylor, the RSA's chief executive, will join Mustafa for the discussion after his opening keynote speech. And we're looking forward to hearing your thoughts in the Q&A."}, {"start": 56.98, "end": 74.38, "text": "We're especially pleased to hear from the staff for this evening as the RSA from its very beginnings has always sought to balance encouragement for technical innovation and creative progress with a concern that advances in human achievement must be pursued for the benefit of society as a whole."}, {"start": 74.74, "end": 101.95, "text": "You will find that that concern underpins our current action and research work from inspiring the next generation of socially responsible innovators via our student design awards to ensuring that our children receive an education that will allow them to thrive as creative and critical thinkers in the new digital age so that more of them can enjoy meaningful and fulfilled working lives. So we're very much looking forward to hearing from you this evening, Mustafa."}, {"start": 101.95, "end": 139.06, "text": "But before we do, it is my great pleasure and honour to ask you to join with me in first welcoming the Society's President, Her Royal Highness the Princess Royal, to formally introduce this year's President's Lecture. Thank you very much. Good evening, ladies and gentlemen. It's a pleasure to join you on this President's Lecture evening and to introduce our distinguished speaker, Mustafa Suleiman. Thank you very much for joining us."}, {"start": 139.15, "end": 168.43, "text": "I think this is a classic area in which the RSA likes to get involved, and I think my father would have enjoyed this particularly, as the RSA has long been involved in bringing about a world where humans' creative capacities are joined with that of how to deliver 21st century enlightenment. I'm particularly interested in the potential for technology to drive the greater connectivity"}, {"start": 168.43, "end": 196.98, "text": "Public services to deliver power to people and places. And how often do we all in our various environments say, ah, technology may have the answer, not quite knowing what that might be, but kind of feel that it's out there somewhere. And we've become increasingly focused on the impact of the major technological advances, including robotics and artificial intelligence, and where that might be."}, {"start": 196.98, "end": 227.04, "text": "Best used, not just because it exists. Last month, for example, we published a new report assessing the impact of artificial intelligence and robotics on the workplace and how society can harness these technologies to support the RSA's vision of good work. And the RSA will also be working with DeepMind on a new project on the topic of AI and ethics, which neatly brings us to our speaker."}, {"start": 227.04, "end": 257.57, "text": "This evening, who is co-founder and head of applied artificial intelligence at DeepMind, where he is responsible for integrating the company's technology across a wide range of products. In February 2016, he launched DeepMind Health, which builds clinician-led, patient-centered technology in the NHS. And he's been doing this already for a while, in spite of the fact it was only the other day he was 19."}, {"start": 257.57, "end": 284.4, "text": "And decided to leave Oxford to help set up a telephone counselling service. And since then has made really rapid progress, helping to start Rios Partners, a global consultancy specialising in designing and facilitating large-scale, multi-stakeholder change labs aimed at navigating complex problems."}, {"start": 284.59, "end": 313.12, "text": "And that's the bit we're really interested in, is how do you tie these things together and make them work? And we're looking forward to the way in which Mr Suleiman uses his experience to explore the ways in which technology is currently developed and developing, and how technology companies, ever more implicated in the key social institutions we value, how do they remain..."}, {"start": 313.12, "end": 338.67, "text": "It's a huge area of work, and we know that it's changing all the time. We very much look forward to what Mustafa Suleiman has to say with his experience, and we welcome him to the 2017 RSA President's Lecture, The Technologist's Dilemma."}, {"start": 352.5, "end": 382.19, "text": "Well, thank you very much for that very kind introduction. It's a great honor to be here. The RSA is obviously an organization which I think is doing some of the most interesting current thinking around tech's role in society. And as Vicky mentioned earlier, I think it's unique in that it can draw that line of intellectual and ethical engagement right back to its origins in the age of enlightenment when the society was formed for the purpose of supporting innovation for the public good."}, {"start": 382.45, "end": 396.56, "text": "So it really is a very special privilege to be here in the great room this evening, a room that has witnessed so many vital debates about the direction of human and social progress, and reflect on the moment that we currently find ourselves in."}, {"start": 396.82, "end": 425.65, "text": "Today I want to talk about the technology industry and some of the challenges that I think it faces. I'll start by explaining a little bit about why I got into technology, spend some time then describing three asymmetries that I believe have shaped the industry over the last 15 years, and then propose a few meaningful changes to the sector as we know it. About eight or nine years ago now, after working in government, in think tanks and in the charity sector, trying to tackle our most intractable social challenges,"}, {"start": 425.65, "end": 451.95, "text": "It became clear to me that the complexity of our social systems was far outstripping our capacity to intervene. I'd just finished a year-long piece of facilitation work at the Copenhagen climate negotiations, where we were collectively struggling to understand the very basics of the issues before us, let alone make sense of the causes of the crises ahead, or even consider or imagine potential solutions."}, {"start": 452.18, "end": 480.35, "text": "Traditional vehicles for addressing climate change, the various meetings of minds, grassroots campaigning, high-level political negotiations, waiting for spontaneous market-driven outcomes, were, to put it bluntly, just not working fast enough. Time and again, we found ourselves failing to come to grips with a dizzyingly complex world, with groups of the smartest experts struggling to make sense of the relationship between cause and effect. Of course, climate change is just one of many..."}, {"start": 480.35, "end": 509.57, "text": "If we don't tackle these problems, the future of humanity and the planet is at best uncertain. At worst, it's an extremely grim prognosis. And yet we are struggling to innovate within the confines of a political and economic system that grew out of a different age and I would argue needs radically rethinking. If we're going to address the world's greatest needs,"}, {"start": 509.57, "end": 537.25, "text": "We need new kinds of institutions that combine the rigorous and expansive thinking we most often find in academia with the social justice ethic of the nonprofit sector and the speed of execution and scale of resources that we most often find in companies. And that's why I started DeepMind. I wanted to build an organization with the capacity to keep pace with a changing world and make a real impact at scale with an ethical mission at our heart."}, {"start": 537.25, "end": 563.09, "text": "And the resources that we need to invest in the very long term. Our organization is dedicated to boosting our powers of comprehension and cognition to help solve society's most pressing challenges. That to me is the true value of distilling what has made us so effective and unique as a species, our intelligence, and trying to recreate that intelligence or at least those capacities in a digital world."}, {"start": 563.44, "end": 580.43, "text": "But frankly, all of that is currently a long-term aspiration. Today, it's also clear that technology is losing society's trust. Of course, many technologies begin with this altruistic and arguably naive mindset at the outset."}, {"start": 580.59, "end": 607.01, "text": "This trope is so ubiquitous that the sitcom Silicon Valley has made a running joke of claims made by founders, just like me, that their inventions will, quote, make the world a better place, however insignificant or incremental they actually are. Despite the fact that five of the ten most valuable companies in the world are actually technology companies, most of us still want to see ourselves as plucky upstarts trying to change the world for the better."}, {"start": 607.01, "end": 633.36, "text": "But the truth is that our good intentions, initially captured by these well-meaning slogans, are now met with increasing unease by commentators and, of course, the public. I want to be clear that this is not a critique of purpose-driven businesses. In fact, I genuinely believe that these types of organizations will be key to our future. And I also want to be clear about the sincerity of the motivations of the vast majority of funders"}, {"start": 633.36, "end": 661.04, "text": "And founders and execs I've met over the years, these people do really want to make a difference and do the right thing. So is this just a perception gap or is there something far deeper at work here? My experience as a co-founder of a tech company is that there is a genuine challenge here. It's not simply a matter of tech companies better informing the public about the good that they're trying to do, although that's certainly needed. Rather, I think this skepticism on the part of the public..."}, {"start": 661.04, "end": 691.12, "text": "must be heard as an urgent wake-up call for technology companies. It's time for us to deeply reflect on the tech industry's role and think about how we're going to put our principles into practice, not just as individuals, but as institutions formally. So I propose that there are three problematic asymmetries that shape how technology companies currently interact with the public. And it's these asymmetries that have contributed to the deterioration of public trust in tech."}, {"start": 691.12, "end": 719.41, "text": "And should now form areas of urgent action for us all. First, there is an asymmetry between people who develop technologies and the communities who use their products. If you look at Silicon Valley, salaries are twice the median wage for the rest of the US. What's more, the technology industry's employee base is unrepresentative of broader society when it comes to gender, to race, and very importantly, class. For reasons that include cultural bias,"}, {"start": 719.41, "end": 742.38, "text": "The cost of living and discriminatory hiring and management processes. A first of its kind study released by the Kapoor Center earlier this year on why people leave technology reports that 40% said unfairness played a major role in them leaving and 78% of employees' surveys said that they had at some point been treated unfairly in their careers."}, {"start": 742.54, "end": 768.37, "text": "This included being passed over for promotion, stereotyping as well as bullying, humiliation, and sexual harassment. Now the recent spotlight on these issues has meant that more people are woke to the need for workplace culture to be addressed, but these underlying inequalities also make their way into the companies in other more insidious ways. Groups representing more diverse communities, and often the public interest,"}, {"start": 768.37, "end": 798.54, "text": "Usually don't have a seat at the table when it comes to making the important decisions that influence the direction of product development in the big tech companies. As a result, employees at tech companies are largely detached from the day-to-day realities of the people they are meant to serve, meaning that unintended harms and biases can often come built into the products we ship. Second, there is an asymmetry of information regarding how technology works and shapes society."}, {"start": 798.54, "end": 820.24, "text": "Congressional hearings on the Russian interference in the US election show even people in the highest strata of government have little understanding of the novel ways in which technology is being used to shape our world across the board. It's pretty clear that we as a society are only just beginning to realize how much these digital systems deeply influence our daily lives."}, {"start": 820.3, "end": 848.3, "text": "Even if companies provided their code publicly, our civil society and regulatory institutions often lack the technical expertise that's needed to verify the claims that are made by the industry about their own algorithms and systems. It's a genuinely hard problem to attempt to close the gaps between what companies are already building, how to anticipate and of course direct the potential future effects on society, and then how to engage"}, {"start": 848.3, "end": 879.2, "text": "And of course, explain all of that to the public. Thirdly, there is an asymmetry of motivation between market-based incentives and the societal goals that we all, of course, aspire to. Money and growth cannot be the only arbiters of success. And we can no longer assume that relentlessly driving for efficiency will necessarily generate sufficient societal benefit to justify that modus operandi. The standard metrics we use to measure progress and achievement."}, {"start": 879.2, "end": 907.62, "text": "Investment round valuations, number of active users per month, revenues, only reflect the fiduciary duties of companies. They can often pay insufficient regard to the societal responsibility that comes with changing the world. And since board seats are of course occupied by investors, and competition is the name of the game, companies can slowly inch towards questionable practices to aggressively drive up user acquisition, sometimes without sufficient regard."}, {"start": 907.62, "end": 937.65, "text": "to the longer-term societal consequences of those actions. The structural legal incentives are simply too tempting to resist growth at all costs, to reach and to sustain the coveted unicorn or billion-dollar status that many startups are pursuing. So we have an asymmetry of representation. The people who use technology and often rely on it are poorly represented among those who are actually making it. We have an asymmetry of motivation."}, {"start": 937.65, "end": 964.3, "text": "Growth and financial reward are seen as paramount when we desperately need richer and more socially purposeful measures of value with equal status at the decision-making table. And when you combine these with the asymmetry of power, given the ascendancy technology companies have gained over the past decade, suspicion and skepticism are unsurprising to say the least. And now I think good intentions are no longer good enough."}, {"start": 964.66, "end": 994.06, "text": "We have to do much more to regain the public's trust. We must take responsibility for the ethical implications of our work, anticipating the challenges at the start of a project's life cycle instead of grappling with them midway through or worse after the fact. And we need to confront those asymmetries head on, not hope that they will go away. They won't. Tech's impact is clearly enormous. The effect it has on people's lives demands that we do much better."}, {"start": 994.06, "end": 1022.74, "text": "So what might some of these changes look like in practice? Well, first of all, I'd argue that we need to radically widen the circles of influence over how technology is actually built in practice. Diversity and inclusion must be priorities to ensure we don't leave all the decision-making power in the hands of a frankly, predominantly white male population, and in doing so, end up reproducing existing structural biases and injustices in our digital systems."}, {"start": 1022.96, "end": 1046.54, "text": "The underrepresentation of women and minorities at technology companies is clearly an urgent problem. Culture matters, and tech leaders themselves need to take responsibility proactively for trying to break the mould. Technology is not value neutral. I do not believe that it is only up to the people who use our technologies to do so ethically and responsibly."}, {"start": 1046.54, "end": 1068.02, "text": "It's up to us as creators of these incredibly powerful tools, which hold so much promise and so much potential to do good, to do everything in our power to ensure that these systems, by their very design, reflect our highest collective selves. And so addressing our second asymmetry, that of the asymmetry of information, I want to propose..."}, {"start": 1068.02, "end": 1093.42, "text": "That it's our responsibility as technology companies to help create new containers where feedback can be heard and direct those societal insights back into the creation of our technologies. We need to find ways to make new spaces both within and outside technology companies where we can do a better job of designing these new technologies together in a genuinely open, respectful and safe way."}, {"start": 1093.68, "end": 1122.1, "text": "Ethical outcomes in tech depend on far more than algorithms and data. They depend on the quality of our societal debate and genuine accountability. If we are to thrive as a civilization over the next two, three, and four decades, we have to create institutions which are not working to sidestep public criticism, but are actually designed to embrace the clarity and accountability that constructive, nuanced,"}, {"start": 1122.1, "end": 1148.5, "text": "And this requires facilitating genuine understanding of our work and its impacts and allowing all corners of society to engage in difficult, messy, but above all, honest conversations about what is needed from technology. The fear of being misunderstood and of being attacked unfairly in the media has in part contributed to the echo chamber of technologists."}, {"start": 1148.5, "end": 1177.7, "text": "Sometimes only talking to each other instead of talking to the people who might otherwise change our world views. I can't emphasize this enough. We cannot allow our institutional prejudices to continue to dominate meaningful ethical conversations. We must move beyond the default sector-based way of thinking where civil society organizations criticize governments for not listening enough and governments criticize companies for opposing new regulation."}, {"start": 1177.7, "end": 1202.1, "text": "And everybody plays the old institutional roles that they're stuck in, resisting everybody else. This is the way of the old world. As the nature of work evolves, as the means by which we produce, exchange, and consume knowledge explodes, as automation becomes seamless and ubiquitous, and as all these technologies make our world smaller and in many ways more visible,"}, {"start": 1202.1, "end": 1226.64, "text": "Now is the time for us to deeply reconsider the institutional role play that for so long has dominated our world order. That's how I believe we'll ensure these systems can be designed to be fair and just, and that's how we'll effectively govern and control them, and that's how we'll ensure the benefits are distributed widely and fairly. We urgently need to experiment with new forms of organization."}, {"start": 1226.9, "end": 1246.45, "text": "One example that we have been working on is the partnership on AI to benefit people and society, which is an attempt to build a new multi-stakeholder forum charged with bringing together industry competitors, academia, and civil society to discuss the key ethical issues that arise in developing and deploying machine learning systems."}, {"start": 1246.77, "end": 1261.52, "text": "The organization is set up to examine a wide range of research themes, including questions of bias and discrimination in algorithms, safety and robustness of machine learning systems, and the impact of machine learning on automation and labor, amongst many other topics."}, {"start": 1261.71, "end": 1289.2, "text": "It was written into the constitution of the organization that the decision-making board must always have equal representation of both corporations and non-profits. And the board now includes representation from the ACLU, the MacArthur Foundation, a former chief economist to President Obama, as well as all the big tech companies. My hope is that the fabric of technology production will begin to more accurately reflect the fabric of society itself."}, {"start": 1289.2, "end": 1318.59, "text": "In fact, as I'm trying to persuade you today, it's vital that we start to make it a top priority for technology companies to proactively expose themselves to meaningful scrutiny and accountability. A fair world will not emerge by accident. In addition to these changes, I also believe that we need new technical solutions that enable a much wider range of stakeholders to have more visibility around what is happening to information that is handled within these digital platforms."}, {"start": 1318.59, "end": 1348.4, "text": "There's been some progress on this front in recent years with Google and Twitter, amongst others, having launched annual transparency reports to help civil society and the general public get more information about sensitive issues like surveillance and censorship. But there's much more the industry can do. End users, service providers, contracting organizations, and technical auditors should each be able to better understand who has had access to their information, for how long, and under which authorized policy."}, {"start": 1349.01, "end": 1378.26, "text": "These logs, which essentially describe the detailed interactions of human users with data and other technical systems with data, could potentially be useful when making significant public policy decisions. At this point, we're all very familiar with the promise of the Internet of Things and smart cities, but in order to actually deliver these benefits, we need a far higher degree of technical security and, crucially, verifiable transparency."}, {"start": 1378.51, "end": 1404.4, "text": "One piece of research that we've been working towards is a mechanism that we call verifiable data audit, which we believe could help us to collectively better manage the more sensitive kinds of information we now produce. We plan to release it as an open source tool that creates a cryptographically verifiable log of all the interactions that have taken place with a specific data set."}, {"start": 1404.4, "end": 1432.08, "text": "Which I think can help create real accountability between the organizations using the data and the individuals or groups who are actually being served by the use of that data. Beyond data transparency, several leading nonprofit organizations are also developing new ways to make the impacts of algorithms themselves more easy to understand so that people can anticipate and ultimately influence or control how technology may affect their daily lives."}, {"start": 1432.08, "end": 1449.46, "text": "For example, MIT Media Lab researcher Joy Boyolamwini and the Algorithmic Justice League have created exhibits at museums in Boston to drive awareness about the deeply disturbing ways that facial recognition algorithms often fail for individuals with darker skin tones."}, {"start": 1449.71, "end": 1477.66, "text": "Researchers at the non-profit Upturn have hosted community conversations in Washington DC along with the police to discuss their study on the effectiveness of police body-worn cameras, including the crucial questions of whether the public can get access to that footage and how facial recognition technologies are being used in those systems. These efforts to me represent some of the more exciting possibilities before us as a society. Machine learning systems"}, {"start": 1477.66, "end": 1502.69, "text": "Present unprecedented opportunities to introduce more transparency, more accountability, and more justice into our everyday lives. I'm personally driven to make these systems reflect the very best that we can be as a species, our very highest humanity. Why shouldn't credit scoring systems be more transparent than a human bank manager's decision which might not always come with an explanation?"}, {"start": 1502.69, "end": 1527.65, "text": "Why shouldn't a company's algorithmic recruiting tool promote greater fairness and equality than we've seen in the past, rather than reproducing the injustices and biases historically? However, if we are to achieve changes like these, we need to think hard about how we start to shift some of the structural and legal incentives that can lead to commercial priorities continuing to dominate."}, {"start": 1527.65, "end": 1555.47, "text": "Only using metrics like investment round valuations, active users, and revenues to measure success has, I think, so far failed to deliver the, quote, more humane and fair civilization promised by optimists like John Perry Barlow, who authored the 96th Declaration of the Independence of Cyberspace. You only have to spend a day with a startup in Silicon Valley to understand how deeply the motivation to drive up these metrics affects founders and funders alike."}, {"start": 1555.57, "end": 1584.51, "text": "Getting any idea off the ground obviously requires the promise of future profitability and convincing somebody to buy into that possibility. It takes a single-minded, relentless focus which leaves little practical room for working on the thornier realities of the societal externalities produced by this race to growth. To some extent, the culture this creates can lead to a belief that objections are there to be overcome. Naysayers are there to be ignored."}, {"start": 1584.51, "end": 1614.1, "text": "Existing rules and standards are there to be, quote, disrupted. And all because you're always just one step away from going bust. Combine that with investor-led boards that are legally mandated to focus on financial return rather than societal accountability, and resisting the growth at all costs pressure becomes, thanks to the system, much harder than it should be. And that's partly why we see some of the world's brightest minds gravitating towards the simplest ways to get the most cash in the door."}, {"start": 1614.1, "end": 1642.58, "text": "We create messaging apps when more than 800 million people go malnourished on the planet each night. Why is it that anyone can order a pizza on their phone, but half a billion people still don't have access to clean water? Why is it that we can go on a date with a stranger we meet on an app in just a few minutes, but nurses and doctors carrying out life-saving treatments still use pages and fax machines to communicate with one another in the NHS today?"}, {"start": 1642.8, "end": 1666.9, "text": "We have to figure out new ways to incentivize startups and technology companies to prioritize tackling these kinds of societal challenges. The private sector must bring the same innovative drive that has created so many amazing new products and services over the decades to the modern challenge of designing systems that are ethical and are accountable. And there is clearly room for far more innovation here."}, {"start": 1667.54, "end": 1690.54, "text": "I believe we need to work towards a world where companies create business models that are connected directly to improving measurable outcomes and generating much better social impact, rather than just on shipping product for its own sake. It's worth us really taking seriously the promise of value-based pricing models and reconsidering the way monetization and the business models in tech largely work today."}, {"start": 1691.09, "end": 1716.86, "text": "I genuinely believe that new scientific breakthroughs driven by advances in machine learning may well make the crucial difference in helping us to prosper in this increasingly complex world in the coming decades. I think that these breakthroughs will one day enable us to fundamentally reimagine the world we live in, helping us understand and respond to our toughest challenges from climate change to malnutrition, from curing complex diseases to addressing discrimination."}, {"start": 1716.86, "end": 1746.75, "text": "These are the reasons that I am excited about AI. So although there are many, many challenges, my hope is that these kinds of breakthroughs will one day give us the tools we need to make real progress towards solving our toughest social problems and enabling a fairer and more just world. There are now many organizations, including the RSA, convening events and workshops where you can learn about the field of AI, the rate of progress, the types of applications that are already being deployed, and of course the ethical challenges that arise. For example,"}, {"start": 1746.75, "end": 1768.37, "text": "The RSA will be running a series of citizen juries on the use of AI in criminal justice and democratic debate, and these will use immersive scenarios to help participants understand the ethical issues raised by AI in this context. The Royal Society will be running a series of public lectures in 2018 on how AI may affect people's daily lives and companies and organizations and public institutions and how they should look to respond."}, {"start": 1768.75, "end": 1796.46, "text": "The AI Now Institute, recently founded at New York University, has just released its second paper on the state of AI, with 10 recommendations on how the field can deploy and manage AI systems more responsibly. And I think it's well worth reading for anyone interested in the field. And of course, you can always drop us a line at DeepMind Ethics and Society, the new research unit that we launched a couple of months ago. We send out periodic updates on our events and initiatives, and we're always looking for new opportunities to collaborate. So please do get involved."}, {"start": 1797.2, "end": 1822.59, "text": "Of course, changing how the industry has traditionally designed and deployed software is not going to be easy, but this stuff is just too important for us to keep putting off. I believe that every one of us in this room and every single person in society has the right to participate in the design of systems that are currently used by billions of people each day and clearly have such a profound impact on our daily lives."}, {"start": 1822.59, "end": 1849.46, "text": "We must work together to raise the standards and reject the lazy compromises when it comes to the longer term consequences and the social responsibilities that come with deploying these kinds of technologies. With rigorous attention to technology's capabilities, more research into the impact of data that we use as inputs and greater transparency into their workings and with a reorientation of market incentives to include accounting proactively for societal impact."}, {"start": 1849.46, "end": 1877.2, "text": "I believe that we can identify new ways to break through the complexity that makes our problems in the world so seemingly intractable. It is in this collaboration between people and algorithms that incredible scientific and social progress lies over the next few decades. If we can deploy these tools more broadly and more fairly, fostering an environment everybody can participate in, we have the opportunity to enrich and to advance humanity as a whole."}, {"start": 1877.58, "end": 1912.32, "text": "Let's take this chance to shape our world together and try to make this a reality. Thank you very much for your time. Mustafa, thank you for that. I felt it was a very significant and brave speech. In fact, as I was listening to it, I thought this is one of those speeches that's going to be quoted back at events I'm going to be in future. So thank you. We're proud to have hosted it here. I should say..."}, {"start": 1912.32, "end": 1940.83, "text": "Due disclosure that I am the ethical board that oversees the work of DeepMind Health, a role I'm proud to take. And we might look at these kind of regulatory issues in a minute. Can I just ask you one question, however, before we get into the meat of your lecture? Just around the technology itself, just give us a very quick update. In terms of where you thought AI machine learning would be three years ago and where it is now, is it kind of ahead of your expectations, on your expectations, or are..."}, {"start": 1940.83, "end": 1970.77, "text": "There are things that you thought you'd have solved that you haven't been able to solve. I mean, so I think lots of things are true in that context. First of all, I actually do believe that the rate of change is increasing beyond our original expectation. Our systems are definitely getting better faster and they're driven by a whole series of underlying systemic realities. We have access to far more data than we ever anticipated. We have access to far more compute at lower cost."}, {"start": 1970.77, "end": 1992.77, "text": "Our algorithms are getting more efficient, so they're able to deliver the same output with less computation, which means that the search space of possible combinations of algorithms is actually now smaller. We can try different combinations of algorithms together systematically. So in that respect, I think the underlying sort of context has enabled algorithms to be developed much, much faster."}, {"start": 1992.77, "end": 2014.46, "text": "On the other hand, I do think we're many, many, many decades away from surpassing human intelligence. I think what's clear is that we're going to be augmenting and enabling human intelligence for many decades to come, and that's the bit that I'm most optimistic about. Do you think the speech you made..."}, {"start": 2014.46, "end": 2030.85, "text": "It's the kind of speech, although not as eloquent as your speech, I say, we've started to hear from people leaving the industry, reflecting back on it. You've taken the brave step of making the speech while you're still a leader within it. Do you think the sector as a whole gets this? Look, I think..."}, {"start": 2030.85, "end": 2049.68, "text": "One thing that I've realized over my career is that it's really important that we don't get stuck in our institutional roles. As I sort of tried to make clear, I don't want to be seen as a technology person. I am first and foremost someone with principles, who cares about politics. I have strong values."}, {"start": 2049.68, "end": 2079.31, "text": "And they are part of my identity as someone who also uses technology to try to do useful and interesting things in the world. And so I think we should resist the temptation to silo one another as the policy person or the government person or the academic. The sooner we start to share those roles and responsibilities, we'll feel the struggle of trying to explain when we criticize one another and empathize a little bit more deeply and more sensitively with respect to how complex the answers are. There are no clean solutions."}, {"start": 2079.31, "end": 2108.74, "text": "Anyone who points a finger and says, if only you did this, everything would be fine, is reducing the complexity of the reality we face to something that is just a useless narrative. And so we sort of have to sit in the fire, if you like. I mean, there are clearly lots of things about the industry that I struggle with, and I am clearly trying to bring a different perspective to how we operate. You talked a bit, we talked a lot in the lecture, about the way in which, in certain ways, the sector is compromised. But of course..."}, {"start": 2108.74, "end": 2135.01, "text": "We as citizens too are compromised. So if you want to read kind of corrugating critiques of what Facebook has done, there's no better place to do it than Facebook. So it is in a sense our dependence on the technology, the way in which we're compromised in terms of our ability, as it were, to challenge the sector. Because as you're kind of implying, the sector does need external challenge as well as an internal ethical awakening. I think that's right."}, {"start": 2135.01, "end": 2161.76, "text": "That's why I'm trying to frame this in a relatively urgent way. Think about it. Ten years ago, we didn't even have mobiles, let alone access to Facebook in the ubiquitous way that it is now. In ten years' time, we will have become so accustomed to the ubiquity of artificial intelligence in our daily lives, making decisions about sentencing and parole, making decisions about credit allocation, making decisions about who we connect with, what products we buy."}, {"start": 2161.76, "end": 2185.25, "text": "This will feel seamless. It already is. And so that's why I think there's urgency. There's urgency that we, right now, start to think about new models of governance and oversight and proactively hold those systems to account. But when it comes to other products, we can simply go to a different shop, or we can choose not to buy clothes from that store, or we can even watch a different television channel."}, {"start": 2185.25, "end": 2206.13, "text": "92% of us use the same search engine. 2 billion people on Facebook. So is there a sense in which it's very hard to see how it is you can challenge such corporate power? And that's also why I mentioned the challenge of growth. I mean, digital systems essentially accelerate the default tendency that we have in the real world anyway."}, {"start": 2206.13, "end": 2236.77, "text": "Measure things in a particular way and optimize for a particular set of objectives. And digital systems just turbocharge that dynamic. And so we should continue to expect that the systems that already have significant influence in our world today will in the future accrue more of that influence. And so that's why I think the challenge is as urgent as I'm saying it is. I'm going to open up in one moment. So please do think of questions that you want to ask, Mustafa. But just one last one from me."}, {"start": 2236.77, "end": 2260.45, "text": "One of the things I took out of your lecture is that technology and the pace of technological change creates huge problems for the way in which we think about regulation. So that's the way the state does things. But also for the traditional model of capitalism, which is what you were talking about towards the end of your lecture, the conditional investment model. So what we're basically saying is technology makes these two kind of twin institutions, the state and the market."}, {"start": 2260.45, "end": 2283.62, "text": "Look, unfit for purpose. So the scale of institutional change, societal change that you're talking about here, I mean, it's epochal, isn't it? We're not talking about a bit of tinkering here. We're talking about the world having to change fundamentally if this technology is to serve the public good. Well, and the real challenge is that it does both of those things while simultaneously driving incredible utility."}, {"start": 2283.62, "end": 2312.02, "text": "So we love the platforms that provide us with information, that allow us to message and communicate with our friends, to connect with a stranger that we've never met before, to order a taxi and have food delivered to our house. Now, that utility is what's driving this accelerated growth. We want to consume it. And in many ways, that's a very good thing. They're delivering, I think, improved products and services in our world. The hard question for us to focus on is what scrutiny and accountability looks like in a meaningful way given..."}, {"start": 2312.02, "end": 2338.98, "text": "Those three trends that we described. So in that sense, we have to remember what Aristotle said, which is that science is the best thing to describe what the world is, but it's not much good for describing the world as you want it to be. And that process of describing the world as we want it to be is something that we need to urgently discuss. Or else the science, or else we'll end up just thinking the science can provide every answer, which it can't. The exciting thing is that these systems are eminently controllable. We have designed them."}, {"start": 2338.98, "end": 2368.54, "text": "They are actually transparent. We can set them to our order. And that's why I say I want them to represent our highest collective selves, not just the way that we do things at the moment or the way that we've done things for the last few decades. These actually represent an opportunity to instill the highest values that we actually care about with respect to justice and fairness. And that's the kind of exciting opportunity before us. Fantastic. Okay, so let's see some questions from the room. And we'll start with..."}, {"start": 2368.54, "end": 2399.33, "text": "The lady there in the middle row, wait for the mic to come to you. If you could give us your name, that would be great. Yes, my name is Lopa Patel. My question is about AI, particularly where AI is no longer requiring human intervention. So the question is, do you think AI needs its own human rights? Didn't Saudi Arabia give citizenship to a robot the other day? It did."}, {"start": 2399.33, "end": 2411.94, "text": "I think there's sort of two questions there. So our systems are becoming increasingly autonomous. And so there is a very real conversation that we urgently need to have about the definition of meaningful human control."}, {"start": 2411.94, "end": 2441.58, "text": "And we have, my co-founders and I have publicly called for a ban on the use of lethal autonomous weapons in the field because I think we haven't got to a stage where we can reassure ourselves that there is meaningful human control, a human in the loop deciding when to pull the trigger, if you like. So that's sort of one position that we take, particularly on lethal autonomous weapons, but I think will increasingly apply as the correct safety standard to many other autonomous systems that sort of operate in our world."}, {"start": 2441.58, "end": 2464.37, "text": "The second question about whether robots need rights, I think this is, frankly, a distraction from the more urgent, practical, ethical questions that I think I'm trying to raise. A lot of the debate over the last couple of years has been around Skynet and Terminator and should these sentient beings have human rights and so on. I think that's an interesting philosophical question."}, {"start": 2464.37, "end": 2495.94, "text": "But we're luckily many, many decades away from that reality. And I think in the immediate term, we have sort of more urgent priorities, especially on lethal autonomous weapons. Yeah. Here comes the mic. Thanks for a great speech, Mustafa. I'm Ollie Boston from Future Advocacy. You didn't say a lot about government and democracy and some of the solutions for some of the great..."}, {"start": 2495.94, "end": 2525.54, "text": "Some of the challenges we have, some of the actors you mentioned were civil society and academics and businesses themselves. And the partnership for AI, which is a great development, is a partnership between businesses and NGOs largely. So I wonder if that's part... Is that because you sort of despair of liberal democracy in our current circumstances with Trump and the terrible things that our male politicians are doing and all the rest of it? Or is there a new model of democracy that's part of this new future?"}, {"start": 2525.54, "end": 2539.25, "text": "There's certainly not a new model of democracy and there shouldn't be. I mean, there is a practical reality, which is that we need, as I said, new spaces to have conversations which don't..."}, {"start": 2539.25, "end": 2558.46, "text": "Cleanly fit in the current legislative paradigm and there isn't quite enough space in that regulatory framework. And I think it will catch up and there need to be sort of changes in that respect urgently. The objective of the partnership on AI is kind of different. It's to create spaces where we can have these..."}, {"start": 2558.46, "end": 2575.47, "text": "Honest, messy, difficult conversations where a technologist can explain the concern that she has about a system that she's developed that is in production which is making a set of recommendations that are not as transparent as we might otherwise like or are delivering a set of outputs that are..."}, {"start": 2575.47, "end": 2608.32, "text": "Based on a training data set which has inherent biases and is reproducing that discrimination. And that is a difficult conversation to have given what's at stake for the technology companies. But it's the right conversation that we need to have in a relatively open forum. And that's the kind of scrutiny and sort of critical accountability that I'm talking about. Thank you. I'm Michael Tanari. So you've hinted at having..."}, {"start": 2608.32, "end": 2637.34, "text": "Almost anti-capitalism sentiments. And I think that's something that I've heard increasingly. And I wonder if you've thought about what society needs to look like. If we move beyond the market economy, what do we have? Or if we maintain a market economy, how do we build in more consideration for the social, the environmental? Thank you. No, I'm not an anti-capitalist. I think..."}, {"start": 2637.34, "end": 2658.48, "text": "I think that one of the greatest drivers of progress has been the motivation to generate and hoard surplus. And the reality is that that has been extremely effective in simplifying the production chain. And that has been a great thing. We've generated a great amount of value from that. And I think we will continue to do that."}, {"start": 2658.48, "end": 2687.95, "text": "That doesn't mean that it's a perfect system. And I've tried to point out some of the weaknesses that drive aggressive growth at the expense of pricing in other externalities. And we've clearly done that, not just with the environment, but ignoring the longer-term social consequences of the kinds of platforms that we build. And there are errors in these systems which propagate very, very quickly because of the scale of their action space. Because they are hitting a billion or two billion users a day, because they are one of the primary sources of our information,"}, {"start": 2687.95, "end": 2708.74, "text": "Thank you."}, {"start": 2708.74, "end": 2738.13, "text": "Propose with equal importance and weight the value of considering the effect on our planet and the effects on people who don't immediately benefit from those systems. And that's the kind of legal structural challenge which I think we need to resolve. Companies are currently incentivized to look at only one side of that equation. And there's a second very important side which can, I think, be worked into a company or a new kind of organization's motivations."}, {"start": 2739.5, "end": 2775.3, "text": "Yeah, there. Hello, Richard Dijkstra. Should we be worried that companies like Fox and News International or whatever are now seeing that they may be too small to compete against the very large tech companies? And then is there also a kind of parallel that if you look back in the 1980s that the Bell company was effectively split up?"}, {"start": 2775.3, "end": 2804.75, "text": "Into about 20 or 30 separate companies to try to get rid of monopoly in the States at the time. I think this is an urgent question that we need to have an open, honest conversation about. I think the parallel is a reasonable one to draw. I think Fox has had its own question with respect to monopoly. And in some sense, I am glad that we have a new era."}, {"start": 2804.75, "end": 2835.07, "text": "What about education, Mustafa? When one talks about technology and education, people immediately start talking about teaching kids to code. What do you think children..."}, {"start": 2835.07, "end": 2860.56, "text": "What are the skills and insights they need most to have for the age that is coming towards us? And then associated with that, engineers themselves, do they need a richer education given the fact that they have this capacity, as you just put it yourself, to be able to do things which have worldwide repercussions incredibly quickly? Yeah, so, I mean, I'm not an engineer. I don't code."}, {"start": 2860.56, "end": 2882.29, "text": "And there is no reason why people should be investing many years of engineering education in order to engage with these systems. There's no doubt that it's helpful. I did philosophy, but my basic understanding of being able to grapple with abstract concepts and learn new ideas quickly put me in a good position to be able to make sense of..."}, {"start": 2882.29, "end": 2896.4, "text": "Abstract diagrams, technical diagrams on the board. It's entirely possible to build and maintain enormous technical systems without having that prerequisite. So I think sometimes that's a little bit overstated."}, {"start": 2896.4, "end": 2914.35, "text": "On the other hand, it's clearly an urgent priority that we rethink the way that we have constructed computer science degrees. So they are far too singularly focused on simply learning the skills of engineering and not considering the important questions around..."}, {"start": 2914.35, "end": 2941.86, "text": "Is my data set statistically significant? Does my data set accurately represent the target that I would like to predict? And they are fundamentally sociology questions. They are political questions. And so what we need is far more interdisciplinary courses that give both sides of the kind of stuck debate what they need to understand one another. And what about the resilience that we need? So if one reads about social media addiction..."}, {"start": 2941.86, "end": 2947.25, "text": "in teenagers and one knows that some of these social media platforms are designed indeed to encourage."}, {"start": 2947.25, "end": 2976.53, "text": "So what do we need in terms of our capacity to understand technology and what it can do to us? Yeah, I mean, I think some of the literature analyzing the kind of social mechanic, the sticky or viral addictive mechanics of liking and sharing, these are clearly incentivized to drive more watch time and incentivize us to spend more time on the platform. And those are incredibly good things. We generate enormous utility from being able to see that"}, {"start": 2976.53, "end": 2987.54, "text": "Thank you."}, {"start": 2987.54, "end": 3010.82, "text": "Where in the filter bubble we currently sit? It's clearly not healthy to only see information and knowledge of a particular kind. And so we need different kinds of representations that help us to understand where we are on that multidimensional continuum of information access. And so we're sort of in the natural order of things. These need to be worked into the platforms as we grow. Great. I think we've got time for... Oh."}, {"start": 3010.82, "end": 3041.34, "text": "Ooh, look at this. This always happens, doesn't it? There should be an algorithm to tell you how many people are going to put their hand up when there's any time for one more question. And what I'll do is I'll take three questions, but make them really, really quick, and then you'll have, like, one minute to answer and stuff. Sorry about that. But, okay, we'll start here with Andrew, and then we'll end up here in the front row. Here's the... Taglines like Google's don't be evil. I think they've supplanted it with do no harm or something now."}, {"start": 3041.34, "end": 3073.04, "text": "Do you think these are kind of passive slogans that fool the public into trusting companies? If lethal autonomous weapons are the best example of where technologists shouldn't unite, what would be the one kind of shared societal challenge that you think technologists should unite around as a kind of shared mission to address, assuming that we all believe that we should solve climate change, hunger, human rights abuses, etc.? And then finally?"}, {"start": 3073.55, "end": 3099.04, "text": "I'm Jibida, hello. I recently got someone to call for not paying my council tax. It is relevant, I promise. But I pay my council tax by direct debit. So everybody that I told this to, they went, oh, that's just automatic, isn't it? It's a computer error, isn't it? And I just couldn't... Computers don't make... I mean, it's so basic. Computers do not make these kinds of errors. So my question to you is, why is that?"}, {"start": 3099.04, "end": 3130.75, "text": "Why are people so negative? And how am I going to sell my robots to people when they're so suspicious? Well, actually, probably my question is more, what are the big companies doing about that to change people's perceptions and change this negative attitude about, oh, it's the computers that are going to ruin society when actually we're quite good at doing it ourselves? Last question loops you nicely. Oh, hang on, one more. Oh, one Twitter question. We ought to do that given it's a technology event. Yeah, so one question from Twitter. What can every citizen do?"}, {"start": 3130.75, "end": 3157.34, "text": "Okay, let's try those four in two minutes. So, you know, don't be evil, I think, is a sign of a very well-intentioned, you know, very well-motivated group of people some time ago now, you know, 12 years ago, trying to encapsulate a willingness to do good in the world in a short slogan."}, {"start": 3157.34, "end": 3185.12, "text": "There is a big culture underneath that. And I think having now spent three years at Google, I've got incredible admiration for the goodwill and genuine good intention that comes with that. And it needs to go further. Things have changed in the last 10 years. And there's lots to say about how that could go much, much further. And we'll see what happens next year. Single challenge."}, {"start": 3185.12, "end": 3206.1, "text": "The single challenge that we as technologists need to address, the most urgent one, I think, is this question of how do we validate that content that has been produced and sits on our platforms is not fake? People always ask me, can AI help with this problem? Unfortunately not. I don't think so."}, {"start": 3206.1, "end": 3228.64, "text": "Algorithms are not yet good enough to distinguish better than human performance what is fake and what is real. And so what we urgently need are more humans in the loop, more meaningful human control who can help us to curate and sift through this information. And I think that's something that's being worked on. And I think your speech was designed to address this issue about public skepticism."}, {"start": 3228.64, "end": 3254.26, "text": "Right, right. I mean, I think public skepticism is healthy. I don't want there to be a world where there isn't public skepticism. I think we urgently need more of it. It needs to be part of our contemporary public narrative to critique and challenge and demand a higher level of accountability of these systems. And only by that measure will we get more comfortable. I need to understand what this thing is doing in my everyday life. That is a very reasonable and fair position. It's a fair request."}, {"start": 3254.26, "end": 3278.03, "text": "Despite the utility, the utility cannot justify darkness. Transparency is the thing that helps us to, I think, manage these systems well. And then finally, to reinterpret that question, I think what you're saying is that in a way, if we are going to get the fruits of technology, we are going, as a human race, to have to develop. We are going to have to reach further than we have done before in terms of our sophistication, our capacity to solve problems, to collaborate."}, {"start": 3278.03, "end": 3298.02, "text": "Right, and I think we're already doing that. We are clearly changing as a collective species because we have access to so much information instantly, and that is already a phenomenal change. But when I talk about us being our highest collective selves, I mean that this is, frankly, an opportunity for us to..."}, {"start": 3298.02, "end": 3317.46, "text": "You know, take the Human Rights Act and really think about what the right balance between those conflicting and competing rights actually should be in different contexts. Because we have the ability to control them. And so provided those platforms are transparent and accountable, we actually do have the capacity to..."}, {"start": 3317.46, "end": 3339.04, "text": "Really deliver a higher standard of fairness and, I think, justice than we have done, I think, in the past when things have been much more opaque and much more difficult to scrutinize. Well, sorry, I'm afraid we've run out of time. It's been a fantastic session. As I say at our speech, I'm sure that it's going to be quoted. Very important."}, {"start": 3339.04, "end": 3368.11, "text": "Speech, a very important moment. Thank you all for coming and for an excellent set of questions. We have an opportunity now to continue the conversation at a drinks reception downstairs immediately below us in the Benjamin Franklin room. Could I just ask you to remain seated until the Royal Party has withdrawn finally. Please join me in once again thanking Mustafa for his terrific lecture. Thanks for listening. If you like this podcast, head to our YouTube channel for inspiring talks, interviews and animations."}]}