[0:00:00] Hello, everyone. I'm worried that this session isn't going to help our online poll because I think confidence levels are going to drop since we're focusing on problems. But for the next hour and five minutes, I think, Nicola, you took some of our time, but that's okay. For the next hour and five minutes, we're going to focus on gaps and challenges when it comes to accountability in our global AI governance space. Probably a lot of different topics will come up in the table, but maybe so I can help center the conversation, I want to make sure we focus on at least five.
[0:00:27] The first one being that we need better coordination to at least move towards better global AI standards. And of course, interlinked with that is what a lot of people talked about, the need for scientific consensus on AI risks. Then the second one, also to have stronger and clearer rules and consequences for those who fail to manage AI risks effectively. The third one, looking at what I think are flawed industry incentives that are driving rapid AI development. Fourth, looking also at power imbalances.
[0:00:57] And as I'm saying this, I'm thinking that this is going to be a lot in one hour, but let's try. So the fourth one, looking at power imbalances and also race dynamics between countries, but also between companies. And lastly, the importance of inclusivity and the participation of currently underrepresented groups, even in this room today. So especially from those coming from the global south. So the session in terms of format is going to follow.
[0:01:21] Mir, the previous one. We're going to have some short interventions by a few participants, and then hopefully an interesting open discussion afterwards. I echo my colleague, Nicholas. I encourage you guys to be bold, kind but bold, right? We want to be answering and asking hard questions here. So with that, we'll first be hearing from Ambassador of Rwanda to Friends, Mr. Francois Nkulikimfura.
[0:01:49] I tried my best. Thank you for joining us today. Thank you very much. So I think actually one of my colleagues, Mr. Fotis, mentioned this earlier. When we talk about accountability, part of that conversation needs to include accountability to whom. And since AI is going to impact everyone at the end of the day, I really think that to whom should also be everyone. It should include underrepresented voices. It should include people from the Republic of Rwanda.
[0:02:16] And the Global South as a whole. So my question to you is, what specific steps would you prioritize when we talk about accountability mechanisms that actually reflect the needs and the realities of the people of the Global South? Thank you very much, Nikki. Can you hear me? Yes. Thank you very much, Nikki. Thank you all. And thank you for inviting me to this important event.
[0:02:51] Exploring how we can collectively address key gaps in establishing accountable AI governance. I'll try to be very brief because a lot has been said in the previous sessions. And yes, AI is a transformative force reshaping industries worldwide, including on the continent of Africa. AI potential was really underappreciated a decade ago, but today its impact is undeniable.
[0:03:20] And for instance, I'll just take two examples. On the agriculture side, AI-driven analytics could increase yields by 30%, adding 26 billion annually to African economies through precision farming. In the health sector, by 2030, AI could save 1.8 million lives annually via early diagnosed and optimized resource allocations. So these advance highly.
[0:03:50] Highlights artificial intelligence, a potential to foster growth and prosperity. However, realizing them requires bridging the persistent AI divide between high income and low income nations. In some of their recent studies, McKinsey has predicted that generative AI impacts on the global GDP could reach approximately 7 trillion.
[0:04:21] My apologies. Your excellency, my apologies. It seems with your computer, the mic is not captating what you are saying. Oh, apologies. On the side. So I have to start again? All right. You heard part of it. All right. So I was saying, yes, it is therefore imperative that we collectively endeavor to use AI to improve human capabilities safely.
[0:04:59] Equitably, I think some of you mentioned it earlier, while fostering inclusive growth and shared prosperity. So on the side of Rwanda, Rwanda's national AI policy is that the cornerstone to our strategy to harness AI for economic growth and improve quality of life. It is focusing on six key areas. Number one, building 21st century skills and promoting AI literacy.
[0:05:29] The second one is related to strengthening infrastructure and computer capacity. Next is developing robust data strategies. Probably our last one is establishing ethical guidelines for responsible AI systems. So this policy aligns with Rwanda's ICT-driven socio-economic agenda and highlights our commitment to ethical, inclusive AI deployment. So in Rwanda, maybe I'll just...
[0:05:59] Let's move a bit on Render Prioritize International Partnership to advance responsible AI governance. I'll share two or three examples. The first one is an AI toolkit developed with the Commonwealth Secretariat and the Intel, helping nations regulate AI risks while sizing its opportunities. There's also an AI playbook for small states. And of course, the Global AI Summit for Africa,
[0:06:31] Amplifying Africa's voice in shaping AI governance grounded in our unique context. To bridge the global digital divide, it is crucial. It is crucial to, one, localize AI particularly by incorporating African context, and two, include diverse voices in global AI governance discussions, ensuring Africa's benefit.
[0:07:00] Equitably from AI-driven economic growth. Rwanda supports the African continental free trade area as a driver of economic transformation. As I conclude, dear colleagues, as government of Rwanda, we are committed to inclusive and responsible AI development as we strive towards becoming a high-income country by 2050. Globally, we advocate for collaborative governance to bridge.
[0:07:30] Divide and unlock AI full potential for humanity. We stand ready to partner and provide a testbed for innovative AI solutions that promote global economics, growth and equity. I thank you for your kind attention. Thank you, Ambassador. And I'm sure during the open discussion, we'll go back to the issue of inclusivity further.
[0:07:58] Let's now go to our colleague, Jack Clark. I think many of us in this room already know Jack. He's the co-founder and also the head of policy at Anthropic. He's also the brain behind Import AI, which a lot of us get our AI news from. So Jack, I actually have two questions for you. The first one, given the pace of AI development, what incentive systems, and please let those be unconventional, they don't have to be the usual ones, but what incentive systems would actually force
[0:08:24] Thank you very much. I'll start with optimism. I was here in end of February 2020 talking about
[0:08:49] Large language models. We just released a now small one called GPT-2 and we're talking about many of the same issues. But at the same time, something else was happening. Many ministers were here and they were getting text messages about cases of something called COVID, which was beginning to happen in their countries. And there was a strange air to the proceedings. And in the subsequent years, governments did succeed at doing something ambitious and unprecedented.
[0:09:18] You know, we're all sitting here without masks because of sustained massive government action to deal with a kind of surprising attack from the future which had not been anticipated. And I think that there are real analogies here. You know, my colleague and co-founder Dario recently wrote that by 2026, we expect that powerful AI systems will be capable of kind of Nobel Prize winning performance on many different...
[0:09:48] Scientific benchmarks. And they will be able to carry out large, kind of long-scale tasks for us in almost arbitrary domains. He compared it to a country of geniuses inside a data center. And this is not kind of marketing. It's what we believe. And to your question of incentives and to the discussion of power, do you want a system of that capability to be centralized within a company or a small set of companies?
[0:10:18] Absolutely not. That would be completely crazy and would be an affront to democracy. And so the question for us here is how do we actually change the incentive system so that we can give the public a vote on these systems, so that we can give governments the ability to govern these systems and the companies that build them. And I think that there are kind of three basic things which compound on one another.
[0:10:44] First is getting really clear on what the transparency requirements are on the companies developing these models, not just about your capabilities, but about the risk assessments that you're conducting and the kind of security and safety protocols you follow. You do not trust these companies, though, and you shouldn't. The second part is that you need to run third-party tests and validation of what they claim. You need institutions that test.
[0:11:15] Validate and in some cases audit us, the companies. This is probably key to public trust. And finally, you need to build an international network that can coordinate about this. And through that coordination, you create the possibility of hard power governance interventions. Because if companies are able to
[0:11:41] I was here four years ago, and since then we have built the network of international AI safety institutes. We have built early testing capacity.
[0:12:09] Despite my English dour nature, I am more optimistic than I was a few years ago. But I think that the stakes are extraordinarily high. And just to finish, I believe ultimately this whole question of opportunity and risk is also bound up in government capacity. Of course, we talk about risk and that's important, but I believe to recognize many of the true large-scale civil benefits.
[0:12:39] Governments are going to have to get better at prototyping and using this technology so citizens, when they ask, what has AI done for me lately, have profoundly good civil uses of the technology as well as those from companies. Thank you very much. Thanks, Jack. And building from that, I think many of us in this room would argue that accountability on its own would fail without actual binding rules. So I'll go to Mr. Brando Benefe. Hello.
[0:13:12] I think the EU AI Act offers a lot of valuable lessons for all of us when it comes to AI regulation worldwide. So I'm wondering if you can elaborate with us in this room how you see the AI Act ensuring that different AI actors, especially developers of general purpose models, can be accountable and also looking forward if you see gaps in that framework as a legal institution. Okay. Yeah. So thank you for this opportunity.
[0:13:41] My name is Brando Benifei. I'm a member of European Parliament, and I was one of the two chief negotiators for the Parliament of the AI Act. In fact, exactly one year ago, we had the long, long marathon of negotiations, in fact, with two people in this room, with Ms. Artigas and Lucila Scioli. And today, so we mark one year from that result. And in fact,
[0:14:09] We tried to think all the time about how we could have an effective governance of these legislation. Now I'm sharing together with another colleague, the implementation working group in the European Parliament of the AI Act. In two days, we will have a very important meeting to scrutinize what the AI office has been doing until now, especially on the code of practice, which I know is especially...
[0:14:39] Important for everyone here. It's a matter of particular attention because we are trying to do something.
[0:14:46] Which to some extent is one of the exceptions to the general spirit of the regulation, which is, as you know, mostly based on use cases. So in this case, we are diverting partially. So we identify some technological results that have systemic risk. This is how we say. So in fact, I would say on this topic, just to have just a few brief.
[0:15:16] Then we have the discussion, but we try to give strength to the AI office. One thing that came clearly is that on different legislation that we had at European level that were also
[0:15:31] To some extent having a global effect, I think of the GDPR for example, we always had the issue of having a non-fragmented implementation, to have legal clarity, to not have lengthy confrontations on how we should implement this legislation in a way that was
[0:15:55] Effective and not dividing between the different member states and having so an issue of legal certainty and an impact on our competitiveness. In fact, this is the topic, the way
[0:16:11] You see also Draghi talks about this topic in his report. At least this is how I see it, how I interpreted it. In the case of AI Act, we try to give a strong role to the AI office and this will be crucial. In fact, we will have discussions with the new commissioner to, in fact, ask for continuing in this direction. We want a strong office so that we have a strong
[0:16:36] EU level governance and not fragmentation. At the same time, we need cooperation of the national organizations.
[0:16:47] The regulatory bodies that we identified to do supervising at national level, but we think that we need a strong European level. Also to play a role in the ongoing debates at the supranational level, OECD is doing an important work, especially on the definition, but also we have the UN that has done its part.
[0:17:15] And we see the continuing work at G7 level on the generative AI. And I think it's very important because I think we need to have both at the level of the global governance, both domestic legislation. We are doing that with the AI Act.
[0:17:31] Domestic legislation with some global spillover for sure. The code of practice as a global spillover effect. I'm sure, but also we need to be sure that we do our part in the international governance. And I think that on some areas, like the generative AI, but I think.
[0:17:49] of building a global corpus of international law around AI that we do not deal with on the AI act. I think of, for example, the security and defense use of AI. We need to strengthen a global governance and a global set of legislation. I conclude saying that I agree with what was said earlier by Ms. Artigas that we do not only should rely only on legislation. We see that we
[0:18:19] Can work on different ways of doing things. In fact, with the code of practice, we are doing legislation through a process that is co-legislated. And it's something where we can see a kind of an experimentation that I think it's needed due to the nature of the frontier technologies we are talking about when we deal with these aspects. Thank you very much.
[0:18:50] Thanks, Brando. To move across the pond, across the roundtable in the setting that we're in here, Ms. Maya Wiley, the Executive Director and CEO of the Leadership Conference on Civil and Human Rights. Welcome. As we previously discussed, the risks of AI are several. They cover systemic civil rights violations and also group-based harms. I'm wondering, especially in the U.S., is...
[0:19:15] You've identified specific federal rules that are missing to protect us from those harms. And there's a lot of civil society voices actually in this room. And I know that the Leadership Conference is a coalition of civil society voices. So as a civil society actor myself, I was wondering if you have advice for us for the next administration, how we can really help tackle those harms.
[0:19:36] Well, first of all, let me just say it's an incredible privilege to be here with you. Just to say a little bit about the Leadership Conference on Civil and Human Rights so you understand the perspective I'm going to share. We are the nation's oldest and largest civil rights coalition. We will be 75 years old.
[0:19:56] In 2025, and we have created a center for civil rights and technology because of the existential threats to racial justice, gender justice, trans justice.
[0:20:08] I could go on and on and we don't have time for the list, but we do represent marginalized communities across a broad, all the swaths of US society and have been working in the area of technology for over a decade. But I say that because right now, and one of the things that's not very popular to say sometimes out loud as a civil rights lawyer and advocate is that United States laws do not actually address
[0:20:37] The bias that we have already been experiencing with the technology, with these technologies, I should say. And that is for many reasons.
[0:20:48] Previous conversation that said we must have regulation, we must have a legal regime, is very much something that we experience in the US. And that is not to the exclusion of standards. We support and agree with standards and ethics. All of those things need to work together. But in the absence, let me just give you a very practical example. In United States law, we look constantly to, and there's tensions in international human rights law as well, around what does it mean to say someone has, or a company,
[0:21:17] or an actor has violated someone's civil rights. Normally we look for bad action or we look for a sufficient statistical analysis that says so many people have been harmed by this of a particular group but in a way different from other groups and if so we will work backwards and say we will find liability.
[0:21:39] We will call that disparate impact. We can call it lots of different things. But the reality then is once we talk about a technology built on a frontier system where no one intentionally said we wish to do harm to a particular group, we already have a weakened system of assessment of bias and discriminatory impacts.
[0:22:00] But then we get use cases on top of that. For example, a use case that actually happened, which was a software company said, we want to make sure we can help people understand how many people, who's more likely to need hospitalization services. And then find 250 million hospital admissions to determine that if you are white and have insurance, you're more likely to need hospitalization. And why?
[0:22:27] Because you're more likely to have insurance to pay for the hospitalization. And so for all the communities of color who did not have the insurance and could not go to those hospitals, who did not show up in the data, suddenly would be erased from the very analysis that there was a need.
[0:22:45] In fact, a disproportionate need for those same services. Now, we could talk about healthcare. We can talk about housing, access to credit, entrepreneurship, education. We can go across the board with examples, and I don't have time for all of them. But I will tell you, we already have and made advancements in the United States with the AI Bill of Rights.
[0:23:08] And an executive order that looked at federal government and across agencies of federal government, how it could behave. NIST has done incredibly important work on standards. But at the end of the day, it is voluntary. We have systems of technology that simply don't fit the civil rights legal structure we have created. And at the same time, talk about culture, we have an attack on the very notion that
[0:23:37] Disparate impact itself, this very concept that we would use right and are using right now to say we must look at the impacts and outcomes on real people, particularly marginalized people. This very analysis is under both political attack and judicial attack in the United States, which means without legislation, we will simply see harms go.
[0:23:59] Unaddressed unless there is simply so much voluntary goodwill and interaction with civil society that companies prevent it. It's not happening now, and we should not assume it will happen in the future. And I don't want to put you on the spot, but I noticed that you didn't give us any tips for how to handle it in the next administration. So maybe though. I will give you this tip.
[0:24:31] It is all about power and power of accountability, but that power is going to reside outside of government. So one of the things we are doing is actively working with companies to say, like Anthropic, to say what will we be able to agree upon that we can set out not just as principles, but as starting to create examples of where and how we have transparency and accountability and the kinds of parameters that ensure we're disrupting inequality rather than reproducing it.
[0:25:02] Great, thank you. Speaking about power, the next person I'd like to invite to speak is Barry Lynn, the Executive Director of the Open Markets Institute. Barry, when we talk about unchecked power concentration in the AI industry, what do you think are the greatest risks? And also a second question for you as well. Yes, yeah, thank you. And it is a great privilege to be here today. You know, a few people have really touched on this issue already, you know, Paul.
[0:25:40] And Gaia and Asma, but concentration of power and control is the key issue when it comes to dealing with AI. Over the last 40 years, we've seen almost every sector of our economy monopolized in various ways. But we see this most dramatically in digital because it was never really properly governed at all and at any point in its history. If we view AI within this context, what we see is that
[0:26:10] AI, as we are experiencing it today, is a function of concentrated technological capacities and capabilities, mainly by Microsoft, Amazon, and Google. These monopolists are using AI to further entrench their existing dominance. These monopolists largely control the direction, speed, and nature of AI innovation. It's not our friends at OpenAI and Anthropic who are in charge.
[0:26:41] Their approach to AI, the approach of these corporations, amplifies many of today's most existing harms caused by monopoly. These include suppression of trustworthy information, the spread of propaganda and disinformation, the manipulation and extortion of users, sellers, and workers, the further concentration of risk and hence the reduction in security and resilience. None of this is natural.
[0:27:08] It is due to the intentional overthrow of competition policy in the 1980s and 1990s. It was an economic, political, and economic coup of the first magnitude. The good news is that people around the world are awakening. Europe has made great advances over the last decade in strengthening and learning about how to better use competition policy.
[0:27:34] The Biden administration has revolutionized competition policy. This includes a DOJ victory over Google in court and the plan to break up Google and a second case against Google that will aim for the break off of its ad tech capabilities. There is a vast suite of tools in competition policy.
[0:27:59] We have every model and every principle that we need to deal with these other tech corporations in ways that allow us to fully capture the promises of AI. This is a very exciting moment in certain respects. If we manage the power of the biggest, we can begin to capture all the promises of AI while managing all the risks.
[0:28:25] But practically, this does mean focusing not on AI per se, but on the power of the corporations that dominate. My father grew up as a sharecropper in Florida. You know, it's not a good life, I can tell you that. Absent action, every person, every business, every nation will come to understand what it feels like to live on someone else's land, to live in someone else's world.
[0:28:59] Have a power analysis. If you do not have a power strategy, you will not be able to make AI safe or accountable. And all of your business ventures, all of your businesses and all of your business strategies and nations will be captured and or ultimately dismantled. There is enormous wisdom in this room, but I do encourage you to sharpen your focus. Yes, new laws will help.
[0:29:29] At the margins, yes, new networks would help, but the biggest immediate challenge is to master power. Right now, Microsoft, especially, is leading attack on democratic regulation. The CMA in the UK is under attack by Microsoft. It has had an enormous effect on what the UK government is doing. Right now, in addition, people associated with President-elect Trump are threatening regulators in the UK.
[0:29:58] In Brussels, around the world. This is a fact. Competition policy is your first and last line of defense. Margaret Vestager actually said this at a speech for our organization back June 27th. She said, the threat we face today, unfortunately, because we forgot how to use these tools properly, is one of totalitarianism. Hold the line. Hold the line and keep your focus on power. Keep your focus on these institutions.
[0:30:33] Our democracy will be safe and everything is possible. Thank you, Barry. And I'm sure a lot of us have a lot of comments and reactions already. Before we go to the open discussion, perhaps I can invite Yuray Chorba to the floor. Hi, Yuray.
[0:30:54] Well, your leadership and fostering collaboration through forums like OECD.AI and the Global Partnership on AI, I think are prime examples of international coordination. So I was hoping that you could share with us a little bit of learnings from that, especially given the integration of GPAY and OECD.AI. Thank you very much, Niki. Thank you, the Future Society, for hosting us again at this outstanding event.
[0:31:24] The first time I have participated was a few years ago already and it's been some journey. First of all, I will try to be very brief and we'll just try to convey three simple messages. As we are discussing gaps and as we are asking ourselves what can be done about it, I recommend to focus on what we have heard from our colleagues from the companies, from Istral, from Amazon.
[0:31:58] from Jack that pointed out that actually we need to make sure that we are clear about what we are actually discussing. In other words, I see there is still a gap in knowledge. There is a strong information asymmetry. We need to make sure that we receive or at least try to obtain
[0:32:26] Up-to-date information and that we speak to people who develop the models and the systems to be able to better understand and assist the risks and the needs for AI governance on a local or global level. We have received recently some news that there might be some limits to the scaling of AI models. We understand that maybe not necessarily all the expectations are
[0:33:00] Being met, especially the investors expectations as regards practical applications, day to day applications. So we need to better understand where we really stand and how can we respond appropriately and in proportionate way. Secondly, I think we need to focus on interconnections with other governance frameworks.
[0:33:31] In other words, we have already a lot of rules in place, binding or non-binding, as regards data, online platforms, Internet of Things, digital finance, and so on. We need to have a deep look into how our discussions on AI relate to these other frameworks and vice versa, and how successful we are or not in grasping
[0:34:03] The whole interconnected cyberspace services and actors in it. Of course, and this goes beyond the mandate of OECD, where I have the honor to chair the working party on AI governance and co-chair the global partnership on AI. But we also need to keep in mind the relationship with the governance of weapons and rules for armed or hybrid conflicts.
[0:34:35] Only then, arguably, will we be able to grasp what we are up to. Finally, and we discussed this at a recent event at Oxford University at a kind of invitation of Robert Trager, who is with us in the room, we need to realize that at the end of the day, it will be the judiciary, the judges, in other words, concrete humans, that will be deciding on harms and risks.
[0:35:09] No matter whether we introduce legislation or not, because as long as people or companies have an opportunity to file an action with the court, the court will need to somehow decide and will not always be able to return the whole issue back to the legislature. So there the key questions are, of course, how prepared is the judiciary to handle all these complex issues? What standards can
[0:35:44] And last but not least, given the new volumes of information, interactions, given the new speed of the models and systems we are developing, how can these legacy institutions
[0:36:14] How do we handle these complex cases? Are they ready at all or do we need some kind of new dispute settlement solutions which will more resemble or better reflect the nature of the cyberspace we are trying to govern? Thank you very much. Thank you, Juraj. I think you left us with more questions. So with that,
[0:36:49] Perhaps I can open the floor. I already see some hands. We can start here from the back of the room. I would just like to make the link between different presentations that we had today. We had some US perspective telling us quite frighteningly from a European perspective that
[0:37:22] Competition law is the first and the last defense. I would say, thankfully in Europe, we're not there yet. If antitrust and competition law is the last defense is because regulation is not working. We do have regulation in Europe. It is not just about giving...
[0:37:40] The hot potato, handing the hot potato to the judge. We need to enforce our regulation. When I hear about, oh, what are these risks? We need benchmarks. We need to know what they are. We do know what they are. All around this table, we do know what is affrontal rights. These are not domain areas in a taxonomy of risks.
[0:38:03] The taxonomy of risks is a list of fundamental rights that we have, for example, in the EU Charter of Fundamental Rights. The OECD has defined two categories, OECD incidents and OECD hazards. We can make all the list, and you have at the OECD already quite a lot of list of risks and harms to fundamental rights, and this needs to be implemented. And for regulation, I would like to react to what Borando Benefe just said.
[0:38:33] Thank you very much for having setting up this monitoring European Parliament joint working group on this. I would say that yes, it's great that we have the EU Act. It was difficult, but we have it. Now with implementation and the general purpose AI code of practice, it would be also great if it were not to destroy the minimum that we managed to put into the EU Act, meaning
[0:38:59] What we hear right now about the GPI code of practice is that it should take first and foremost the perspective of GPI providers because they are addresses of the code. I would like to remind that according to Article 1 of the EU AI Act, the first people we should think about are the affected people because what we're talking about is preventing and mitigating the risks.
[0:39:25] Two frontal rights of affected people. So there is no reason for GPI providers to have a preferential role in the process. And second, I would like to raise awareness about the issues that civil society organizations, academics, et cetera, have had in participating in the drafting process, because it has been anything but a level playing field.
[0:39:50] There have been some discussions, one person making a presentation after the other without any exchange. We don't know how the comments that we sent are taken into account.
[0:40:01] There is absolutely no transparency at all. And if there is a JPI code of practice workshop with the chairs, I think there should be also with those representing civil society or the affected people and also academics. So by and large, yes, we have regulation. Yes, we should protect it.
[0:40:23] Thank you for your action. And I hope that in the next meeting you will have with the AI office, you will manage to raise these points because it is very important. We have to know whether right now, states which are democratic states are ready to defend their values or not. This is simple as that. We know what is at stake. We know what are the rights at stake. Thank you very much. Thank you.
[0:40:49] Perhaps before we go to Duncan, I wanted to see if Brando or Lucia, maybe you wanted to quickly respond to that. Yeah, well, we have a meeting in two days, so I will discuss about that there. But I agree that we should have that every actor needs to be heard. At the same time, I want to also underline that it's important to talk.
[0:41:17] with the developers because I've seen that negotiating the actual initial text the legislative text this work is based on that sometimes it's also useful to understand the perspectives but also some technical feasibility that I think you can get some information on from those that are doing developing the
[0:41:44] The models and the systems. At the same time, I can also mention that on the transparency work that we have done, I've seen this level of technical unfeasibility moving a lot during the debate. So at some point, it seemed that a lot of things were technically unfeasible. Then the opinion changed based on what was
[0:42:09] More important, for example, when we move from the transparency on the synthetic origin of the content to the protection of copyrighted material. But this is another story. But just to say that I think we should listen to everyone with a good ear, but I'm sure the commission is doing that and we will check on that in a few days.
[0:42:39] Thanks a lot. Just to complement, I want to reassure you that the chairs and we are listening to all the different parties. As Brando explained, there is a dedicated group for providers, as you also say, which is important. But we are listening to everybody. Everybody is providing input. We are having processes to process also this input.
[0:43:08] The transparency is such that we also publish the minutes of the providers' workshops. So I think we are doing the best efforts to provide transparency to the whole process. And the second draft, as you know, will be published probably next week before Christmas. Thanks, Lucia. I want to go back to the issue of inclusivity, but maybe before we do that, Duncan, I know you had a question for a while. And then over here. We'll go to you next, okay?
[0:43:41] Thanks, Nikki, and thank you all for putting this together.
[0:43:45] I'm Duncan Kaspegs. I'm with the Center for International Governance Innovation, which is an international think tank based in Canada. And our main focus is on developing, helping to develop and design the mechanisms for international cooperation that might be needed to address the global scale challenges that we could see from next generation AI systems. And I think that's the main point I'd like to make here is just that I think it's really crucial for the remainder of today that we
[0:44:15] imbue our discussions with a very disciplined and rigorous understanding of the range of potential scenarios that we could be facing in the coming years. It's very
[0:44:27] Enticing to sort of imagine that we're getting ready for the scenario that we might hope for, you know, a sort of gradual development of AI systems where we're making great success in our mechanisms to control and align these systems and where it's, you know, widespread and diffuse. But the reality is that we are on track now from some scenarios that might be quite different from that, from, you know, a rapid, very rapid emergence of systems that we don't yet know how to control.
[0:44:55] And I think it's crucial that we be thinking about how do we design our governance mechanisms to be capable of handling those situations, the most challenging situations and scenarios. A lot of that will happen at the level of the companies themselves. It will happen at the level of domestic policy. But a huge amount of it will require international collaboration and governance. And I think this is probably under a variety of different configurations, United States and allies, United States and China.
[0:45:25] broader multilateral, but it's really important that we begin thinking now proactively about what those mechanisms look like, how do we build the building blocks essentially that will enable successful
[0:45:39] Collaboration on these issues. There are a number of key pieces that we can start focusing on now, whether it's from sort of the shared objectives to what would work in terms of technical cooperation to shared safety standards right through to verification methods and the designing actual institutions.
[0:45:58] We at CIGI are leading a process now to be doing some of that proactive design of some of these building blocks of future international agreements. But we look forward to working with others here and would encourage again just to make sure that we are putting on the table the most challenging scenarios in the rest of our discussions today. Thanks. And for the sake of time, I want to make sure we...
[0:46:21] Listen to as many people as possible. I know there's a question here and then maybe after we can go to Ambassador Pavel Latos and go back to the roundtable for a bit. Thank you so much. I'm Vania Škorić, Program Director of the European Center for the Non-Profit Law. I'd like to address very quickly two points from this session. One on the global standards for managing risks and the second one on inclusion of diverse perspectives. Now we can agree that AI systems without
[0:46:50] Adequate safeguards and inconsistent with the international human rights, democracy and rule of law pose the actual risk today. And the recent example you can see from this weekend from the EU member state when the Constitutional Court annulled the elections also because of the AI impact. Moreover, the civil society and especially those marginalized and vulnerable groups that are most impacted by the AI, they are
[0:47:17] Continuously missing from the discussions and shaping the policies and rules. Now two points can address that. AI policies, standards, and regulation really must integrate and exercise democratic, accountable, rights-based, and participatory approach across all the fields. And the second one, AI must be developed with meaningful input from the different society actors. Now in concrete, on the new AI governance mechanisms or processes,
[0:47:45] We call upon all of you to keep them open, inclusive, transparent, and facilitate meaningful engagement in full, and especially not duplicate or delegitimize the existing fora and existing arenas where these issues are being discussed or newly embedded.
[0:48:05] Like the UN WSIS process, the GDC Global Digital Compact, the UN IGF processes. We urge complementarity and coordination across these fields seeking synergy. We also call for more effective coherence and collaboration, including with key human rights bodies, regional or global or national, to ensure strong guidance for such governance mechanisms on the application of international human rights norms. On a more practical note,
[0:48:35] We call for a global wide mandatory assessment of the impact of AI prior to deployment and use. Somebody already mentioned Exante here to assess, manage, address potential risks to human rights, equity, fairness, democracy, safety and rule of law before we see negative consequences play out. We also
[0:48:57] We want to assess those risks and safety against, as already mentioned, the human rights benchmarks. We have the benchmarks. We have the standards. We just need to apply them. In the afternoon session, I can lay out more concrete proposals for objectives and deliverables for the upcoming AI summit. Thank you for the attention. Ambassador Pavoulatos. Thank you so much.
[0:49:23] Truly substantial discussion and for inviting me. And I'd like to make three points. The first has to do with the scope of the ambition. And it seems to me, even though we are sort of in a way representing like minded, more or less partners, the ambition of this exercise should be global.
[0:49:44] Why regulate AI at the global level? Because human safety is a global public good, because of the benefits of acting with economies of scale in this effort, because of the need to internalize the negative externalities and the need to bring in the parts of the world that are outside the sphere of like-minded jurisdictions, because the risks from not managing
[0:50:12] The safety challenges are cross border and they are part of the broader geoeconomic fragmentation, which is the result of the geopolitical polarization and bifurcation that we are seeing. And AI is an instrument that could become a factor of exacerbating this global bifurcation, leaving a good part of the world behind.
[0:50:41] And that is not something that we would welcome. The second point is that going from the global level to the national level, our democracies are particularly vulnerable to the undesirable effects of unregulated and ungoverned AI. And that is not only because of the obvious challenges of misinformation and disinformation in eroding the civic sphere without which our democracies cannot flourish. It is also because failing to regulate.
[0:51:10] To govern the safety challenges of unregulated AI, we would end up exacerbating the divides within our democratic societies, leading not just to digital, but also socioeconomic inequalities that could lead to a backlash against our democracies in search of strong leaders, which are quite in demand in our world today.
[0:51:39] Which brings me to the third point. I think it is important in seeking to develop a global framework of setting rules that are also enforceable at a global level to take account of the risk of a competition to the bottom, of a race to the bottom, and the problem of free riding that we always face whenever we try to develop inclusive global frameworks. There are some jurisdictions that refuse to play along.
[0:52:07] And the other response would be either to loosen up our frameworks so much so as to be able to include them, but then we end up with meaningless rules, or to try and devise strategies to bring them in while also preserving the standards and the objectives that we have agreed upon. And I think this is one of the major challenges that we need to work on. Thank you. Let's definitely, I think...
[0:52:36] Elaborate on this issue around global inclusivity. And on that, I want to bring Linda into the conversation. My name is Linda Bonio. I am the founder of Lawyers Hub. We're based in Kenya, and we run the Africa AI Policy Lab. I want to talk about the digital policy divide, just borrowing from what Ambassador has talked about. And Member of Parliament, you mentioned GDPR.
[0:53:01] And the impact GDPR has had in Africa out of 34 countries that have come up with privacy laws in the past few years after post-GDPR, 18 of them are GDPR-like. And if you look at it keenly, you realize that many countries cannot afford to come up with their own policies. We're seeing this with national strategies that are funded by the Germans and Europe in Africa. And so ideally,
[0:53:30] National governments do not have the capacity to even run their own policy programs, even from the African Union AI strategy. I think that challenge should concern all of us, especially when this gap is filled by big tech companies, especially the policy gap. And we see this with...
[0:53:50] We've seen this with the data workers and data labelers within the AI ecosystem. Sometimes we say that the global south is not really involved in the conversations around AI policy. But what we see is that any time big tech companies are held accountable in the global north, then they move to the global south and begin this particular, you know,
[0:54:12] They begin this interventions and businesses within the local ecosystems. In Kenya, data workers took to court several big tech companies and what they did was then to move to Ghana because then they did not have any sort of accountability within the Ghana ecosystem. I think that we need to look at ways in which we can support policy development in the global south without necessarily being
[0:54:39] Either like big tech or whatever, but to look at a more impartial approach towards having AI policies. There's a lot of talk by governments. If you look at, for instance, my government, you know, they say they have an AI safety institute, but then you don't see that. It's just lots of talk at the global level, but we don't see these particular interventions. And I like what Nicolas said at the beginning, that let this not be a talk show.
[0:55:03] But how do we progress towards incentives? We've seen this coming from Europe where developers are saying the data is clear about people getting O1 visas to move to the United States. Everyone is moving to the United States to build. That's not a really great thing to do. How do we make sure that our developers have incentives to stay home, to build, and then now we can hold them accountable because they build within our national frameworks? Thank you. Oh, I see so many.
[0:55:34] Name cards. I know, Deb, actually, you wanted to talk from the last session, so perhaps. Okay. All right. Yeah, I actually, this is more connected to an earlier conversation, so to move away from the global dynamic, but this was actually a question for Jack, not to put you on the spot. But I think the comments that you made...
[0:56:03] around open source is definitely worth revisiting. So one thing, for example, so at Mozilla, there's been a lot of discussion around open source. Mozilla is sort of the original sort of open source browser and has been working with groups like Linux Foundation and just like the kind of OG open source folks for quite a while and has been working with them on this transition towards
[0:56:24] Open source AI and trying to define that. So they actually held multiple convenings in partnership with Camille Francois at Columbia University. The latest one, which was in partnership with the French AI Action Summit. So this conversation around defining open source is definitely something that Mozilla has published a couple of technical reports on. I highly encourage folks to check that out. Also on the point of third party auditing.
[0:56:48] We have been doing work on the technical and institutional infrastructure required for third party auditing to exist. A couple of people made references to other industries in which these audits happen. It's not that those audits came out of thin air. There's actually a lot of institutional infrastructure and technical infrastructure, everything from
[0:57:10] Inventories of the existing products in that ecosystem, incident databases, monitoring tools that we currently don't have for AI systems. And this kind of loops me back to my question to Jack, which was around kind of the dynamic around open versus closed source when it comes to safety. So far, we've had a couple instances of NCII or CSAM sort of inappropriate images being produced of
[0:57:38] Different celebrities, et cetera. And one of the most visible cases, I keep using this example because it's quite striking of Taylor Swift having sort of non consensual, intimate imagery being made of her was through a closed source model. I mean, you're aware of that. And I think part of the difficulty here is that these closed sourced proprietary models, and you mentioned it in your opening statements, Jack, of we don't actually have visibility into what that accountability process is internally. We don't actually have
[0:58:07] visibility into what kind of internal monitoring OpenAI or Anthropic is doing. And I know there are initiatives. There might be legal limitations and how open you guys can be about that. But at least to regulators, I think there's a lot of progress that could be made to increase the visibility of these internal accountability mechanisms that that would then, you know, support the case of the fact that these closed source models are quote unquote, safer than these open source models. Because at the moment, I think something that
[0:58:37] Does sort of work in favor of these open source models that we have visibility into the vulnerabilities of these systems and we can build defenses around them. And there's this more open dynamic and dialogue between those that are sort of attacking these systems and those that are defending these systems because the systems themselves are fully visible and fully intractable by anybody. And so I guess I'm kind of curious as to what the plan is in terms of making that accountability and evaluation.
[0:59:05] Thanks very much. I'm glad we're having a discussion. It's really useful.
[0:59:27] So on your point specifically on the issue of stuff to do with non-consensual imagery, we work with Thorn, which is a third party that validates some of how we do internal safety procedures here and also checks how we're working with a third party database, which stores logs of these imagery so that we can check against our internal systems. And I think that's representative of where we need to go.
[0:59:55] The company is working with a third party that's providing oversight of their internal safety processes and reporting that out. And I think that we need to basically find a way to massively scale up those organizations like Fawn and others so they can work across the ecosystem. In terms of long-term plan for visibility around the degree of monitoring, the degree of...
[1:00:24] The sort of description or the detail of your internal accountability processes to regulators and the public. What is that plan and what does that look like? And it doesn't have to be you. It can be any corporate actor. Yeah, sorry. I think a lot of how we're approaching this is trying to really kind of lean in with the AI safety institutes to do some of this testing and validation where we ultimately think the ACs and their network can become that kind of large scale actor. And as they
[1:00:55] get kind of more wins on the board they can massively grow the probably remit of things that they study and I view that as the best path to getting there and is our current kind of mainline strategy and the other thing we're doing is trying to do lots of experiments and domains that the ACs don't cover like before an example or we worked with the department of energy on on nuclear non-proliferation issues recently
[1:01:18] So running a bunch of other experiments, because if the ACs don't work, maybe one of these other experiments can scale. That's great. And with that, we actually have Samir, who's representing the Canadian AC. Maybe you can speak next. Very glad to be here. Samir Chhabra, Governor of Canada Marketplace Framework Policy at ICED. In that role, I'm responsible for things like competition policy, RAI policy on governance, as well as standing up the new Canadian AI Safety Institute. I think the conversation today has thrown into pretty stark relief for me.
[1:01:53] Both the massive scope of risk as well as massive scope of opportunity. I think many different speakers have highlighted those kinds of issues. I think it also speaks to why Canada has taken a very multi-pronged approach to its AI policy, starting back in 2017 with the Pan-Canadian AI Strategy, investing in the science, tabling the Artificial Intelligence and Data Act in 2022, which is a risk-based horizontal legislative framework for AI risk.
[1:02:18] Launching a voluntary code of conduct on generative AI and advanced AI systems last year, 2023, which now has over 40 signatories. And then just last month, the Canadian AI Safety Institute was launched, working very closely in partnership with a number of colleagues around this room and part of the broader network.
[1:02:37] Also, last year completed generational reforms in competition law. I'm very glad to hear the reference made to power and competition law a number of times in this discussion. And in fact, facilitated our competition regulator taking action against Google Ad Tech just a few weeks ago.
[1:02:54] Finally, I'll highlight a couple of other pieces that are related, which I think are important to understand the opportunity and how to build and ensure that organizations and entrepreneurs and innovators within a number of different jurisdictions have the opportunity to engage in this space meaningfully. Just last week, we announced the details of our compute strategy, $2 billion for building a compute infrastructure, including $700 million for a compute challenge.
[1:03:19] $300 million for compute access fund for innovators and businesses, and $1 billion for public supercomputing infrastructure capacity. Last I'll note the issue of AI and copyright, which has come up here as well. We did a consultation on that that closed earlier this year, and the report will be coming out very soon. More than 1,000 inputs and submissions and feedback on that, thinking about issues like text and data mining and the attribution of copyright ownership.
[1:03:45] I raise these issues because I think they speak to the breadth of what we're talking about when we're talking about AI. And I think it also speaks then to how do we engage in these issues internationally. And a multi-pronged approach internationally seems absolutely inevitable. And I would urge the organizations that are here at the table to think about how do we start moving towards clear responsibilities.
[1:04:12] Clear roles in that broad international dynamic. A network of networks or a forum of forums where we can bring those different ideas together, ensuring that we're leveraging things like competition, regulator authorities getting together and having conversations in that space. And how do we build on those to feed them into the broader conversation? It feels to me like having singular tables or separate conversations that are each trying to wrestle with the breadth of AI is incredibly challenging to do.
[1:04:41] And one of the only ways we're going to get at this is by thinking about where are we breaking down and specializing and thinking about the various issues, and then how do we feed those back up in a way that allows us to take cogent actions. Thanks, Samir. And I realize, I think, at least to me, a lot of this open discussion felt random points from one person to the next. During lunch, I hope that there's opportunity to actually go and follow up with specific individuals and discuss a lot of these things further. Perhaps I'll just take one more comment now.
[1:05:10] And then we'll continue in the afternoon as well. Stephanie, do you want to go? No pressure to be there. OK, thanks to the... Is that better? Thanks to Future Society for hosting. This has been such a useful dialogue so far.
[1:05:33] A few, I think three points or so to quickly raise. I personally think regulation, I think that's come up a few times, is critical. But I think from Maya, you mentioned the AI Bill of Rights and the US executive order.
[1:05:45] I think across jurisdictions, the feasibility of legislation shifts. And even where progress is made, frameworks can be done and undone. And I know that we're probably all sort of thinking ahead to January and what will happen with the US executive order. So I think something that comes to mind for me is what other tools and places do we have to drive accountability? And so, Jurek, I know you mentioned the courts. I think the courts will be an important and critical place for decision making, but that will likely come in the future.
[1:06:11] But they will need evidence is something that came to mind when you were speaking. And so for me, what technical evidence might support how the decisions are made and weighed up even in the courts in the future and what does good look like? So I agree with Jack and Nosha's points from earlier around the need for transparency, but also I think benchmarks for evaluating good practices.
[1:06:32] And one aspect of evidence for me comes back to the role of standards. And folks who know me in the room know that I'm a standards geek, so it's no surprise that I mentioned their importance. But more specifically, the role of the presumption of conformity that standards can provide. And that's the assumption that you undergo tests and evaluations, and then you come up with this presumption of conformity, that the assumption is that you have adopted this good practice.
[1:07:01] I think that the role of the presumption of conformity is also good because it doesn't just give evidence to courts in the future, but can also be used by governments and citizens as sort of trust that you underwent a process where there is a clear understanding of what good looks like and you attested against that. I do, however, think that through discussions such as this, we have to be very intentional. I think, Jack, you made up three tiers and one of them was this kind of testing mechanism in the second.
[1:07:28] The second phase, which the outcome could be this presumption of conformity that you get. But we have to think very carefully about what third party audits look like, what conformity assessments ecosystems look like, and then who is involved in designing this. And I think there's a lot of good practice in this room to help inform designing those processes and the benchmarks. But I think getting that right will be absolutely critical.
[1:07:50] And then finally, I know we'll come in the afternoon, I think, also to talk about institutions and the safety institutes and all the exciting work that we're doing. I think the institutional question will be an important one. And I think I'm just keep borrowing from Jack's model there. I think the third part of what you said was that international network. And I think that that's important because presumption of conformity largely sits with technical standards. But I think what we're seeing in the international landscape in this new stack that we're developing is that we are shifting away.
[1:08:17] At the moment, at least from the kind of formal standards ecosystem, as we know it. So just so you give one example is the EU code of practice, which is an important process that Brando, you mentioned, versus the sense and luck. And I am pointing at Sebastian, who's led JTC 21 and the European standards process. So as we shift to new mechanisms, such as the code of practice, which might not and
[1:08:40] Lucila, tell me if I'm wrong. If it doesn't have the same sort of legal kind of presumption of conformity role, what does that mean when it comes again to the future to courts? What does that mean when it comes back to formalizing things like conformity assessment processes? And so maybe my final sort of thought there and maybe a question to the government in the room is what sort of traditional mechanism such as this
[1:09:05] you know presumption of conformity what should we be trying to retain as we build this kind of new governance ecosystem and we think about the role of new institutions and processes and that goes back also to the UN high level uh advisory group but the scientific panel that you specifically proposed and what role it will what role it will play