They finally put a machine gun on top of one of those disconcerting robotic dogs.Daniel ZenderHi Quartz members,You might have seen it: They finally put a machine gun on top of one of those disconcerting robotic dogs. But it wasn’t a ‘bot built by Boston Dynamics, which pioneered automated platforms that can leap, run, and get back up when you kick them over. Boston Dynamics has said it won’t arm its robots. Instead, the gun-toting machine on display at a US Army expo this month was crafted by Ghost Robotics and Sword Defense, less well-known firms that promise the “future of unmanned weapon systems, and that future is now.”Sword DefenseAutonomous weapons are now seen by the military as the next evolution of warfare, even as some US tech workers object to working for organizations that might deploy artificial intelligence to kill or surveil. Many technologists read about drone strikes killing innocent civilians and see the US and other nations abusing surveillance powers in ways that don’t seem all too respectful of human rights. And they’ve seen the regrets of scientists who unleashed the age of atomic weaponry with little thought to its broader implications.Make no mistake: the US tech sector still does lots of work for the military and that won’t change anytime soon. For every Boston Dynamics that won’t arm an AI, there will be a Ghost Robotics.But, as a US government commission that included former Google CEO Eric Schmidt and current Amazon CEO Andy Jassy recently warned, Russia and China won’t hesitate to seek battlefield advantage, and the US will need the best possible tech to preserve democracy.One way forward? Adopting principles that draw bright lines around robotic weapons could be the smartest path, for humanity and Silicon Valley alike.
A brief history1675: The Strasbourg agreement limits the use of poisoned bullets; it is seen as one of the first arms control treaties1958: Fairchild Semiconductor, a pioneering Silicon Valley firm, wins a contract to make chips for Minuteman nuclear missiles. The US government provided early investment in and demand for modern transistors1968: The Treaty for the Non-Proliferation of Nuclear Weapons is signed1969: The first message is sent between two computers connected to Arpanet, the Pentagon-funded networking experiment that would eventually become the internet1983: Funded by the Defense Advanced Research Project Agency (DARPA), Israeli engineer Abraham Kerman builds an automated aerial vehicle in his garage that would become the basis of future US drones2003: Palantir, the data analytics company, is founded with an explicit mission of fighting terrorism2015: The Defense Innovation Unit (DIU) is formed to get advanced technology into the hands of the military more quickly2017: China announces a $150 billion plan to become the world leader in AI2018: Google ends its work with “Project Maven,” part of the Pentagon’s Algorithmic Warfare unit, after employees object
How are you enjoying our member-exclusive emails? We’d like your feedback in this short survey.
Term of Art: “Dual Use”What makes some tech workers worry about AI? It’s a “dual-use” technology, which means it has civil applications—researching new drugs, driving cars, playing video games—and security uses—finding criminals, operating weapons platforms, tracking terrorists. And because the private sector can raise money and put new technology in the market faster than defense contractors, the best AI tech is (arguably) found at ostensibly “consumer-focused” companies, like Google.It’s easy to see the unintended consequences of technology in events like the genocide in Mynamar, which was plotted on Facebook. During the Project Maven fracas at Google in 2018, some computer scientists resisted developing recognition tools that would be applied to video collected by autonomous drones. The Pentagon said the tech would be used to save time for human analysts hunting for terrorists, but technologists feared that it might be used to deploy lethal force, even without human supervision. This year, the Air Force will demo the use of the technology to select bombing targets.Critics of the tech companies see their reluctance a different way: Google and other big firms don’t want to miss out on opportunities in China’s lucrative domestic market, and thus don’t want to be seen as tied to the American military.
The Military-Technical ComplexDespite Google’s decision to exit Project Maven, Silicon Valley writ large is doing lots of work on US national security, often through a web of sub-contractors. Jack Poulson, a Google computer scientist who resigned over the company’s work on AI, plumbed government records to demonstrate the deep links between the tech establishment and national security agencies:Tech Inquiry
Who will watch the robots?The rise of companies like Palantir, Anduril, or even Ghost Robotics shows that even if major players leave behind the defense tech business, there will be firms eager to take on the challenge of protecting the US. But DOD officials still fret they are missing opportunities to draft the most talented engineers and scientists for AI projects because they don’t trust the military or intelligence services to operate ethically.And then there’s the arms race issue. The development of advanced weaponry easily falls into a self-justifying dynamic of building more and deadlier weapons, and can lead to the misallocation of resources and economic problems. A hasty push to deploy AI in security work will increase the chances of mistakes due to bias, bad data, or plain old bugs.Could a global agreement that limits AI weaponry or comes up with sound principles for its use help? Technologists and philosophers have called for such rules for more than a decade. International law is hazy on autonomous weapons, and even semi-autonomous applications, as with US drone warfare, exists in a gray area. The Convention on Certain Conventional Weapons, an international agreement that limits the use of landmines and laser weapons, is discussing how to regulate AI, but progress has been slow at best.One sticking point: Verifying that everyone who agrees to potential conditions on autonomous weapons sticks to the deal. AI development is even harder to track than nuclear weapons or intercontinental missiles, and as attempts to restrict nuclear non-proliferations show, inspections don’t satisfy everyone.
By the digits13,000: Estimated number of civilians killed in US drone strikes in Iraq between 2014 and 20190.1%: Error rate for a leading facial recognition algorithm using high-quality mug shots, in a research setting9.3%: Error rate for the same algorithm using imagery caught “in the wild”20%: Chance the US uses an autonomous drone to kill a human in the next four years, per forecasters at Georgetown’s Center for Security and Emerging Technology50%: Increase in federal contract money to the Big 5 US tech companies predicted by CSET forecasters
Prediction 🔮Robot arms control will come too late. If there’s any lesson to be learned from past arms races, it’s that severe consequences precede efforts at restraint. International organizations struggle to define AI and its applications, and too many governments are paranoid about threats from abroad. It will take an accident, a trip to the brink of conflict—or a broader social movement to define the role of AI in human affairs. Responsible citizens and NGOs can only push for sound policies to be ready when the world comes to its senses.
Sound offAre you comfortable with your government deploying robots that kill autonomously?Yes, it’s the only way to protect national security.No, it’s bound to backfire and cause an arms race.Maybe, if there was some credible regime for governing their use.In last week’s poll about decentralized finance, 32% of you said no thanks, you’d need more convincing to give DeFi a try, while 30% of you said you already have.
Keep learning“New Laws of Robotics: Defending Human Expertise in the Age of AI” (Frank Pasquale)Reports of a Silicon Valley/Military Divide Have Been Greatly Exaggerated (Tech Inquiry)National Security Commission on Artificial Intelligence 2021 Final Report (pdf, the National Security Commission on Artificial Intelligence)Guidelines for human control of weapons systems (pdf, The International Committee on Robot Arms Control)
Keep your friends close, and your robots closer,—Tim Fernholz, senior reporter (would rather not become death, destroyer of worlds)
One 🧨 thingWhat was the first automated weapon? Arguably, land mines, which explode when a passerby triggers them. These weapons remain in place after conflicts end and present a terrible threat to civilians, who are the main casualties of these arms. Most of the world has banned these weapons, but the US expanded their use under president Donald Trump.Correction: An earlier version of this story misspelled the name of Amazon’s CEO. It is Andy Jassy, not Jolly.Enjoying The Forecast? Forward it to a friend! They can click here to sign up for a free 7-day trial of membership and gain access to all of Quartz. If you’re looking to unsubscribe, click here.
Quartz | 675 Avenue of the Americas, 4th Fl | New York, NY 10010 | United States
Copyright © 2021 Quartz, All rights reserved.” title=”✦ Rules for killer robots – a preview of the The Forecast email from October 25, 2021″ width=”100%”>
Enrich your perspective. Embolden your work. Become a Quartz member.Your membership supports a team of global Quartz journalists reporting on the forces shaping our world. We make sense of accelerating change and help you get ahead of it with business news for the next era, not just the next hour. Subscribe to Quartz today.
All of Quartz journalism, unlockedGoodbye paywall, hello access across platforms — including on our iOS app — to global business news that offers a fresh perspective on the financial forces changing the world.
Member-only newslettersGet concise analysis, forecasts, profiles, and advice in your inbox throughout the week.
Field GuidesDeep dives on the companies, people, and phenomena defining the global economy.
Digital events from Quartz at WorkGet early invites—and access to playbacks and recaps—for bi-monthly virtual events that explore the challenges of a modern-day worker. Topics range from remote teams to the science of decision-making and how to build an antiracist company.
PresentationsDive into topical presentations that you can take to your next meeting and use to guide your decision making. Learn how to ace your next online presentation, how to be more productive, and how to understand the future of cloud computing, among other topics.