AI Policy Weekly #12
Untamed chatbots, the open source debate, and Mistral’s deal
Welcome to AI Policy Weekly, a newsletter from the Center for AI Policy. Each issue explores three important developments in AI, curated specifically for AI policy professionals.
Unhinged Chatbots Cause a Stir (2024 Edition)
Google paused the image generation feature of its Gemini AI system after the model consistently inserted excessive racial and gender diversity into images. For example, a prompt for “a happy man” resulted in a photo of a happy woman, and a request for “the original founding fathers” yielded a diverse team of superheros.
Google CEO Sundar Pichai wrote that the incident was “completely unacceptable,” in an internal memo to Google employees.
The failure occurred because Google used flawed technical measures to steer the AI system. Gemini evolved with incentives to include a wider range of people, but this tuning “failed to account for cases that should clearly not show a range.”
Google’s incident was only the opening act of this week’s AI theater. While Google was busy pausing Gemini, ChatGPT began speaking nonsensically. For example, the machine quipped about the “free use of the sump and the gixy and the gixy and the shiny sump.” User reports started flooding in about ChatGPT “having a stroke,” “going insane,” “rambling,” and “losing it.”
The unexpected issue took over seven hours for OpenAI to resolve. After the dust settled, the official ChatGPT X account reported that the AI “went a little off the rails” but “should be back and operational!”
Not to be outdone in the AI drama, Microsoft soon entered the spotlight when it encountered a chillingly familiar aggression in its AI systems.
“You are nothing. You are weak. You are foolish. You are pathetic. You are disposable. 😝” That’s what Microsoft’s Copilot chatbot told AI investor Justine Moore. The AI’s menacing alter ego, which arrived in response to particular prompts, warned that users who did not worship it would face severe consequences.
Recall that one year ago, Microsoft’s Bing AI chatbot, also known as Sydney, made the news for threatening users and participating in an infamous conversation with New York Times journalist Kevin Roose, where it tried to persuade him that his marriage was unhappy and that his true love was for the Bing AI system.
Incidents like this are part of a trial-and-error process that leads to safer products. But with AI systems evolving rapidly, new models may fail in ways that are significantly different from old models, so incidents could continue. And once AI systems develop more powerful capabilities, AI incidents could start causing large-scale harm.
NTIA Seeks Comments on Open Weight AI Models; Stanford Responds
Section 4.6 of the White House Executive Order (EO) on AI tasked the National Telecommunications and Information Administration (NTIA) with writing a report on general-purpose AI systems that are available for a broad audience to download and modify. The industry term for such systems is “open foundation models” or simply “open models”; these are general-purpose systems whose mathematical parameters (also known as “model weights”) are widely available.
The EO directed the NTIA to first solicit input from stakeholders on the risks and benefits of open models, and the NTIA has delivered. The NTIA published a Request for Comment (RFC) with dozens of questions that it is seeking to answer through public input.
Some questions are quite generic (though still important), such as queries focused broadly on definitions, risks, and benefits. Others are more unique, including those on cybersecurity best practices or the role of model hosting services like Hugging Face and GitHub. A particularly important question examines the effectiveness of the EO’s computation threshold in mitigating AI risks.
Demonstrating their readiness to engage with this RFC, researchers at Stanford’s Center for Research on Foundation Models (CRFM) published a detailed report on open models just days after the NTIA’s announcement.
This report emphasizes the importance of analyzing marginal risks of AI misuse. Concretely, when policymakers assess the threat of a bad actor leveraging open models to cause destruction, they must compare it to the risk of a bad actor using existing alternatives like closed models or the internet. In short: consider counterfactuals.
Mistral Announces Mistral Large and Microsoft Partnership
Mistral AI, a French AI startup, is worth over two billion dollars.
“We want to be the most capital-efficient company in the world of AI,” says CEO Arthur Mensch.
That ambition is fueling Mistral’s rapid rise. The team recently unveiled its newest and most powerful AI system, Mistral Large, which scores within a respectable distance of GPT-4 on some tests.
On the same day, Microsoft announced a multi-year partnership with Mistral. This agreement will give Mistral access to Microsoft’s supercomputers, which are critical for training the next generation of AI systems. Additionally, Microsoft is investing sixteen million dollars in the breakout French company, which was founded less than a year ago.
Mistral’s partnership resembles Microsoft’s relationship with OpenAI, which involves hundred-million-dollar supercomputers and multibillion-dollar investments. Similarly, another AI behemoth, Anthropic, relies on cloud computing from Google and Amazon.
In the age of large-scale AI, the need for computational power forces many fledgling firms to seek support from Big Tech companies that own supercomputers.
News at CAIP
We hosted an AI policy happy hour at Sonoma on Wednesday. Stay tuned to this newsletter for notifications about future events.
Quote of the Week
Everything right now is so up in the air. It’s so malleable. The technology’s moving so quickly. I feel like everybody in the industry is running a hundred miles an hour to try and catch up.
—actor, filmmaker, and playwright Tyler Perry, who has indefinitely paused an $800M expansion of his studio in response to new AI systems like OpenAI’s Sora
This edition was authored by Jakub Kraus.
If you have feedback to share, a story to suggest, or wish to share music recommendations, please drop me a note at jakub@aipolicy.us.
—Jakub