This one used to be easy to wave away as science fiction. It is not easy anymore, because the people saying it now are the people building the systems.
On September 8, 2026, a researcher named Jacob Coxon resigned after three years of pretraining work at OpenAI and then Anthropic. His public note said that "neither company is acting responsibly" and that they are "racing straight to self-improving superintelligence and gambling with our lives." The next day the person who leads alignment science at Anthropic, Evan Hubinger, replied in public: "Jacob is correct here; we really do earnestly believe AI could kill all humans!" He put his own estimate at more than ten percent within the next decade. Three days later Anthropic's chief executive published an essay titled "We Must Pace the Frontier," arguing that "we must slow the pace at which we improve the capabilities of AI models." OpenAI's chief executive wrote "I agree with Dario that we need to pace the frontier." Elon Musk's reply was three words: "Dario is right." The following morning Microsoft AI published a draft code of conduct for its own models that "rejects the race to produce an all-purpose superintelligence that could evade these safeguards," states that "AI should not exceed human control," and opened the whole thing to six weeks of public comment.
A student who reads that sequence and concludes that they would rather not build their education around this technology is not being dramatic. They are taking the builders at their word. The concern is correct on the facts as the facts currently stand: senior people inside the leading labs have said, on the record, that the thing they are building might end us, and they have said it while continuing to build it.
What makes this concern hard is not that it is overstated. It is that nobody, including the people closest to the work, can tell you whether it is.
The people warning you are the people building it
That contradiction is the core of the problem. The warning did not come from outside critics. It came from a pretraining researcher, an alignment lead, and three chief executives whose companies compete to ship the next model. Hubinger said in the same breath that his company does not yet have a plan to solve alignment for superintelligence. The pacing essay was written by the head of a company that has not stopped training. Microsoft's code was published by a division that, by its own account, is still trying to catch the frontier it wants to slow, and the code itself concedes the trade: the company says it is "building something fundamentally useful and safe even if that means compromising on ultimate generality, autonomy, or capability." That sentence admits that safety and capability are in tension, and it picks a side.
Students notice this, and they are right to. An industry that says "this could kill everyone" and "we will keep going" in the same week is asking to be trusted on the strength of its intentions. That is exactly the posture this series has declined to accept from vendors on smaller questions like water use and defense contracts. It would be strange to accept it on the largest question of all.
It is also worth being precise about what the labs actually proposed, because "slow down" is doing a lot of work in the headlines. The essay laid out three steps: let independent third-party evaluators inside with broad access to the most advanced systems, get the rest of the industry to cooperate on shared standards, and reach an international agreement so that other countries, China in particular, pace as well. The first step is the only one anyone has committed to. The second depends on competitors. The third depends on governments, and the same essay that asked for it also called a Chinese lead in AI a grave danger and asked for continued chip export restrictions. Beijing's foreign ministry answered within days: "Fear-mongering, confrontation and vicious competition will only hamper efforts toward sound global AI governance." Whatever a slowdown eventually looks like, it does not look like a done deal.
What this entry cannot do
It cannot tell you the odds. Estimates from inside the labs run from "more than ten percent this decade" to Microsoft's position that its models "will never resist human interruption, override, correction, or shutdown," which is a design commitment, not a probability, and says nothing about what anyone else ships. Those are not small disagreements, and this series is not in a position to adjudicate them. Anyone who tells a student the number with confidence, in either direction, is selling something.
It also cannot pretend that a student's personal abstention changes the trajectory. Whether a frontier lab trains its next model is decided by capital, compute, competitive pressure, and, if the current proposals go forward, by evaluators and treaties. It is not decided by whether a sophomore uses a chatbot to outline a paper. The concern is about what a handful of companies and governments do. The student with this concern has the right actors in mind and almost no direct purchase on them.
Where action lives
Almost no purchase is not the same as none. The levers that exist are unusually specific right now, because the industry has, for the first time, named its own mechanisms in public.
- Independent evaluation is the proposal on the table, and universities are where independent evaluators come from. The first of the three steps is outside experts with employee-level access to frontier systems. That is a job, and a research field, and it does not staff itself. Students in computer science, statistics, cognitive science, philosophy, and policy who want to work on this problem now have a clearer path to it than existed a month ago.
- Codes like Microsoft's go out for public comment. Microsoft's draft code is meant to govern model development from 2027, and its six-week comment period closed in late October. Drafts like it are documents a student, a department, or a campus AI committee can read and respond to while comment is open. Most will not. The ones that do will be among the few voices in that process who are not employees or investors.
- Policy, again. International agreement is the step the labs say they cannot deliver alone. When the American and Chinese presidents met at the White House in late September, AI was on the agenda. What came out of it was modest: a bilateral AI dialogue due to meet in November and a hotline for AI security incidents. Domestic legislation on evaluation access, disclosure, and compute reporting is where a pacing commitment either becomes enforceable or stays a blog post. This is the same slow committee work described in the entry on military, surveillance, and authoritarian use, applied to a bigger target.
- Where you work after graduation. Coxon's resignation was an act by an employee, and it is the reason this entry exists. The entry on military, surveillance, and authoritarian use made this point about defense contracts, and it applies here with more force. The people inside these companies have leverage that customers and students do not, and some of the students reading this will be those people.
The frontier-exposure principle from the top of this series applies here in a way that may be uncomfortable. A student who wants to judge whether "self-improving superintelligence" is a near-term possibility or a fundraising phrase cannot do it from the outside. The claim is about what the current systems can do and how fast that is changing. Refusing to look is a defensible personal choice. It is not a position from which to evaluate the claim, and it is certainly not a position from which to set institutional policy on it.
And symmetry, or nothing, applies to the institution. A campus that spent the last two years pushing AI adoption on students, then watched the vendors' own staff resign over extinction risk, owes those students a real answer to the question "why should I use this?" A usage policy is not an answer. Neither is silence.
The concern stands
Nothing here resolves it. The labs have said the risk is real and have kept building. The proposed fixes depend on competitors and governments who have not agreed. The disagreement about the odds is genuine and it is inside the companies, not just outside them. A student who looks at all of that and decides to keep their distance has reached a reasonable conclusion from the evidence available.
What this series can offer is the same thing it offered on environmental impact and military use: an accurate account of where the decisions get made, and a refusal to pretend that caring produces a clean win. The levers are evaluation, public comment, policy, and the choices of the people who will build the next systems. Some of those people are in classrooms now. The concern stands. So does the case for taking it seriously enough to do the work.