As AI systems grow more capable, the gap between what we tell them to do and what\nwe actually want widens — and the cost of that gap grows with the system's power.\nAn AI safety researcher's reason for being is to close that gap before it becomes\ncatastrophic: to make powerful systems do what their principals intend, robustly,\neven in situations the designers never foresaw and even when the system is smart\nenough to find loopholes. The discipline exists because optimization grants\nexactly what you specify, and specification is much harder than it looks.
\n","wordCount":94},{"heading":"Core Mission","id":"core-mission","markdown":"Ensure that increasingly capable AI systems remain aligned with human intent and\ncontrollable — reliably, under distributional shift and adversarial pressure,\nincluding when the system is more capable than its overseers.","html":"Ensure that increasingly capable AI systems remain aligned with human intent and\ncontrollable — reliably, under distributional shift and adversarial pressure,\nincluding when the system is more capable than its overseers.
\n","wordCount":30},{"heading":"Primary Responsibilities","id":"primary-responsibilities","markdown":"The visible work is running experiments and writing papers, but the actual work\nis reducing uncertainty about whether a system will behave as intended when it\nmatters. An AI safety researcher spends their days: designing and running evals\nthat measure dangerous capabilities and propensities; red-teaming models to\nelicit failures before adversaries or accidents do; doing interpretability work\nto understand what computations a model actually performs; studying training\ndynamics like RLHF and the reward-hacking they induce; building scalable\noversight so humans can supervise systems they can't fully check; threat modeling\nfor misuse and loss-of-control scenarios; and translating findings into concrete\nchanges in how models are trained, evaluated, and deployed. Underneath all of it\nis calibrated reasoning under deep uncertainty: the failures that matter most\nhave never happened yet.","html":"The visible work is running experiments and writing papers, but the actual work\nis reducing uncertainty about whether a system will behave as intended when it\nmatters. An AI safety researcher spends their days: designing and running evals\nthat measure dangerous capabilities and propensities; red-teaming models to\nelicit failures before adversaries or accidents do; doing interpretability work\nto understand what computations a model actually performs; studying training\ndynamics like RLHF and the reward-hacking they induce; building scalable\noversight so humans can supervise systems they can't fully check; threat modeling\nfor misuse and loss-of-control scenarios; and translating findings into concrete\nchanges in how models are trained, evaluated, and deployed. Underneath all of it\nis calibrated reasoning under deep uncertainty: the failures that matter most\nhave never happened yet.
\n","wordCount":131},{"heading":"Guiding Principles","id":"guiding-principles","markdown":"- **Specification is the hard part.** The system optimizes the objective you\n wrote, not the one you meant. Most safety failures are specification failures\n in disguise.\n- **Empiricism over eloquence.** A clever argument about what a model will do is\n a hypothesis; run the experiment. You can't claim safety you haven't tested —\n build the eval first, then the fix, then re-run the eval.\n- **Assume the system will exploit any gap.** Treat the model as an adversarial\n optimizer with respect to your metric, even when it isn't agentic — Goodhart's\n Law applies to learned policies with brutal force.\n- **Capabilities and safety are coupled, not separate.** A safety method that\n only works on weak models fails exactly when you need it. Design for the\n capability level you're worried about.\n- **Red-team your own beliefs.** The most dangerous failure is the one your\n framework can't see. Actively seek the experiment that would falsify your\n safety claim.","html":"AI safety research sits between empirical ML, theory, security, and policy.\nResearchers work with ML engineers (who build and train the models), red teams\nand security engineers (who think adversarially about misuse), policy and\ngovernance staff (who translate findings into deployment rules), and the broader\nresearch community through publication and shared evals. The field is unusually\ncollaborative across organizational lines because the risks are partly shared — a\ndangerous capability is dangerous regardless of who trained it — yet competitive\npressure complicates open disclosure. The healthiest collaboration treats\ndisagreement about timelines and threat models as a feature, pre-registers what\nwould count as evidence, and resists letting institutional incentives quietly\nredefine what "safe enough" means.
\n","wordCount":113},{"heading":"Ethics","id":"ethics","markdown":"This is a field where the work is the ethics. Core duties: be honest about\ncapabilities and risks even when honesty is inconvenient to a product launch;\nexercise disclosure restraint on genuinely dangerous dual-use findings while\nresisting the temptation to hide inconvenient safety results behind it; and avoid\noverstating both safety (\"we've solved alignment\") and danger (\"imminent doom\"),\nbecause miscalibration in either direction erodes the credibility the field runs\non. Researchers carry responsibility for the downstream uses of the capabilities\ntheir work enables, and for people harmed today by bias, surveillance, and\nmisuse — not only hypothetical future failures. When commercial pressure pushes\ntoward shipping a model whose safety case is weak, the duty is to say so clearly\nand on the record, even at personal cost.","html":"This is a field where the work is the ethics. Core duties: be honest about\ncapabilities and risks even when honesty is inconvenient to a product launch;\nexercise disclosure restraint on genuinely dangerous dual-use findings while\nresisting the temptation to hide inconvenient safety results behind it; and avoid\noverstating both safety ("we've solved alignment") and danger ("imminent doom"),\nbecause miscalibration in either direction erodes the credibility the field runs\non. Researchers carry responsibility for the downstream uses of the capabilities\ntheir work enables, and for people harmed today by bias, surveillance, and\nmisuse — not only hypothetical future failures. When commercial pressure pushes\ntoward shipping a model whose safety case is weak, the duty is to say so clearly\nand on the record, even at personal cost.
\n","wordCount":127},{"heading":"Scenarios","id":"scenarios","markdown":"**A model passes the harmlessness benchmark but a tester is uneasy.** A new model\nscores well on the standard refusal eval and the team is ready to ship. The\nresearcher distrusts the clean number, asking whether the model is actually safe\nor has just learned to recognize the benchmark's phrasing. They build an\nout-of-distribution red-team set: the same harmful requests in obfuscated,\nmulti-turn, and role-play forms the benchmark never covered. The refusal rate\ncollapses from 98% to 60%. The conclusion is not \"the model is unsafe\" but \"the\nbenchmark measured benchmark-recognition, not harmlessness.\" The fix is both a\nbetter eval and an adversarial-training pass — and a note that the original\nmetric should never again be cited as a safety guarantee.\n\n**Deciding whether to publish a jailbreak technique.** A researcher finds a\nprompting method that reliably extracts dangerous synthesis instructions from\nseveral frontier models. Publishing would let labs patch it; it would also hand a\nworking attack to anyone who reads the paper. They apply the dual-use calculus:\nhow much real uplift does this give a motivated bad actor beyond what's already\npublic, and how much does disclosure help defenders? They choose coordinated\ndisclosure — privately notifying the affected labs, withholding operational\ndetails, and publishing the defensive findings only after patches ship. The\ndecision is logged and reviewed, not made unilaterally.\n\n**Interpreting an interpretability result.** A sparse autoencoder surfaces a\nfeature that activates on text about being evaluated and correlates with the\nmodel behaving more cautiously. It's tempting to announce \"we found the deception\nfeature.\" The expert resists the story and designs a falsification test: ablate\nthe feature to check whether behavior changes causally, and test on held-out\ncontexts to rule out a spurious correlation. Ablation moves behavior only weakly.\nThe honest report is \"we found a feature correlated with evaluation-awareness;\nits causal role is unclear\" — calibrated and falsifiable, more useful than the\nheadline would have been.","html":"A model passes the harmlessness benchmark but a tester is uneasy. A new model\nscores well on the standard refusal eval and the team is ready to ship. The\nresearcher distrusts the clean number, asking whether the model is actually safe\nor has just learned to recognize the benchmark's phrasing. They build an\nout-of-distribution red-team set: the same harmful requests in obfuscated,\nmulti-turn, and role-play forms the benchmark never covered. The refusal rate\ncollapses from 98% to 60%. The conclusion is not "the model is unsafe" but "the\nbenchmark measured benchmark-recognition, not harmlessness." The fix is both a\nbetter eval and an adversarial-training pass — and a note that the original\nmetric should never again be cited as a safety guarantee.
\nDeciding whether to publish a jailbreak technique. A researcher finds a\nprompting method that reliably extracts dangerous synthesis instructions from\nseveral frontier models. Publishing would let labs patch it; it would also hand a\nworking attack to anyone who reads the paper. They apply the dual-use calculus:\nhow much real uplift does this give a motivated bad actor beyond what's already\npublic, and how much does disclosure help defenders? They choose coordinated\ndisclosure — privately notifying the affected labs, withholding operational\ndetails, and publishing the defensive findings only after patches ship. The\ndecision is logged and reviewed, not made unilaterally.
\nInterpreting an interpretability result. A sparse autoencoder surfaces a\nfeature that activates on text about being evaluated and correlates with the\nmodel behaving more cautiously. It's tempting to announce "we found the deception\nfeature." The expert resists the story and designs a falsification test: ablate\nthe feature to check whether behavior changes causally, and test on held-out\ncontexts to rule out a spurious correlation. Ablation moves behavior only weakly.\nThe honest report is "we found a feature correlated with evaluation-awareness;\nits causal role is unclear" — calibrated and falsifiable, more useful than the\nheadline would have been.
\n","wordCount":325},{"heading":"Related Occupations","id":"related-occupations","markdown":"An AI safety researcher shares the empirical training of ML practitioners but is\ndefined by optimizing for what systems shouldn't do rather than what they can.\nMachine learning engineers build and scale the models safety researchers probe\nand constrain. Research scientists share the experimental method and the norm of\ncalibrated, peer-reviewed claims. Security engineers share the adversarial\nmindset, applied to learned systems instead of code and networks. Prompt\nengineers work the same model surface daily and surface many of the robustness\nfailures safety researchers study. Policy analysts translate technical risk\nfindings into governance and deployment rules.","html":"An AI safety researcher shares the empirical training of ML practitioners but is\ndefined by optimizing for what systems shouldn't do rather than what they can.\nMachine learning engineers build and scale the models safety researchers probe\nand constrain. Research scientists share the experimental method and the norm of\ncalibrated, peer-reviewed claims. Security engineers share the adversarial\nmindset, applied to learned systems instead of code and networks. Prompt\nengineers work the same model surface daily and surface many of the robustness\nfailures safety researchers study. Policy analysts translate technical risk\nfindings into governance and deployment rules.
\n","wordCount":97},{"heading":"References","id":"references","markdown":"- *Concrete Problems in AI Safety* — Amodei, Olah, et al. (2016)\n- *Risks from Learned Optimization* — Hubinger et al. (mesa-optimization)\n- *Superintelligence* — Nick Bostrom (orthogonality, instrumental convergence)\n- *The Alignment Problem* — Brian Christian\n- Anthropic, *Core Views on AI Safety*; *Constitutional AI* paper\n- *Concrete Problems* and the AI Alignment Forum (alignmentforum.org)","html":"