Safe Is the Floor, Not the Goal

An AI product can pass every safety check and still leave people worse at thinking, and most companies only run the first test.

Two different tests

A recent MIT Technology Review op-ed by Sam Hiner and Emma Leiken makes the case at the policy level. Lawmakers are passing industry-backed bills that guard against the worst outcomes (a chatbot that won't encourage self-harm, say), while the quieter damage goes unmeasured. The authors point to MIT research in which young people using ChatGPT to write essays showed weaker brain connectivity, with "cognitive debt" that lingered after the task. The authors' broader argument is that companies can clear those low bars and still build products that quietly change how people think and relate to each other, and that safety should be the minimum, not the finish line. The long-term effects are still being studied, and that work is about teenagers and students. But I keep asking whether our own AI policies do the same thing: define success as "no incident" and call it done.

Start with the floor, then write the ceiling

Step one: look at your internal AI policy and ask what it protects against. Data leakage, bad outputs, and misuse are the floor. Step two: write down what you want people to still be able to do in two years without the tool (draft a client proposal, debug an outage, tell a good idea from a plausible one), and measure that. If it is not written down, it is not measured, and a skill you do not measure is one you can lose without noticing.

Zero incidents is not the same as success

A team with zero incidents and no remaining judgment has not succeeded, it has just not been caught yet. The op-ed's authors put the underlying point well: we build judgment through effort and friction. A tool that removes all of that is making a trade someone should choose on purpose.

What to do

What does your AI policy say about the skills you refuse to lose?