Build the Panic, Then Sell the Cure

There's a sentence in 404 Media's reporting on AI book-banning tools that I keep coming back to, because it describes a machine I've watched get built one vendor pitch at a time. It's from Laney Hawes, a volunteer with the Texas Freedom to Read Project: "All of this is not real — it's manufactured."

She's right, and the manufacturing has a supply chain. A state passes a vague law about "harmful" or "indecent" material. Districts panic about liability. Companies appear, selling AI scanners that promise a "defensible process." The scanners over-flag, that's what they're built to do, and the flags become paperwork, and the paperwork becomes removals. Then next year the same companies sell the upgrade. None of it requires a single actual instance of the harm the law names. The market runs on the fear of the harm, which is a renewable resource.

I want to walk through how that machine works, because the people running it are getting good at making it look like neutral compliance, like a librarian doing due diligence with a better tool. It isn't that. It's the censorship side learning the same trick the vendors learned years ago: turn a public institution's obligations into a recurring bill.

1. The tools don't do what they claim, and that's not the point

Start with what these things actually are. 404 Media documented a tool called BLOCKADE (it stands for "Blocking Lustful Overzealous Content, Keeping Away Depravity and Extremism," which tells you most of what you need to know) that takes a book file, runs it against a list of roughly 300 flagged words with severity scores, and asks a commercial chatbot (xAI's or OpenAI's) to rate the book's "appropriateness." The script defines "educational inappropriateness" outright as "content offensive to conservative values," and instructs the model not to explain its decisions.

Sit with that for a second from the desk side. A keyword tally is not a reading. Asking a model to score a book against "conservative values" doesn't define those values: it outsources the definition to whatever the model guesses they are. Jeremy Blackburn, who directs the Institute for AI and Society at Binghamton University, put the problem plainly to 404 Media: there's "a lot of responsibility being abdicated" when you hand a chatbot the job of deciding what counts as pornography with "no real thought into how to evaluate what comes out."

The output is bad in a specific, predictable direction: it over-flags. Even the people using these tools say so. And the bad output is the feature, not the bug, because the goal was never an accurate reading of a book. The goal is volume: enough flagged passages, formatted to look official, to attach to a formal challenge. Strip a sentence from its page, score it, print it on letterhead, and to a review committee that never read the book it looks like an expert finding. PEN America's Kasey Meehan named the through-line: take the shortcut, and "you're going to end up pulling so many books that tend to be the most targeted anyway."

The tool doesn't have to be right. It has to be fast, and it has to look like process.

2. The Utah case: own the format, then sell the scanner

The sharpest version of this market is in Utah. 404 Media reports that Rated Books, the operation run by activist Brooke Stephens, popularized the scored "this book is porn" report itself, the official-looking document built to be stapled to a challenge, and trained committees to expect a numeric verdict. Her own how-to video walks parents through the method: copy the pre-collected excerpts out of a Rated Books report, paste them into a chatbot, ask it to flag "brightline" violations, attach the output. No step asks anyone to read the book.

This year a Rated Books affiliate, the National Book Rating Index, began selling NarraTrue, a $5 AI scanner that outputs exactly that flagging input. And the accuracy problem is already on the record: Jessica Horton of the watchdog Let Davis Read told 404 Media she has won challenge appeals because committees leaned on Rated Books reviews that took books out of context. Known-unreliable, sold anyway. That gap is the tell that you're looking at a market, not a method.

It is also, on the Utah corporate record, a household business: the nonprofit that popularized the reports and the for-profit that sells the scanner trace to one Layton address, Stephens on the nonprofit, her husband on the company. That conflict, laid out from the filings, is its own filing on this site. Read it: Rated Books Isn't Just a Parents' Group. It's a Family Software Business →

3. The vendor case: sold to the district, by the district's own crisis

The other end of the market sells straight to schools. The Texas company BookmarkED pitches districts an AI "book intelligence" dashboard as the answer to their compliance liability under the state's new book laws. 404 Media obtained emails from the company to New Braunfels ISD promising a "truly defensible process," "a review process you can stand behind." The district fast-tracked a contract of nearly $9,000 while maintaining it was still "exploring." Per the Texas Freedom to Read Project, NBISD has removed more than 1,400 books to comply with the new laws, with new purchasing suspended indefinitely.

What makes this more than one district's bad call is that the staff using the tool could see it wasn't working. In a November 2025 email 404 Media obtained, an NBISD elementary librarian wrote that BookmarkED "seems that if it has any content at all, it flags rather than taking it as a whole." "Taken as a whole" is not a casual phrase: it's close to the actual legal standard these laws are supposed to apply. The librarian was describing, in real time, a tool failing at the one thing it was bought to do.

And the wider reporting backs her up. Texas Standard found BookmarkED had signed at least $130,000 in contracts across the state; librarians said the product "did nothing that it said it was going to do effectively"; and in eight districts with usage data, none had more than 0.04% of parents actually using the parent-facing features the law was sold to enable. A six-figure statewide spend to serve a fraction of one percent of parents, with the actual function being collection removal, not parent choice.

Meehan's warning to districts is the one I'd staple to every procurement file: be wary of the rent-seeking baked in, "if not for the 'grifty' energy these companies give off, then for the local decision-making power that's being abdicated to Silicon Valley":

"Your state passes harmful legislation that removes and censors books, and then you have companies appear that then want to charge districts to review their collections."

That's the whole machine in two sentences.

4. Why this rhymes with everything else here

If you've read the rest of the Investigations, this shape is familiar. The work this site mostly tracks is the lending side: who owns the systems public libraries use to lend digital books, and how that ownership turns a public good into a recurring bill. The censorship side is a different fight, but it's running the identical play: manufacture a scarcity (here, of "compliant" collections), then sell the public institution a subscription to relieve it. Same grammar, different object.

And both come down to the same thing: the binding constraint sits above the app, not inside it. On the lending side, that's publisher licensing and first-sale law. Here, it's the statutes: Utah's HB 29 "bright-line" rule, Texas's SB 13, and now the federal H.R. 7661, the "Stop the Sexualization of Children Act," introduced in February and advanced out of the House education committee, which would tie federal funds to removing "sexually oriented material" and writes gender dysphoria and "transgenderism" directly into that definition. The AI tools don't create the censorship. The laws create it; the tools just industrialize the paperwork and bill you for the privilege. Following the money runs up to the legislature, not just to the vendor.

Which is also why no amount of "but our AI is more accurate" fixes anything. A better scanner pointed at a worse law removes more books, faster. The tool's quality follows from a choice the tool is designed to launder: the choice to treat a whole book as the sum of its most-flaggable sentences.

5. What this looks like from the desk

This is a founding-document site, not an autopsy one, so here's the forward part: what a working library person can actually do when one of these reports lands on the reconsideration table.

Make the whole-book standard non-negotiable. Nearly every one of these laws says material is judged "taken as a whole." An AI flag list is, by construction, the opposite of that. When a challenge arrives backed by a scored report, the first question for the committee isn't "how many flags?" It's "did anyone read the book?" Horton's appeals in Utah won on exactly this ground. Out-of-context is not a vibe; it's a procedural defect you can name.

Treat "won't explain its reasoning" as disqualifying. Blackburn's point cuts both ways: a tool that's told not to explain itself can't be evaluated, and a flag that can't be evaluated isn't evidence. If a submitted report can't say why beyond a severity score, that's not rigor. It's the absence of rigor wearing rigor's clothes.

Read the procurement like a vendor contract, because it is one. If a "book intelligence" product comes through your district, the questions are the same ones we ask of any vendor here: who owns it, what does it do with your collection data, what's the recurring cost, and, the one these companies hate, what's the independent evidence it does what the pitch says? When the answer is "$130,000 statewide and 0.04% parent uptake," you have your answer.

Keep the reconsideration process human, slow, and on the record. The whole value proposition of these tools is speed: flag faster, remove faster, generate more challenges than any committee can carefully hear. The counter is not a faster tool. It's a process that refuses to be rushed, documents its reasoning, and puts a person who read the book in the room. That's not nostalgia. It's the part the machine can't manufacture.

The censors are reaching for AI, in Hawes's words, "to give themselves the illusion of control." The reporting suggests they're mostly buying that illusion from each other, at public expense. We don't have to buy it for them.

What's already reported, and what this adds

Most of the facts above are other people's work, and they deserve the credit. The tools, the BookmarkED contracts, and most of the quotes in sections 1 and 3 come from 404 Media's original reporting, cited throughout. That a family business sits behind Rated Books was flagged before this piece, too, by an independent newsletter, and by a local Michigan reporter who noted a "husband-and-wife team."

What this piece adds is documentation. The conflict of interest isn't an allegation here; it's on the filings: the Rated Books nonprofit and the for-profit S&S Apps that operates the National Book Rating Index trace to one household in the Utah corporate record. That paper trail, with the entity records laid out in full, is its own filing on this site. The point of pulling those records was simple: to move the story from "people are saying" to "here is the document."