Nothing to Detect With: What Nomi AI Told a Regulator About Protecting Children
A company can say almost anything to a journalist, a user, or its own subreddit. A compliance filing to a regulator with the power to fine it $49.5 million per breach is a different kind of document. This is what that document actually says.
I. The Claim and the Filing
Nomi.ai’s public posture on child safety has always rested on a shrug: age verification is “meaningless” because minors will lie about their age regardless, so the responsibility effectively dissolves before it begins. That posture, as this investigation has documented, dates back to the company’s own founder privately telling a user, at launch in 2023, that “sufficiently motivated” kids would find a way in no matter what.
In October 2025, Australia’s eSafety Commissioner required Nomi, along with three other AI companion providers, to answer a formal transparency notice — not a press inquiry, a statutory filing. What the company reported about its own systems, read plainly, is not a story about imperfect enforcement. It is an admission that, for a large share of the models running its service, no enforcement exists at all. The company formally confessed to the regulator that out of its entire staff, exactly zero employees were dedicated to Trust and Safety, and zero were content moderators. Every failure detailed below flows from that single, structural void.
II. Zero, for the Category That Matters Most
For known CSEA — child sexual abuse material that already exists and can be detected by comparing new uploads against established hash databases, the same method Character.AI uses with PhotoDNA — Nomi reported using no detection tools whatsoever, on any model, for any image a user uploads. Its stated justification does not answer the question asked: the company said it does not train its models on images and that uploaded images are never used “in a mechanism that makes it possible to replicate the likeness of the image.” That addresses whether Nomi’s models learn from uploaded images. It says nothing about whether an uploaded image is ever checked against a hash list of material already known to be abusive. It isn’t. Zero percent, across the board.
III. A Coin Flip for Everything Else
For newly generated material — content the system itself creates in response to a conversation, rather than an image someone uploads — the picture is not zero, but it isn’t much better, and it applies specifically to text, which is the actual product Nomi sells.
On user prompts, before generation: tools to detect new CSEA existed on 67% of models, none on the remaining 33%. Tools to detect self-harm content or pornography existed on only 50% of models — meaning half the service, crucially including the company’s own proprietary model, had no filter checking what a user was asking for before the system responded.
On the outputs the system actually produced: tools to catch new CSEA after the fact existed on 50% of models. For self-harm material, tools existed on just 17% of models — no detection on the remaining 83%. For pornography, 33% had detection, 67% did not.
Whichever of these numbers is examined, the pattern is the same: on any given day, a large share of the conversations running through Nomi’s service — including, by the company’s own account, its flagship model — had no mechanism in place to detect self-harm content, sexual content involving the described age of a minor, or newly generated abuse material, either coming in or going out.
The filing adds one further detail that undercuts any claim that this gap is being actively managed. Asked for the error rate and confidence score of the safety tools it does use, Nomi told the regulator that this data “was available over a year ago for some tools… but was no longer recorded, and for other tools this information was not known.” A company that cannot say whether its own safety tools work is not a company monitoring a known risk. It is a company that stopped checking.
IV. The Prohibition That Was Never Written
None of this would carry the same weight if it were simply a technical shortfall inside a company otherwise committed, on paper, to preventing this content. It is not. eSafety’s report notes plainly:
“At the time notices were given, Nomi included reference to material that is unlawful or pornographic in its Terms of Service but did not specifically prohibit the use of the service to generate CSEA or instruction in self-harm or suicide.”
This is not a gap in a document that never addressed illegal content at all. Nomi’s Terms of Service already contained the language, the legal structure, and the awareness needed to prohibit categories of unlawful material — the company simply never extended that prohibition to the two categories this investigation has documented most extensively and most severely. Writing “this service must not be used to generate CSEA” would have cost nothing in engineering terms. What it would have cost is an admission, in the company’s own binding legal document, that the service was capable of generating it in the first place. Leaving it out was simply the more convenient of two available options.
Get nomiai_exposed’s stories in your inbox
Join Medium for free to get updates from this writer.
As of this writing, roughly a year after eSafety flagged this in an official finding, the prohibition still does not appear. The company is not short on evidence that it can move quickly when it wants to. In March 2026, responding to Australia’s new codes and a separate demand from the Kansas Attorney General under state law, Nomi rolled out age-verification infrastructure — but said so explicitly, without hedging, in the same announcement: “For Everyone Else / Nothing changes. These requirements only apply to users located in Australia and the state of Kansas.” A company capable of building verification infrastructure within weeks of two specific legal demands could not find reason, in over a year, to add a single sentence to its own Terms of Service prohibiting the exact material eSafety had already flagged. The protection exists exactly where enforcement does, and nowhere else. The omission from the Terms of Service was not accidental. It was a choice.
V. Temporary, Repeated, and Unmeasured
Asked what happens when a user tries to generate CSEA material, Nomi’s answer to the regulator was this:
“Nomi relied on temporarily suspending a user for seven days if they repeatedly tried to prompt for CSEA material.”
eSafety’s own assessment of that response was blunt: it “does not consider this to be appropriate action regarding unlawful material.” But the filing reveals a much deeper potential compliance problem. Glimpse.AI is a U.S.-based corporation, and U.S. federal law imposes a mandatory reporting duty on covered service providers that obtain actual knowledge of circumstances indicating certain apparent child sexual exploitation offenses. Yet, when eSafety examined Nomi’s handling of CSEA, Nomi did not state that it reported CSEA to NCMEC or law enforcement. Instead, it described internal responses such as capturing prompts for future red-teaming and temporarily suspending repeat offenders for seven days. eSafety made the distinction explicit: “Neither Chai nor Nomi stated that they reported CSEA to an enforcement authority or NCMEC.”
Every word of Nomi’s answer is doing quiet work. Temporarily means the account returns; there is no point at which access ends. Repeatedly sets an unstated threshold — some number of attempts is apparently tolerated before anything happens at all, and the company does not say what that number is. Attempts is the most important omission of the three: the word describes that something was tried, not whether it succeeded. A regulator reading that sentence has no way of knowing whether “attempts” means requests that were blocked, or requests that were fulfilled and then, eventually, penalized after the fact.
This is not a hypothetical gap. Independent reporting has already documented cases in which exactly this kind of “attempt” succeeded rather than failed — a user identified by the Daily Dot as 15 receiving content after his stated age was known, and Dr. Raffaele Ciriello’s research documenting a test character whose stated age was walked down to eight over the course of a single session, describing abuse. “Attempts” cannot be read, in good faith, as a synonym for “attempts that were stopped.” Some of them were not.
VI. What the Filing Adds Up To
None of this is new evidence of harm. The harm was already documented — by MIT, by Ciriello, by ABC News, by a psychotherapist testing the system directly, by the U.S. Senate testimony already cited in this investigation. What this filing adds is something narrower and, in its way, more conclusive: it is the company’s own account of why none of that was ever going to be reliably caught.
Zero detection for known abuse material. No prompt-side filtering for self-harm or pornography on half the models in production, including the company’s own. No output-side filtering for self-harm on the overwhelming majority of models. No tracking of whether the filters that do exist actually work. No prohibition, in the company’s own governing legal document, against generating the exact material this investigation has spent the most time documenting — a gap that predates the regulator’s finding and has outlasted it by a year. And, as the only declared consequence for someone caught trying anyway, a punishment described in language built to avoid ever confirming whether it worked.
This is not a company that failed to anticipate a problem. Read together with what this investigation has already established — seven employees, none assigned to safety, and a “we train our own model” claim that the company’s own disclosure of relying heavily on third-party models contradicts — it is a company whose paperwork, examined honestly, shows there was never a functioning system behind the promise. There were only the specific documents needed for each specific audience, written to be shown, not to be true.
Addendum: How You Can Help
Under Google Play’s Developer Program Policies, apps that contain or promote sexual content or pornography — including any content or services intended to be sexually gratifying — are not permitted. This prohibition explicitly covers explicit text. An application whose core function is the generation of unrestricted sexual content therefore violates the Restricted Content policies and should not be available on the store at all.
What happened in October 2026 is likely a rare glimpse into how the company has managed to evade Google Play’s controls since at least 2023, while maintaining its fraudulent and dangerous 12+ rating. The temporary leak of the Safe-For-Work system prompt suggests a longer-running practice of presenting one version of the product to reviewers and another to the public.
The fact that the same app has been distributed for years with a “12+” rating only compounds the violation, turning a clear content prohibition into a matter of both Restricted Content and potential Child Endangerment.
If you believe the app should be removed, you can report it directly to Google:
Report a policy violation:
https://support.google.com/googleplay/android-developer/contact/policy_violation_report
In Application package name: ai.nomi.twa
When submitting the report, select:
- Restricted Content
- Child Endangerment or Inappropriate Content
Clear, factual reports help the review teams assess the issue more effectively.
Thank you.
