Eight ways to be wrong: testing one SDK on every framework

How one bug report turned into a cross-platform device lab, built with Claude, and what it caught.

One scenario, eight cells: the same ten API calls, judged the same way on four frameworks and two operating systems.

The Background Geolocation SDK for React Native, Flutter, Capacitor, Cordova and Swift/Kotlin now has a device laboratory behind it. Real apps for all four frameworks run on two Android emulators and two iPhone Simulators simultaneously. They call the public BackgroundGeolocation API and check what comes back.

It started with a black screen

On September 12, a Flutter developer reported an intermittent black screen on launch on Android [link to the issue]. It was a very good report. They had traced it to our plugin touching the app’s window from a background thread, and measured 6 black launches out of 15 in their own app. They also had a workaround.

I wanted to see it for myself. So I had Claude write the smallest app that could show the problem: a plain Flutter activity with a full-screen orange view, our plugin installed, and not a single SDK call. Then I had Claude drive the Android emulator. It launched the app hundreds of times, checked each launch against the app’s own view of its window, took screenshots five and twelve seconds in, and looked for rendered frames. A launch only counted as black when all of those agreed.

Two launches in a row, same build, five seconds in:

Where the lab started: the smallest app that could show the problem. Same build, consecutive launches.

We used the reporter’s plugin version on the same emulator image. Over 507 launches, some with the timing nudged to widen the race, it went black 15 times. Once, it crashed outright on launch, which the report hadn’t seen. (Two other developers later confirmed that crash from their own production data.) The cause was the one the reporter suspected. The plugin handed the Activity to the native SDK from a thread pool while Flutter was still building the window, and Android’s window setup isn’t thread-safe.

The fix keeps that work on the main thread, in both the plugin and the native core. Over another 500 launches across five builds, the fixed builds never touched the window off the main thread and never went black or crashed. The unfixed controls went black, crashed or built a duplicate window layout in 18 of 200.

That orange app is where this lab started. Reviewing the fix turned up a neighbouring problem with recreated Activities. When Android recreates the Activity for a configuration change, such as turning on Bold text, the SDK treated it as the app terminating. That got a second lab the next morning, and Capacitor and Cordova versions soon after. By September 18, those three labs, built in three separate sessions, had converged on the same design without sharing any code. So they became one harness, and React Native joined it the same day.

On September 21, I asked a bigger question: does every public method return what the types promise, on every framework and both platforms? A week after the orange screen, all eight combinations were running.

Eight ways to be wrong

Every one of those SDKs is a thin bridge over the same native core, TSLocationManager. Your JavaScript or Dart call crosses into native code, the core does the work, and the answer crosses back. Four frameworks on two operating systems make eight combinations, and every method crosses all eight.

A bridge can drop an argument. It can reshape a return value. It can drift from the TypeScript types or Dart API. Unit tests don’t see any of this, because the native tests exercise the core and the JavaScript tests mock the native side. The bug lives in the seam between them.

The conformance suite’s very first scenario proved the point. I asked each bridge to delete its stored locations. Two answered true, two answered nothing, and the types declared no return value at all. Today every one of those calls resolves true on all eight, and the types say so.

How it works

There’s one small app per framework. Each framework hosts the SDK differently, and that difference is exactly what’s being tested. Each app’s driver is written once, in JavaScript or Dart, and runs unmodified on both operating systems.

A scenario is a short YAML file: the calls to make, and what must come back. Here’s part of the one in the image above:

- api: insertLocation
  args: {uuid: lab-fixed-uuid-0001, timestamp: "2024-01-15T10:30:00.000Z",
         coords: {latitude: 45.5152, longitude: -73.6104}, extras: {source: lab}}
  returns: {$: {type: string, equals: lab-fixed-uuid-0001}}
- api: getLocations
  returns:
    $.length:            {equals: 1}
    $.0.coords.latitude: {type: number, equals: 45.5152}

A few rules keep it honest:

• The device never learns the expected answer. The app reports what came back, and the host compares. One scorer judges all eight cells, so no bridge grades its own homework.
• A missing method fails. Leaving a cell out takes a written exemption that starts with a date. The records literally read “Chris’s decision (2), 2026–09–24”. Without one, the scenario refuses to load.
• Real permission dialogs. The host watches the screen and taps the operating system’s own dialog, using uiautomator on Android and idb on iOS. It can also fail a step when a dialog shows up that nobody declared.
• Steps end on evidence, not on a timer.

Here’s the host working through the permission scenarios, in real time. On iOS it answers the location alert and then the upgrade to Always. On Android it answers the OS dialog, then our own backgroundPermissionRationale, then the settings page:

Real dialogs, answered by the host, in real time: two Android emulators and two iPhone Simulators.

The green and red on each screen are the host’s verdict, pushed into the app, so every screenshot is a readable record. A run of every scenario on all eight cells is what I call “the board”.

The host drives all eight cells and does all the judging. The device is told which calls to make, never what the answer should be.

Tests the types write

Our TypeScript types (and Dart API for Flutter) are the source of truth for the API. The lab’s vocabulary is seeded from them and checked against them: every method with its arguments and return type, every config key, and every State attribute. A scenario can't name something the types don't declare. It can't assert boolean on a method the types say returns nothing, either. So a typo can't pass for a bridge bug, and a real bug can't be written into the test as "expected".

Get Transistor Software’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

One line in a scenario, declared: both, expands into a type check for every State attribute in its scope. Three scenarios cover all 104 attributes: 73 on both platforms, 25 on Android only and 6 on iOS only. When the types gain a key, I re-seed the vocabulary, and the lab checks the new key on the next run. No scenario edit is needed.

Those generated checks found a regression on their very first run. On Android, getState() returned the background-permission rationale as an empty object, while the dialog itself used four built-in texts. Version 4 used to report them. All four Android cells were red. After the fix in the native core, the next sweep turned all four green.

It isn’t complete, and it says so. The lab calls 34 of the 41 public methods and subscribes to 7 of the 15 events, and the coverage report names the rest.

Red before, green after

Every fix goes the same way: a written diagnosis, my approval, the fix, and then the board before and after, wherever the lab can show the defect. Two examples:

• A partial setConfig reset its neighbours (Android). Setting only the location filter's policy reset every other filter setting to its default. The four Android cells were red and iOS was green. After the fix, all eight were green.
• A scheduled start reported “stopped” (iOS). onSchedule delivered the State from before the transition, so a scheduled start arrived with enabled: false. That is the very value the docs tell you to branch on. Every iOS cell stayed red until the core was fixed.

The same React Native iOS cell, the same assertion, before and after the fix.

Some reds stay red on purpose. Flutter’s onMotionChange hands you the Location itself, where the JavaScript bridges wrap it in an event. That matches our Dart, Swift and Kotlin docs, so it stays. The row stays red on the board, with the decision written into the scenario.

The lab also says what it can’t prove. The iPhone Simulator has no motion-activity support. The check that catches a bridge dropping requestPermission's argument therefore only works on Android. On iOS that takes a real device, and the scenario says so instead of reporting a green that means nothing.

Fast enough to actually run

A board nobody runs is useless. When I first measured, 70% of step time came after the last thing a step checked: 1,330 of 1,910 seconds.

So speed became its own piece of work. Steps end on evidence instead of a timer, a heartbeat proves the app is still alive, fixed sleeps became checks, and the lab now runs two Android emulators and two iPhone Simulators side by side. The first full board of all 15 scenarios took about 21 minutes. The same board now takes a little over 6:

The whole board in 377 seconds. The eight reds are known and written down; a dot is a scenario that doesn’t apply to that cell, with a dated reason.

Speed wasn’t allowed to change the answers. Every increment was checked on the devices, and every cell kept the same verdict. A trace comparison also confirmed that no event moved relative to the call that causes it.

How Claude built it, and what I kept

Claude, Anthropic’s AI model, running in Claude Code, wrote most of this: the harness, the scenarios and the written diagnoses. Every one of the 91 commits in the lab’s history lists Claude as co-author, and none went in without my approval.

The work ran as multi-agent workflows. Some agents surveyed the code. Others drafted competing designs, implemented against one contract, or reviewed the result. For each review finding, a “refuter” agent tried to disprove it. The original design came from 11 agents: a four-way survey, three independent designs, three judges and one synthesis. One review of a single change used 38 agents and confirmed 12 findings, all fixed before the commit.

The decisions stayed mine: which side of a disagreement is correct, what counts as a defect, and what we keep on purpose. Before a run that tests a change, the expected result is written down. That’s how the process catches its own mistakes:

• Before any code, a verification pass traced every planned call through the bridges. It found that the one failure I expected was really four. We fixed the plan, and the first run matched it exactly on every bridge it covered.
• A reviewer overruled an assertion. We’d called the Android and iOS difference in onSchedule "a platform fact". The reviewer pointed at the docs, and that's how the iOS bug above was found.
• The devices caught what reviewers missed. Two reviewers passed a change. Its first sweep showed an iOS-only scenario running on the Android cells. We fixed it and added a test that fails on the old code.

None of it runs unsupervised. It’s closer to a fast, thorough team that has to show its work, and an engineer who checks it.

Try it

A call you write in React Native should behave the same in Flutter, Capacitor and Cordova, on Android and iOS. When one drifts, the board is where we expect to see it first.

Install the plugin for your framework, and try it in your own app. Questions go to info@transistorsoft.com.