Hell Difficulty: Optimizing Camera Performance on Mobile

The camera is the nastiest part of a phone to work with: dozens of frames per second, and every single one is fighting your CPU, GPU, and memory for resources.

This year I built two camera-heavy apps. Solid(ar)ity needs to scan QR codes and read the MRZ on passports (those two lines of gibberish at the very bottom of your passport). The other one, a pilgrimage app, overlays anime screenshots on the camera feed and does real-time alignment. Both are React Native + VisionCamera.

My takeaway after shipping them fits in one sentence: the camera path is played on Hell difficulty.

# First circle of hell: every frame is expensive

VisionCamera's frame processor lets you run code on every frame. Sounds lovely, until you do the math: at 30fps, you have 33 milliseconds to finish processing a full-resolution image. Can't keep up, and you get dropped frames, heat, and battery drain.

The fix: shrink it first.

Most analysis doesn't need full resolution at all. The pilgrimage app needs the frame's brightness and color temperature, so we just use a GPU resizer to downscale it to 64×64 and average that. For things like OCR that need detail, we only crop out the ROI (region of interest) instead of feeding in the whole frame.

If it can be done on the GPU, don't haul it back to the CPU. This one rule saved more performance than all the other random hacky optimizations combined.

# Second circle of hell: Xcode Cloud won't compile your shaders

The VisionCameraResizer we use for downscaling is a Metal shader. It ran perfectly locally, then fell over as soon as we built on Xcode Cloud, because the CI environment doesn't have the Metal toolchain.

The eventual fix was to precompile the shader into a metallib and commit it to version control with the project, so CI doesn't have to compile it on the spot. No error message is ever going to tell you "go precompile your shader" for this kind of problem; you just have to dig through it layer by layer. The catch, though, is that overall local build times got longer...

# Third circle of hell: crashes that won't tell you why

This is the real hell. Solid(ar)ity's passport MRZ scanner would crash at random on an iPhone XR (A12). The crash report had exactly one line:

abort() called

No exception type, no message, no JS stack. Nothing at all.

After a long hunt I finally dug up two culprits:

Culprit #1: console.log inside a worklet

The OCR worklet runs on VisionCamera's async-runner runtime, not Reanimated's UI runtime. This runtime doesn't have the global functions for JS scheduling, so:

• console.log(...) throws (under the hood it goes through scheduleOnRN)
• The exception bubbles up from runAsync, through Nitro, and straight into std::terminate, taking the whole app with it

Yes: you add one log line to debug, and the log itself is what crashes the app. It's the most absurd bug I've run into all year.

Culprit #2: Apple Vision throwing C++ exceptions on A12

When Vision's text recognition runs in .accurate mode on the A12's Neural Engine, it throws a C++ exception midway through inference, not an NSException. So Objective-C's @catch (NSException *) can't catch it at all; you have to use catch (...).

# How I climbed out

1. Make crashes introduce themselves

We install a global std::set_terminate handler at app launch. Any uncaught exception gets its type and message written into the crash report's Application Specific Information, plus an os_log entry. From then on the crash report no longer just says abort() called; it tells you exactly which function blew up. This trick alone turned debugging from hours into minutes.

2. Write worklets so nothing can possibly escape

The entire worklet body is wrapped in try/catch/finally. The catch and finally blocks only write to a SharedValue or swallow the error outright, and never call anything that might throw. Even frame.dispose() has to be guarded.

If you need to return results to JS, write them into a SharedValue and have the JS side poll it every 100 milliseconds. It looks crude, but it's the only channel that's safe across every runtime.

All I can say is, this is exactly where choosing React Native for native work hurts

3. Native modules shouldn't throw on every frame

When a Nitro method hits a transient failure (this frame isn't in focus, no text detected), it returns an empty result and lets the caller skip that frame. Throw across the boundary on every frame, and sooner or later one of those frames will terminate you.

# TLDR

1. The first rule of camera performance is to do less: lower the resolution, crop the ROI, and do it on the GPU whenever you can
2. Know which runtime your code runs on: the same JS function is fine on the UI runtime and kills the whole app on async-runner
3. Crashes need to be able to introduce themselves: set up your diagnostic tools first, then go bug hunting. Otherwise all you can do is stare at abort() called and try to channel the spirits
4. You absolutely need an old phone: the simulator and new phones won't catch any of this; the A12 is the one that shows things for what they really are. Good thing this developer is broke enough that an iPhone XR was all I could afford

The camera really is Hell difficulty, but after climbing out, my understanding of what's actually going on underneath React Native went more than one layer deeper.