One Photon, Many Worlds: Posteriors and Predictions with Single-Photon Cameras

Haejoon Lee1, Mohit Gupta2, Vijayakumar Bhagavatula1, Aswin C. Sankaranarayanan1
1Carnegie Mellon University   2University of Wisconsin-Madison

arXiv preprint

Paper (arXiv) Code (coming soon) Video (coming soon) BibTeX

Swipe sideways to see all columns.

Real SPAD capture. Each column: the measurement accumulated from Nk binary frames (Nk = 0 to 1000, left to right) and posterior samples generated by our model from it. With few frames the samples show different faces; with more frames they agree.

Abstract

Single-photon avalanche diode (SPAD) cameras operate fundamentally differently from conventional cameras due to their photon-counting nature. Each frame produces a binary image: pixels report zero if no photons arrived during exposure, and one if one or more photons arrived. Reconstructing a scene or inferring its properties from a single binary frame is difficult because many different images could produce the same measurement; thus, the inverse problem is fundamentally one-to-many. As we gather more binary measurements, the inherent uncertainty associated with the inverse problem and any associated inference diminishes. With sufficient photon counts, photon noise becomes negligible relative to the signal mean, enabling near-deterministic scene recovery and inference. This work characterizes the transition from stochastic to near-deterministic scene understanding as photon budget increases, analyzing how the stochasticity in photon arrival affects downstream inference tasks. Technically, we develop a conditional generative framework based on a Hypergeometric frame-thinning process for accumulated binary SPAD measurements. Generative models capture the one-to-many nature of photon-starved inverse problems, enabling empirical characterization of how this ambiguity diminishes with increasing measurements and its impact on downstream tasks like character recognition, QR code decoding, and facial analysis.

One measurement, many images

Swipe sideways to see both panels.

Simulated. SPAD measurement of an FFHQ face image (left) and 16 posterior samples generated by our model (right), as the number of binary frames grows from 0 to 10,000.Face image from FFHQ (Karras et al., 2019), converted to grayscale and resized to 64 × 64.

A SPAD pixel reports one if one or more photons arrived during the exposure and zero otherwise, so each frame is a binary image. From a single binary frame, many different images could produce the same measurement: the inverse problem is one-to-many. As more binary frames are gathered, this uncertainty diminishes; with enough photons, recovery and inference become near-deterministic.

For a deterministic denoiser trained with an L2 loss, the risk minimizer is the conditional mean, which is blurred when the conditional distribution is multimodal. We therefore treat restoration in the photon-starved regime as a conditional image generation problem.

Thin frames, then learn to add them back

Diagram: a high-count face image, two intermediate SPAD images at N_t and N_t+1 frames linked by the hypergeometric thinning step above and the learned reverse step below, and an empty image at zero frames

One step of the chain. Frame thinning keeps a uniformly random Nt+1 of the Nt binary frames, so each pixel's retained count is hypergeometric (top arrow); the learned reverse chain adds frames back (bottom arrow). Images are simulated SPAD measurements.Face image from FFHQ (Karras et al., 2019), converted to grayscale and resized.

  1. Thin. Discarding binary frames is a Markov chain with hypergeometric transitions.
  2. Keep the physics. Thinning preserves the Binomial marginal at the new frame count (Theorem 4.1).
  3. Add frames back. A latent Binomial model predicts a photon-detection probability map and samples the missing counts.
  4. Few anchors. Sampling runs along a short chain of anchor frame counts, with one shared pairwise model.

Real SPAD captures

We captured a printed digit grid and face images with a Pi Imaging 512 × 512 SPAD camera. Hot pixels are suppressed (for the digits, using a dark calibration capture); the digit grid is rectified and split into 28 × 28 patches, and the face measurements are downsized to 64 × 64.

Swipe sideways to see all columns.

Real SPAD capture of the same person without sunglasses (Nk = 0 to 1000 binary frames), with face landmarks drawn on each posterior sample generated by our model.

MeasurementNk = 0 framesPosterior samplegenerated by our modelTargethigh-count capture
Real digit grid: measurement at 0 frames, one generated grid, and a high-count capture of the same grid

Real capture of a printed digit grid at Nk binary frames. The posterior sample is one grid generated by our model from the measurement; the target is a high-count capture of the same grid.

How many frames until the answer is stable?

Posterior samples

Simulated edge measurements from 0 to 250 frames with generated samples, mean, and standard deviation

Simulated. Each column fixes one measurement at Nk frames, followed by generated samples, their mean, and their normalized standard deviation (false color). Paper Fig. 6.

Downstream checks

Simulated: plots of median angle error and median bias MSE against the number of frames

Simulated. Edge prediction on generated samples against Nk; the red dotted line is performance on high-count (N1 = 10,000) images. Paper Fig. 8.

Citation

@misc{lee2026photonworldsposteriorspredictions,
      title={One Photon, Many Worlds: Posteriors and Predictions with Single-Photon Cameras},
      author={Haejoon Lee and Mohit Gupta and Vijayakumar Bhagavatula and Aswin C. Sankaranarayanan},
      year={2026},
      eprint={2610.02675},
      archivePrefix={arXiv},
      primaryClass={eess.IV},
      url={https://arxiv.org/abs/2610.02675},
}