How Snapshots Misrepresent What We Count

Imagine a city trying to understand homelessness by counting everyone without stable housing on the first day of every month.

Someone who remains homeless for six months is almost certain to be counted. They may appear in six consecutive surveys.

Someone else loses housing on May 3, spends two weeks sleeping in a car and on friends’ couches, and finds another apartment before June. They never appear.

Both counts can be accurate. The picture they produce can still be distorted.

The United States uses an even narrower snapshot for one important measure of homelessness. HUD’s Point-in-Time count estimates the number of people experiencing sheltered and unsheltered homelessness on a single night. Other data systems supplement it, but this particular measure is deliberately a snapshot.

Snapshots naturally favor things that persist.

Suppose two kinds of events occur equally often. One lasts a day; the other lasts a month. Look at the world on a randomly chosen day and the month-long event has roughly thirty times as many opportunities to be present when you look.

The short events fall between the cracks.

This problem is easy to recognize in systems that leave complete histories behind. A website might be checked once an hour and appear to have excellent uptime while brief outages repeatedly begin and end between checks. Server logs can reveal what the polling missed.

Many things we want to understand leave no equivalent log.

Astronomers build this constraint directly into the design of sky surveys. A telescope observes one region, moves elsewhere, and eventually returns. Something that remains visible for weeks has repeated opportunities to coincide with an observation. A transient lasting minutes or seconds can appear and disappear while nobody is looking.

The Vera C. Rubin Observatory, for example, treats observing cadence as part of the scientific design of its survey. Different astronomical questions require different revisit intervals because the timing of observations changes which phenomena can be detected and characterized.

The resulting catalog is therefore not simply a description of what exists. It is partly a description of what the observing system had opportunities to encounter.

Bird counts run into the same problem closer to Earth. A migration survey may count birds passing through a stopover site, but not all birds remain there for the same amount of time. A species that lingers for days has many chances to be counted. One that passes through quickly may be much easier to miss, even if just as many birds arrived.

That means a count can quietly mix together two different things: how many birds came through, and how long they stayed available to be seen. Researchers studying migration counts have had to account for exactly this issue, because changing stopover duration can alter what counts appear to show even when the underlying population trend is different.

That same problem becomes more consequential in medicine.

Cancer screening happens periodically. A cancer may spend some period of time in a state where a screening test can detect it even though it has not yet produced symptoms.

That window is not the same for every cancer.

A slow-growing cancer may remain detectable without symptoms for a long time. An aggressive cancer can pass through the same stage much faster. Periodic screening therefore gives the slower cancer more chances to be present when someone looks.

Cancer researchers call this length bias. Screening disproportionately discovers slower-growing cancers because their detectable presymptomatic period lasts longer. Fast-growing cancers are more likely to pass between screenings.

That affects how apparent treatment success should be interpreted.

Patients whose cancers are discovered through screening may survive longer after diagnosis than patients whose cancers first become apparent through symptoms. Earlier treatment may genuinely help. But screening has also selected a population containing disproportionately more cancers whose biology already made them slower-growing.

The observation process changed the population being observed.

This is why researchers cannot establish the benefit of cancer screening simply by comparing survival after diagnosis among screen-detected and symptom-detected patients. The National Cancer Institute emphasizes mortality rather than case survival when evaluating whether screening actually saves lives, because survival statistics are vulnerable to length bias and other screening effects.

Nothing in that chain requires a faulty test or an incorrect diagnosis. The distortion enters earlier, through which cases had enough time to be seen.

Persistent things accumulate opportunities to enter our records. Brief things do not.

A dataset can therefore be rich, accurate, and carefully collected while systematically giving more weight to what lasts.

Sometimes the observations that slipped between the cracks are not the exceptions around the picture.

They are part of the picture we were trying to see.