The same result twice: Debugging an infinite scroll that duplicated, skipped, and died

Three sentences, three bugs

The bug report was three sentences long and, as it turned out, described three different bugs.

Users on a filterable result list were occasionally seeing the same entry twice as they scrolled. Support had also noticed the total count didn’t always match what you could actually scroll through. And once in a while, scrolling simply stopped loading anything, no spinner, no error, no network request, until the page was refreshed.

None of it threw. Nothing appeared in error tracking. Each symptom had a separate root cause, and the most interesting one had nothing to do with the front end at all. The stack: a React front end with an infinite-scroll list, the first page rendered server-side, subsequent pages fetched from an API endpoint, backed by a key-value store.

The symptom that made no sense

Duplicates in a paginated list usually mean an off-by-one in the offset arithmetic. That was the first thing checked, and the arithmetic was fine: page one requested offset 0, page two requested offset 8, the page size was 8 everywhere.

The tell was that a duplicate never appeared alone. Every time an entry showed up twice, a different entry was missing entirely. One in, one out. That is not an off-by-one, which shifts a boundary and shows you a neighbour twice. This was a swap across a page boundary between two requests, which meant the two requests were not looking at the same list.

Root cause 1: the sort was not a total order

The results were ranked by four business criteria in sequence: a status field, then whether the entry had an image, then account tier, then whether an optional field was filled in. Each compares a small number of discrete values, which means enormous tie groups: hundreds of entries can be identical on all four. And when the comparator returns 0, the sort leaves those entries in whatever order they arrived in.

That input order came from the database. Nothing in the query guaranteed it, and it was not stable between the server-side render of page one and the API call for page two. So the two requests each produced a correctly sorted list, with the tied entries in different positions. Slicing offset 8–16 out of a list ordered slightly differently from the one that produced offset 0–8 gives you exactly the observed symptom: something duplicated, something skipped.

Offset pagination is only correct if the sort defines a total order. If two rows can compare equal, their relative position is undefined, and “undefined” is free to differ between two requests for two pages of the same list. The fix is one line, a final tiebreaker on a unique, stable key:

return primaryCriteria || /* … */ || (a.key || "").localeCompare(b.key || "");

Every request now produces a byte-identical ordering, so offset boundaries line up exactly.

This is not a quirk of one database. ORDER BY created_at LIMIT 20 OFFSET 20 in Postgres has precisely the same defect the moment two rows share a timestamp. It just fails rarely enough to look like a ghost.

The complication: fairness wanted the opposite

What made this more than a one-line fix is that the product deliberately randomises the order of equally-ranked entries. Always showing a hundred tied listings in the same order permanently advantages whoever sorts first, so the list shuffles within each tie group to give them a fair rotation across visits.

So the system had one requirement demanding a deterministic order and another demanding a random one. The original code satisfied the second by accident of the first being absent, which is why nobody noticed the conflict until pagination broke. The resolution is to separate the two concerns by layer:

• The server orders deterministically. Total order, unique tiebreaker, identical for every request. This is what pagination arithmetic depends on.
• The client rotates within tie groups, after the page arrives. Fairness is a presentation concern applied to the eight items you just received, not an ordering concern applied to the underlying list.

Randomising presentation is safe because it never moves an item across a page boundary. Randomising the source order is not, because that is the thing offsets are measured against.

One aside: the shuffle was the familiar array.sort(() => Math.random() - 0.5) idiom, whose non-transitive comparator makes the distribution engine-dependent and measurably biased. For a fairness rotation that matters, and Fisher–Yates is the same number of lines.

Root cause 2: the observer that stopped observing

The second symptom, scrolling that silently stopped loading, was unrelated.

Infinite scroll was driven by an IntersectionObserver watching a sentinel element at the bottom of the list. The observer was constructed once, in an effect with an empty dependency array, and stored in a ref; a second effect attached it to whatever element was currently held in state.

That works as long as the sentinel never goes away. But it is conditionally rendered, existing only while there are more results to load, and applying a filter replaces the entire list. The sentinel unmounts, remounts, and nothing re-observes it. The observer is still alive, still holding a reference to a node no longer in the document, and quietly never fires again. No error, no request, infinite scroll simply dead until a refresh.

The fix is to stop treating “observe the sentinel” as a two-effect dance and make it a callback ref, so mount and unmount drive the subscription directly:

const setSentinel = useCallback((node) => {
  observerRef.current?.disconnect();
  observerRef.current = null;

  if (!node) return;

  observerRef.current = new IntersectionObserver(
    (entries) => { if (entries[0].isIntersecting) loader.current(); },
    { threshold: 1 }
  );
  observerRef.current.observe(node);
}, []);

React calls this with the node on mount and with null on unmount, so the observer’s lifetime is tied to the element’s, which is what you meant in the first place. The general rule: if a DOM node is conditionally rendered, any subscription to it belongs in a callback ref, because an effect keyed on state holding that node will miss remounts.

Root cause 3: a reset that only fired when the value changed

The third symptom looked identical to the second from the outside and had a completely different cause.

A counter tracked how many results remained to load, and the sentinel only rendered while it was above zero. That counter was reset in an effect keyed on the incoming count. Scroll to the end of a result set and the counter sits at zero. Now apply a filter producing a different set that happens to have the same remaining count: the prop’s value never changes, so the effect never fires, so the counter stays at zero, so the sentinel never renders, on a list with plenty more to load.

The dependency encoded “when this number changes” when the intent was “when the data changes.” Resetting on the result array instead of the count fixed it. That is the general hazard with derived state: mirror a prop into state and a value-identity dependency will silently skip resets whenever the new value coincides with the old one.

Conclusion

Three bugs, one symptom class, not a single stack trace between them. Each was an invariant nobody had written down: that two requests for adjacent pages see the same ordering, that a subscription lives exactly as long as its element, that a reset fires when data changes rather than when a number does.

The debugging lesson was in the one detail that seemed like noise: a duplicate always came with a disappearance. An off-by-one duplicates a neighbour; a swap means the two requests disagreed about the list. That observation pointed straight past the front end to a sort comparator, and everything else followed from asking why two calls to the same function returned the same items in a different order.