What fraction of a video corpus is content it already contains?

What already exists 01

Near-duplicate detection is not an open problem. Before any of the below is useful, it is worth knowing what ships today:

To deduplicate an image dataset, use one of those. What follows is narrower: the same measurement split by who captured the footage, two gates that refuse to report a figure the embedding cannot support, and the confounds that turned up on the way — published rather than omitted.

The measurement 02

A frame is redundant at threshold t if its cosine similarity to any earlier frame in the corpus is at least t.

That definition is computed threshold-independently, which matters more than it sounds. One pass records each frame's best match against everything preceding it, and the entire curve derives from that single array. The obvious alternative — greedily keeping or dropping frames — has to be re-run per threshold, and its answer depends on the order in which drops cascade. That is a hidden knob, and a number resting on a hidden knob does not survive a skeptic.

At a conservative threshold, 27.2% of frames duplicate earlier content — 95% CI [24.6%, 29.9%]. Redundancy is almost entirely self-redundancy: within-worker measures 27.2% and the global figure is also 27.2%. Workers repeat themselves. They do not duplicate each other.

corpus redundancy

Share of frames whose cosine similarity to any earlier frame in the same corpus meets a stated threshold.

operator spread

Ratio between the highest and lowest per-operator redundancy in a corpus. Measured here at 6.2×.

marginal novelty

Share of a clip's frames that are novel against everything that operator has already delivered.

On that interval. The obvious tool — bootstrap over clips — is invalid for this statistic. Resampling with replacement puts the same clip in the corpus twice, and this metric measures duplication, so every repeated clip scores as a perfect match against its own copy. The first attempt returned a 95% interval of [49.2%, 53.4%] around a 27.2% estimate: an interval that cannot contain its own point, because the method manufactured what it was measuring. The figure above is a jackknife, which never duplicates anything.

The average is the least useful number here 03

Redundancy ranges from 7.4% to 46.0% depending on which operator's footage you bought — a 6.2× spread around the corpus mean.

That spread is roughly 39 points wide, an order of magnitude larger than the ±2.7-point sampling error on the mean. Which means the corpus average — the number this whole exercise set out to produce — is the wrong unit of analysis. A buyer paying a flat rate per hour is exposed to whose footage they received far more than to any property of the corpus.

The commercial implication is not "audit corpora." It is price per operator.

Does buying more hours stop paying? 04

The natural follow-up: if an operator repeats themselves, there should be a point where their next clip adds nothing. So — for each clip in collection order, what share of its frames are novel against everything that operator has already delivered?

It does not collapse. Novelty falls from 100% to roughly 75% within the first ten clips and then holds there — clip 50 is still about as novel as clip 15. Across the whole sample, 79.2% of frames are novel against their operator's earlier output.

This was not the expected result, and it changes the advice. There is no saturation point to stop at within this volume. Redundancy behaves like a steady tax of roughly a quarter, not a cliff — so the question for a buyer is not when to stop buying but what rate they are paying, which loops back to the operator spread above.

Motion blur is a confound, not noise 05

This was not in the plan. It surfaced from rendering sampled pairs and actually looking at them — and it is the reason the sensitivity sweep below is published alongside the headline rather than produced on request.

0.956 · same worker, 8 seconds apart genuine duplicate

Unambiguous. Same bench, same paper, same hands, same part.

0.906 · same worker, different clips blur artifact

0.872 · same worker, different clips genuine duplicate

The inversion. The 0.906 pair matches on shared blur — two fast camera sweeps smeared into green. The 0.872 pair is unmistakably the same workstation, same parts bin, same task. The metric ranked the artifact above the real repeat.

Blur destroys the detail that distinguishes frames, so blurred frames collapse toward each other in embedding space and register as duplicates of one another. Frames are now scored for sharpness — variance of the Laplacian — and the blurriest fraction is dropped before measurement.

Filtering blur raises measured redundancy, because it removes low-information noise rather than real repeats. A filter that moves the headline in the flattering direction is exactly the kind of choice a reader should distrust, so here is the whole sweep:

Blur share is independently useful. Frames too blurred to distinguish are also too blurred to train on, whether or not they repeat.

Does the embedding see this footage? 06

The obvious objection: CLIP was trained on web photography, not fisheye industrial video. If factory interiors collapse into a narrow region of the embedding space, everything looks alike and the redundancy figure is measuring a blind spot rather than a corpus.

The corpus answers this itself. It spans 85 physically distinct factories, which is a free control: frames from different plants must be clearly less similar than frames from the same worker. Six factories were sampled to test it.

The pass criteria were written into validate.py before the gated data was accessible: separation above 0.10, and under 1% of cross-factory pairs clearing the threshold. Measured: 0.155, and 0.00%. Not one pair in twenty thousand.

A sharper version of the same question

Mean separation is a blunt instrument. The direct test: if the embedding had collapsed this footage, it could not tell workstations inside a single factory apart. So — can a linear classifier read a frame's embedding and name the station it came from?

This is arguably a harder test than the cross-factory one: stations inside one plant resemble each other far more than two different plants do. Passing it at 92.9% says the embedding carries fine-grained discriminative structure on exactly this footage — which is the property the redundancy metric depends on.

What remains open. Each worker has their own fisheye intrinsics, so part of that signal may be lens rather than scene. And validating the method is not the same as validating the number: 27.2% is what factory 051 measures, and whether other plants land near it is a separate question this does not answer.

What this does not show 07

• One factory. Factory work is also the easy case — repetitive manual labour should produce repetitive footage. This says nothing about kitchen or household corpora without re-running.
• Fisheye is uncorrected. Per-worker intrinsics differ, so cross-worker comparison is confounded. Within-worker numbers are the trustworthy ones.
• Keyframe sampling ties sample rate to encoder settings rather than a fixed interval. Fine for scene-level redundancy; wrong for anything motion-sensitive.
• Redundant is not worthless. Repetition carries real signal for robustness, and failure footage is valuable for recovery training. The argument is that buyers should be able to price redundancy — not that it should be deleted.

None of this is a criticism of the corpus or the people who built it. Repetitive manual work produces repetitive footage; that is a property of factories, not a failure of whoever pointed the camera. Egocentric-10K is open, documented and Apache-2.0, which is the only reason this analysis exists at all.

Questions this answers 08

Reproduce it 09

Runs on a GTX 1050 Ti in about an hour, network-bound. Nothing is downloaded: shards stream from the Hub, frames decode in memory, only embeddings persist.

git clone https://github.com/eitanp214/episcore.git
cd episcore
pip install -r requirements.txt
python src/pipeline.py --tag myrun --source subset

Source, tests and full method notes: github.com/eitanp214/episcore

The validation gate blocks the report if the embedding fails to discriminate on your footage. That is deliberate, and not skippable from the entry point.