How duplicate photo detection actually works

Near identical grouping saves more culling time than any other check, and it is the one that behaves most strangely. Here is what a visual fingerprint really is, why capture time gets consulted before pixels do, and why your software keeps grouping forty minutes of speeches into a single pile.

Duplicate means three different things

Before any of the technology makes sense, separate three problems that share one word.

• The same file, twice. The import ran twice, or a backup got copied back. Two files, identical down to the last byte.
• The same image, two files. A RAW and its JPEG sibling. An export that came back from a client. Different bytes, same picture.
• The same moment, different frames. Six shots of the ring exchange fired in four seconds. Every file is unique. Every file is different. They are still one photograph as far as the gallery is concerned.

Only the third one is what culling software means by duplicates, and it is the only one of the three with no correct answer. The first two are arithmetic. The third is a judgment call the software makes on your behalf, using measurements that have nothing to do with what was happening in the room.

This piece is about how that judgment is made. If what you want is the practical routine for handling the results, our guide to cleaning up duplicate photos is the one to read. The broader set of checks is covered in how AI picks your best photos.

Why exact copies are the easy case

For byte identical files there is a clean answer, and it has existed for decades. The software runs the whole file through a checksum, a calculation that turns any amount of data into one short string. Same bytes in, same string out, every time. Two files with the same checksum are the same file.

What makes a checksum useless for photography is exactly what makes it reliable elsewhere: change one pixel and the string changes completely. There is no such thing as two checksums being close. A frame and the frame fired a fifth of a second later are as unrelated to a checksum as a wedding and a spreadsheet.

So the whole interesting problem is the third case, and it needs a measurement with a middle. Something that can say these two are ninety percent alike, rather than same or not same.

The visual fingerprint, in plain language

Here is the trick, and it is more direct than people expect.

Take the photo. Throw away the color. Shrink it down to something absurdly small, a grid of a few dozen squares a side, the size of a postage stamp seen from across a room. Now walk across that grid and write down one yes or no for each square: is this square brighter than the one next to it? The string of answers is the fingerprint.

Two photos that look alike produce nearly the same string. To compare them you count how many positions differ. Zero means the layouts are indistinguishable. A handful means the same scene with something small changed. A lot means unrelated pictures. The tool draws a line somewhere on that count and calls everything under it one group.

Shrinking is the point, not a shortcut

Reducing the photo that far destroys focus, grain, small movement, expression and detail. That destruction is deliberate. What survives is layout: where the light is, where the dark shapes are, roughly where the bodies stand. Two frames from the same burst survive identically, which is precisely what you want, because you want them grouped.

But it means the fingerprint is blind to every difference you care about. The frame where he is laughing and the frame where he blinked are the same photo to it. That is not a bug, it is the division of labor: grouping decides which frames belong together, and a completely separate set of checks decides which one wins inside the group. Those are the checks we pulled apart in blur and blink detection.

Newer tools add a second method. Instead of layout, the photo is converted into a long list of numbers describing content, a face, a white dress, a dark room, and similarity is measured in that space. It handles the case where you stepped sideways between frames far better. It is also happier to group two genuinely different moments that contain the same people and the same room, which is where a lot of surprising behavior comes from.

Capture time does most of the work

Before any pixel is compared, most tools sort by the capture time written into each file and only compare frames that fall inside a short window of each other.

The reason is arithmetic. Comparing every frame against every other in a two thousand frame wedding means close to two million comparisons. Comparing each frame only against its immediate neighbors in time means a couple of thousand. That is the difference between grouping that finishes while you make tea and grouping that does not finish at all.

An efficiency decision made in an engineer's head becomes behavior you feel on a Tuesday night. Two frames shot an hour apart are never compared, no matter how identical they look. And the clock in your camera is now load bearing.

Why it groups things you never would

Now the behavior that makes people distrust the feature. Every case below is the fingerprint working exactly as designed.

Notice the pattern. The software groups by what dominates the frame, and at a lot of shoots the thing that dominates the frame is not the subject. It is the room.

And why it splits things you would group

The opposite failure is quieter and costs more, because a group that never formed does not announce itself.

• The flash fired on one frame of the run. Brightness pattern changes, fingerprint changes, that frame stands alone.
• You rotated to portrait mid sequence. To a layout comparison, a vertical is a different picture entirely.
• You zoomed, or took two steps forward. Same moment, different geometry.
• Somebody walked between you and the couple. A large new dark shape rearranges the fingerprint.
• Slow sequences. One frame every thirty seconds during the vows falls outside the time window, so those frames are never compared with each other.
• Files renamed on import in a way that lost the original capture time. No time, no window, no grouping.

None of these are wrong answers. Each one is the measurement doing its job on inputs that no longer look alike. But the practical effect is that you will occasionally review the same moment three times, twenty minutes apart, and pick a frame from each. Then you deliver three nearly identical photos, which is a gallery problem rather than a software problem, and it is worth knowing that this is where it comes from.

The threshold is a preference, not a truth

Every tool draws a line on the similarity count, and where it draws that line is a taste decision with real consequences in both directions.

Imagen exposes this directly, letting you choose the grouping criteria based on how similar you want images to be before they are collected together. Narrative approaches it from the other end with Scenes, ranking every image in a scene by sharpness so the best frame surfaces first. Both are reasonable. Neither is correct, because correct depends on whether you are picking a wedding or a headshot day.

Which leads to the thing worth remembering about this entire feature. A group is a question, not a verdict. It is the software saying these six frames appear to be one moment, would you like to look at them side by side. That framing is the whole value, because people are poor at judging one photo on its own and very good at comparing two, which is also why picking photos is so hard when the frames are scattered through a list.

How we are approaching this at Kepla

We are building a picking app, so treat this as interested rather than neutral.

Grouping is where a phone earns its place. Deciding which of six near identical frames you like is a run of quick comparisons, and comparisons work fine on a phone on the drive home. Deciding whether the focus landed on the iris does not, so the same shoot opens on the Mac later and you carry on from there, with every frame still in place. That split is the point of phone versus computer culling.

And whatever a grouping algorithm concludes, nothing is ever deleted, moved or renamed. Kepla marks which photos it picked. The five frames it passed over in a group of six stay exactly where they were, under the same names, one tap away. Given how easily a fingerprint can put two different moments in one pile, a tool that acts on that conclusion is trusting its own arithmetic more than it should.

Status, plainly: the picking app for iPhone, iPad and Mac is still being built. What is live today is the Kepla booking page, free while we build.

FAQ