How to find and delete duplicate files on a Mac

Where duplicates come from

Nobody copies a file on purpose and forgets. Duplicates arrive sideways: a PDF downloaded again because the first one could not be found, so the browser names it -1 or (1); an attachment saved from Mail and then saved again from Messages; a video exported to Desktop and to Documents; an AirDrop that was already on the Mac; a photo dragged out of Photos to send and left in Downloads afterwards. Each one is as big as the original, and none of them looks like a duplicate from its name, because the name is usually the thing that changed.

Why Finder cannot find them

Finder compares names, dates and sizes, never contents. A Smart Folder sorted by size will put two identical files next to each other if they also happen to share a name, and nowhere near each other if they do not. It will also put two different files together because they are both called invoice.pdf. Neither is a duplicate check. The only test that works is the contents: two files are copies when every byte matches, whatever they are called.

Photos: the Duplicates album

Inside the Photos library, Photos does the work itself. Since macOS Ventura there is aDuplicates album in the sidebar (below Utilities in current versions), which lists the copies it has found in rows. Select a row, or press Command-A for all of them, and click Merge: one photo or video stays where the copies were, and the others go to Recently Deleted, where they can be recovered for 30 days. On a large library the album can take a while to fill, because Photos has to analyse the photos first. Do not go looking for the copies in the library's folders in Finder; that is the one place where deleting a file by hand loses photos. If the Duplicates album is missing, hold the pointer over Utilities in the sidebar and click the arrow; if you use iCloud Shared Photo Library, choose Both Libraries from the Library pop-up menu in the toolbar first, so copies in either library are found. The rest of what the library holds is covered in the Photos library is taking up space.

Everywhere else: one Terminal command

For files in your own folders, compare contents by hashing them. This lists every file over 1 MB in Downloads, Desktop and Documents whose contents match another file's, grouped so the copies sit together:

find ~/Downloads ~/Desktop ~/Documents -type f -size +1M -exec md5 -r {} + 2>/dev/null | sort | awk 'c[$1]++ == 1 {print p[$1]} c[$1] > 1 {print} {p[$1]=$0}'

Each line is a hash and a path. Lines that share a hash are the same file, and the command prints every member of each group, so you see which copy you are keeping as well as the ones you are not. Add ~/Movies, ~/Pictures or ~/Music to the list to widen it, and raise +1M to +50M if you only want the copies worth the trouble. The first time, macOS asks whether Terminal may access your Downloads, Desktop and Documents folders; allow it, and the permission is listed afterwards under System Settings → Privacy & Security → Files & Folders. On a folder full of large videos the command takes a few minutes, because it reads every file once.

Nothing in that line deletes anything. For each group, decide which copy to keep, then move the others to the Trash in Finder (Shift-Command-G, paste the folder's path) so they can come back if you were wrong.

Check that two files are identical

To settle one pair rather than search a folder, fingerprint both files:

shasum -a 256 ~/Downloads/report.pdf ~/Documents/report.pdf

shasum prints a fingerprint of each file's contents. The same fingerprint means the same contents, whatever the names say; a single changed byte gives a different one. It only reads the files.

Which copy to keep

• The oldest. It is almost always the original, and the one anything else was made from. Finder's Date Created column tells you.
• The one in the sensible place. A document belongs in Documents, not in Downloads. If the copies are identical, keep the one you would look for.
• The one other things point to. An alias on the Desktop, a file in a Music, Final Cut or Logic library that references the original rather than copying it, a link in a note. Delete the copy nothing points to.
• Never the last one. A duplicate is only a duplicate while another copy exists. If you cannot find the other copy, it is not a duplicate.

Two hard links to one file look like two copies and are not: deleting one frees nothing. They are rare in a home folder, and the Terminal command above lists them as a pair, so if a "duplicate" frees no space when it goes, that is why.

The copies a hash cannot find

A screenshot kept as a PNG and again as a JPEG, a photo exported at a smaller size, a page rendered by two different tools: the same picture, saved twice, with not one byte in common. No file comparison can see those, because changing the size or the format changes every byte. Finding them means comparing the pictures themselves, pixel by pixel, which is a different job from finding identical files. Inside the library, the Photos Duplicates album is the place to look. For bursts, re-takes and shots that merely look alike, which neither method should offer to delete, Folio Refresh, a separate Caddora app, scans the Photos library and your folders and moves nothing until you approve it.

Why it grows back

Because the sideways routes stay open: you keep downloading, exporting and saving attachments. A duplicate finder is not a thing to run once. The habit that helps most is a single place for downloads and a quick sort of it by date every few weeks, before the copies spread into Documents and the Desktop. Everything else is a check every few months when the disk is short.

What a safe duplicate file finder does

• Matches on contents, not names. A renamed copy is still a copy, and two different files can share a name.
• Never offers the last copy. One of each set is kept and is not something you can tick.
• Says which copy it keeps, and why, so you can disagree before anything goes.
• Leaves libraries closed. A photo library or an app package is one thing to the app that owns it; its insides are not loose files.
• Moves to the Trash rather than deleting outright, so File → Put Back still works until you empty it.

Where the app comes in

RefreshYourMac does the hash comparison above as a category, Duplicate Files: files whose contents match byte for byte, over 1 MB, in Downloads, Desktop, Documents, Movies, Pictures and Music, matched on size, then on the first 64 KB, then on a hash of the whole file, never on the name, so a renamed copy is still found. One copy of every file is kept and is never listed, so cleaning everything shown cannot remove the last copy of anything. The copy kept is the one filed outside Downloads, otherwise the oldest, and each row says which one it keeps. Photo libraries and other packages are left closed, and two hard links to one file are not counted as a duplicate.

A second category, Duplicate Pictures, finds the same picture saved more than once at a different size or in a different format, which Duplicate Files cannot see. Two pictures count as the same only when their pixels agree, so a burst of photos is never offered, and the best copy, the one with the most pixels, is kept and never listed. Both categories are rated Review before deleting: Clean takes the whole list after a confirmation, and what it removes goes to the Trash. In the Mac App Store edition, Desktop and Documents are included once you allow them in Settings → Folders. The download edition also has Identical Copies in App Data, which turns spare copies inside app folders into APFS clones, so every file stays where its app expects it. For the rest of the places worth checking, see how to free up space on a Mac.