We are in the process of consolidating all our data from multiple servers down to one. We have about 250 TB of data, and we’ve moved about 200 TB of stuff already, and I’m seeing we have tons of duplicate files and folders.
Our data is a mixture of RAW video (.R3D), pictures and general business data. Some of our data has alternate data streams. I need to make sure all our data transferred correctly and all the metadata and ADS is intact.
What’s the best way to easily verify all the files are the same and then deduplicate? I’m looking for those who’ve been there, done that for some advice on tools and methods to make sure everything is good.