0

I am looking for the way how to compare two files (especially large files) in S3 within the same bucket using Java AWS SDK.

I do not need to verify whole bucket if there are duplicates. As I understood the Athena tool should be good for that to find all duplicates in a bucket. I need to compare only two files (objects) in S3 and nothing else.

Is there some better way than downloading data to local? I know that I can verify MD5, but if the MD5 hash is the same, I still need to download those files and compare them if those files are really identical. It is pretty ineffective to download two large files from S3.

Improve this question
2
  • Are you worried about an attacker creating 2 files with the same checksum, or are you only worried about random collisions? Commented Apr 17 at 10:34
  • I will suggest below two options, 1 - Compare ETag/MD5 checksums 2 - Compare size first, then use range-based comparison Commented Apr 21 at 4:34
Related questions
1773
How can I avoid Java code in JSP files, using JSP 2?
1603
How can I concatenate two arrays in Java?
967
How to verify that a specific method was not called using Mockito?
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.