Dropbox has a specific way they want you to calculate the content hash for a file you are uploading. You can find the instructions here but to summarize:
- Split the file into blocks of 4 MB (4,194,304 or 4 * 1024 * 1024 bytes). The last block (if any) may be smaller than 4 MB.
- Compute the hash of each block using SHA-256.
- Concatenate the hash of all blocks in the binary format to form a single binary string.
- Compute the hash of the concatenated string using SHA-256. Output the resulting hash in hexadecimal format.
We can convert each of these steps into parts of a pipeline!
First we need to start reading in a file, split into chunks of the specified size. File.stream! is a good way of achieving this.
chunk_size = 4 * 1024 * 1024
File.stream!("my_file.txt", [], chunk_size)
Next, we need to create a new hash from each chunk. Iterating over the stream chunks with Enum.map and passing those chunks into the Erlang module :crypto.hash will get us our chunk hashes.
chunk_size = 4 * 1024 * 1024
File.stream!("my_file.txt", [], chunk_size)
|> Enum.map(&:crypto.hash(:sha256, &1))
After that, we'll combine all individual hashes into one big chunky string of hashes. Enum.join will let us that list of strings and slap them together
chunk_size = 4 * 1024 * 1024
File.stream!("my_file.txt", [], chunk_size)
|> Enum.map(&:crypto.hash(:sha256, &1))
|> Enum.join()
Getting warmer, time to take that big boy and get one final hash using :crypto.hash again. We'll have to wrap it inside an anonymous function since we need to pass the string in as the second argument.
chunk_size = 4 * 1024 * 1024
File.stream!("my_file.txt", [], chunk_size)
|> Enum.map(&:crypto.hash(:sha256, &1))
|> Enum.join()
|> (&:crypto.hash(:sha256, &1)).()
Home stretch! Lastily, base 16 encode the final hash using Base.encode16. Don't forget to lowercase it by passing in case: :lower
.
chunk_size = 4 * 1024 * 1024
File.stream!("my_file.txt", [], chunk_size)
|> Enum.map(&:crypto.hash(:sha256, &1))
|> Enum.join()
|> (&:crypto.hash(:sha256, &1)).()
|> Base.encode16(case: :lower)
There ya go! You should now be able to create the content hash for a file that you intend to upload to Dropbox.
Top comments (0)