mirror of
https://github.com/google/nomulus
synced 2026-10-20 05:12:43 +00:00
gbrodmanandGitHub
052c083bcb
SafeBrowsing provides an API endpoint that returns, for each threat type
provided, a base64-encoded string that represents the concatenation of
4-byte prefixes of sha256 hashes of harmful domains. That means we can
do the following
- decode the base64 string into a byte[]
- interpret the byte[] as an int[], because each int is four bytes
- for each domain:
- hash the domain and take the hash's first four bytes as an int
- if that int is in the prefix hash array, query SafeBrowsing directly
with that domain name
- if the int is not in the prefix hash array, we know the domain is
not harmful (according to SafeBrowsing at least)
The prefix list provided by SafeBrowsing contains about 2.5 million
entries which corresponds to only about 10 MB of heap memory and only
about 0.1% of the space of all integers. Thus, we can assume there won't
be too many hash collisions.
This will reduce our number of API calls from ~5100 calls (each
containing 490 domains) to 6 API calls + (# actually-harmful domains /
490).