Add preliminary spec11 monthly pipeline

This adds the scaffolding for a basic Spec11 pipeline- it gathers all domains from all time for a given project and counts how many there are. I've factored out a few common utilities for beam pipelines to avoid excessive duplication.

Future CLs will:
- Actually process domains via the SafeBrowsing API
- Generate a real spec11 report
- Template queries based on the input YearMonth
- Abstract more commonalities across beam pipelines to reduce boilerplate when adding new pipelines.

TESTED: FOSS test passed, and ran successfully on alpha

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205997741
This commit is contained in:
larryruili
2018-08-10 13:44:25 -04:00
committed by jianglai
parent ded40851d3
commit d199b383e5
14 changed files with 252 additions and 38 deletions
+11
View File
@@ -6,4 +6,15 @@ licenses(["notice"]) # Apache 2.0
java_library(
name = "beam",
srcs = glob(["*.java"]),
deps = [
"@com_google_flogger",
"@com_google_flogger_system_backend",
"@com_google_guava",
"@org_apache_avro",
"@org_apache_beam_runners_direct_java",
"@org_apache_beam_runners_google_cloud_dataflow_java",
"@org_apache_beam_sdks_java_core",
"@org_apache_beam_sdks_java_io_google_cloud_platform",
],
)