For domains (and soon for hosts as well), we output data about the owning registrar. These subrecords wind up being really big if we include all data, because they also list all the registrar contacts. To avoid bloating the RDAP responses, change to output domain response registrar information in summary format, meaning we skip the registrar contacts and events. The requester can still get this information by using the link provided to request the registrar directly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=209189993
Gustavo Lozano confirmed that street addresses should be formatted as a string
if there is only one line, or an nested array of strings if there is more than
one. Luckily we already do it this way, so we just need to update the comment.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196845222
Even when the request is not permissioned to see contact information, we should
show information about the owning registrar.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=187049833
The RDAP Pilot Program operational profile document indicates that domain
responses should list, in addition to their normal contacts, a special entity
for the registrar.
1.5.12. The domain object in the RDAP response MUST contain an entity with the registrar role (called registrar entity in this section). The handle of the entity MUST be equal to the IANA Registrar ID. A valid fn member MUST be present in the registrar entity. Other members MAY be present in the entity (as specified in RFC6350, the vCard Format Specification and its corresponding JSON mapping RFC7095). Contracted parties MUST include an entity with the abuse role (called Abuse Entity in this section) within the registrar entity. The Abuse Entity MUST include tel and email members, and MAY include other members.
1.5.13. The entity with the registrar role in the RDAP response MUST contain a publicIDs member [RFC7483] to identify the IANA Registrar ID from the IANA’s Registrar ID registry (https://www.iana.org/assignments/registrar-ids/registrar-ids.xhtml). The type value of the publicID object MUST be equal to IANA Registrar ID.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=186797360
Changes the code to be in compliance with the RDAP Pilot Profile document,
which specifies:
1.4.11. If permitted or required by an ICANN agreement provision, waiver, or Consensus Policy, an RDAP response may contain redacted registrant, administrative, technical and/or other contact information. If any information is redacted, the response MUST include a remarks member with title "Data Policy", type "object truncated due to authorization", a description containing the string "Some of the data in this object has been removed" and a links member with the elements rel:alternate and href indicating where the data policy can be found. An entity with redacted information MUST include the "removed" value in the status element.
We were using the "removed" status to indicate deleted contacts and inactive
registrars. Instead, we will now use "inactive", so that we can use "removed"
to indicated redaction.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=185039201
This CL also fixes a bug. Registrars were returned in an arbitrary order. This caused cursor-based pagination to fail. Now we always sort by registrar name (even for handle searches), and use the registrar name in the cursor, to ensure proper behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182098187
By default, RDAP entity searches return both contacts and registrars. This CL
adds a new query parameter to request only one or the other. Among other
benefits, this will allow a future CL to permit wildcard searches that return
all registrars.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181605990
In addition, while adding the tests, I became discontented with the thoroughness of the cursor navigation tests, which checked only the number of items returned, not their proper ordering. So I updated them to be more careful, and backported the changes to the nameserver and entity search tests as well.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179442118
A couple methods were moved to new locations so they are accessible to all types of search queries, not just nameservers like they originally were.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179089014
It turns out that the RDAP spec does not envision multiple help pages. We can
still support them (for the TOS, for instance), but we shouldn't expect users
to go searching for help other than the main page. Therefore, consolidate the
useful information on the main page, and get rid of some of the others.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178792548
Also, a couple tests are reworked to make their output more understandable in
case of failure.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176670087
Most of the time, we don't expect incoming requests to have an authorization
header. So this statement gets printed a lot, and doesn't provide much useful
information. We already have a statement listing what type of
authentication/authorization is required by the endpoint, and other statements
indicating either that authorization was successful with a particular method or
was not successful at all.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175969652
Also, login logic pulled out to helper methods in the test class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175870131
This CL also includes a minor refactor of the query size limit calculation; it is computed and stored in a local variable, to be used in two places, rather than computing it separately in each place.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175824713
This CL uses the previously-defined RDAP metrics class to record basic metrics
for all RDAP endpoints, and handles testing of non-search endpoints. Searches
are more complicated, and will be handled in future CLs.
The default wildcard type is now INVALID rather than NO_WILDCARD.
A change to getMatchingResources() (adding an additional parameter) is also included in this CL, as it was needed to set the incompleteness warning type correctly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175719265
The code to use this class will come in future CLs, to avoid a huge CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175162557
I split this out to avoid having a giant CL that changes everything. The actual
metrics will follow later.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174356874
It has been decided that we will link the RDAP help pages, not to local HTML copies, but to the Github documentation.
A typo and word order error in the Github version of the user's guide are also
corrected.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174350665
The former method -- a config string -- was cumbersome, as each Nomulus system would have to configure the link base to its own URL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173567021
I am not happy that another index is required, but the Pantheon console shows that domain indexes are much smaller than the other indexes (because there are fewer domains), so it's not adding an appreciable amount of storage space.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173561771
The RDAP help page has anchors, so that we can then change the internal RDAP help topics to point to specific parts of the documentation. Unfortunately, GitHub does not appear to support the style of anchor we were using. See the current situation:
https://github.com/google/nomulus/blob/master/docs/rdap.md
Experimentation in GitHub confirms that switching to embedded HTML <a> tags solves the problem, and Critique preview shows it working correctly as well, with automatically generated table of contents at the left. If there's another way to fix this problem, I'm happy to try it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173405974
We were relying on Dagger to validate the IP address, but that resulted in 500 errors when the IP address was not valid, which is undesirable. Instead, accept the parameters as strings, then convert them to IP addresses and throw a proper error when conversion fails.
Also fixes an improperly specified test.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173172516
Usually, the correct order happens automatically, because we are searching on either the key or a specific field like fullyQualifiedDomainName, and the results come back in that order. But when searching by nameserver, where we split the query into multiple "IN" chunks, we have to assemble the result set and order after the fact.
The tests didn't pick up the problem, because the domains and hosts were created in alphabetical order, so it happened to work anyway. The tests have now been changed to create things in reverse order, to test the reordering. Also, the previous arbitrary limit of 1000 nameservers in the intermediate query has been reduced to 300, because we now loop through all nameservers no matter what, rather than stopping when we collect enough domains, so there's more of a penalty for having way too many nameservers.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173163121
We no longer find contacts by name if the request is not authorized to see the name.
Several changes cascade from this. Previously, the code assumed that deleted contacts might still have full names, and therefore be searchable. This is not possible in all cases, because Datastore doesn't have the right index to find deleted contacts by name with a matching registrar. However, luckily, this situation can never occur, because contacts always have their name fields nulled out when they are deleted.
So instead, we simply ignore deleted records when searching by name, knowing that none can ever match.
The tests were then changed so that deleted records look the way the really will, meaning devoid of personal information.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172776926
The code was not checking the value returned by getTrid() to make sure it was
not null.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172384499
This CL adds the functionality for domain searches. Entities and nameservers have already been handled by previous CLs.
Deleted items can only be seen by admins, and by registrars viewing their own deleted items.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172097922
RDAP searches for contacts with a specific desired registrar need an additional
index term. The tests were not extensive enough to catch this particular case.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172013843
Soon this will be rewritten with much more complete information. But this fixes
an obviously wrong bit in the meantime.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172004672
The nameserver may be external, in which case its TLD will not appear in our
list of valid TLDs, and the search will be rejected erroneously.
Tests for letter case canonicalizations also added at reviewer's suggestion.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171985702
Also sets the charset to UTF8 explicitly, to avoid confusion, and removes apparently unneeded Truth8 include.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171963981
Also fixed a bug discovered as a result of the more complete testing!
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171156304
This CL adds the functionality for contact and registrar searches. A future CL will handle domains and entities.
Support is also added for filtering results by registrar.
Deleted items can only be seen by admins, and by registrars viewing their own deleted items.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170734664
This CL adds the functionality for nameserver searches. Future CLs will handle domains and entities.
Deleted items can only be seen by admins, and by registrars viewing their own deleted items.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170106014
The gcloud datastore cleanup-indexes command takes its input in index.yaml form instead of datastore-indexes.yaml form. This simple translation script allows us to generate one from the other.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169260811
There was a TODO to consider making RDAP queries look up domains by foreign key rather than directly to the domain itself. But since the TLD is a possible search filter, it makes more sense to stick with direct lookup, so we can take advantage of the index on TLD.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169240318
When searching for domains by nameserver name or IP address, we fetch the matching nameserver keys, then search for domains by those keys. We limit the number of nameserver keys returned, to avoid arbitrarily large domain queries. This CL adds a warning to the RDAP response if we retrieved the maximum number of nameservers. This may indicate that we have not found all the domains.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168885124
We had been allowing lookups by nameserver name using a wildcard and suffix if the suffix was a domain name. That's ok if the domain name is one we manage, but doesn't work efficiently otherwise. A lookup of ns*.zombo.com would require us to search for all nameservers beginning with ns (which could be almost all of them), then loop through until we found those ending with .zombo.com. So we are going to prohibit suffixes after the TLD unless the suffix is an in-bailiwick domain.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168835732
RDAP filters some contact data when the request is not authenticated as coming from a user associated with the registrar owning the contact. This CL adds an exception for admin users, which for the App Engine Users API are defined as App Engine project viewers. This means that the registry team will always get all information when logged in. This will also be useful when building tools that use RDAP output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168293820
This CL adds extra tests to make sure that contact data is hidden for RDAP domain and contact requests when not logged in as the proper registrar.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168234733
This CL changes the RDAP responses. When the requester asks for information about a domain, and is not logged in as the owning registrar, no contact information is shown. When the requester asks for information about a contact, and is not logged in as the owner registrar, the existence of the contact is shown, but not any personal data (the existence is shown to make things easier to test).
The login uses the same functionality as the registrar console.
For the most part, this CL does not include the necessary tests to make sure that data is not returned when not logged in. The CL is so large that I didn't want to burden it further. Those tests will be added in a follow-on CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168022034
One was missing in the ICANN distribution: TreatyStatue-Agent-Arabic-Revoked.smd. I just deleted that one, because we don't need it.
These new files are only valid until June 2018, because the embedded certificate expires then. At that point, ICANN will presumably release a new set of files.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167164915
In standard DNS format, the first thing on an A, NS or DS definition line is a domain label relative to the zone, which in our case is a TLD. However, the generate_zone_files command prints out fully qualified host and domain names, resulting in a discrepancy when compared to the contents of the DNS subsystem. This CL removes the TLD suffix, which should remove one preprocessing step before file comparison.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166103705
Previously, GenerateZoneFilesAction mapreduced its way through all domains and hosts for the specified TLD(s), emitting information for each matching domain and host (subject to constraints like not being deleted and so on). This resulted in host information (aka glue records) for all hosts subordinate to domains in the specified TLD(s). This is incorrect. DNS glue records should only be present for hosts which act as nameservers for their superordinate domains.
The new version of the mapreduce iterates only over domains. When a matching domain is found, a check is made to see whether any subordinate hosts are also nameservers for the domain, in which case host information is generated.
The test was updated to reflect the new reality, and check for a couple additional nuances.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165766472
Attending to this old bug will improve our ability to perform zone comparisons between Datastore and the DNS provider. Right now, zone comparison finds some bogus differences, because the TTL we send to the DNS subsystem doesn't match the TTL we use when generating our local dump files.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164635557
The auth logging has been useful, but it now generates a sizeable percentage of all logging, because it spits out three to five lines for every request in the system. This CL reduces that to two to three. We may eventually want to reduce it further, but this is a good start.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164146182
It was not a problem after all to handle multiple scopes. Also added a temp variable to avoid making the same array conversion over and over.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164002903
When the registrar console code determines that a user has not logged in, it redirects to a login page. But when authenticating as an internal request (which should never happen), the redirection code encountered an exception, resulting in a 500 error.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163867018
A NullPointerException reported via StackDriver appears to stem from trying to load the claims list right at the moment it was being updated. Since the update only happens once every 12 hours, retrying the load once should fix the problem, if this is really the cause.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163732624
Up to now, our search wildcard rules have been that there must be an initial string of at least two characters. If a wildcard is present after that, it can optionally be followed by a suffix specifying the TLD (for domains) or domain (for nameservers). So domain queries can look like:
example.tld
ex*
ex*.tld
and nameserver queries can look like:
ns1.example.tld
ns*.example.tld
ns*
But you can't do a domain query for *.tld, nor a nameserver query for *.example.tld. It would be nice to support such queries, and the presence of a valid TLD or domain makes them relatively efficient. This CL relaxes the restrictions to allow wildcards with no initial string if the suffix is present. For nameservers, the suffix must be a valid domain in the system, to avoid having to loop through all nameservers.
A side effect of the changes is to fix a shortcoming in the logic which caused wildcard nameserver searches to fail if the specified domain suffix referred to an external domain.
Entity searches are not affected, since they do not support suffixes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159856563
The affected actions have been changed to check that the user is logged in by [] so this attribute is no longer needed.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159572365
We are going to remove the requireLogin attribute from the action attribute, because it is specific to the UserService API. This is used by four actions:
ConsoleUIAction
RegistrarSettingsAction
RegistrarPaymentSetupAction
RegistrarPaymentAction
Instead, these four actions will now check the login status directly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159562335
PDT testing revealed a couple ways in which our WHOIS output was non-compliant. First, the Consistent Labeling & Display policy dictates that the contact IDs must be ROIDs. See rule 11 in https://www.icann.org/resources/pages/rdds-labeling-policy-2017-02-01-en. Second, PDT tests expect that a WHOIS response will treat missing values either by omitting the line entirely, or by including the line with a blank value, but not both. So this is legal:
Phone Number: 123-4567
Phone Number Ext:
Fax Number: 123-4568
Fax Number Ext:
and this is legal:
Phone Number: 123-4567
Fax Number: 123-4568
but this is not:
Phone Number: 123-4567
Phone Number Ext:
Fax Number: 123-4568
In the last example, one extension line is present with a blank value, while the other extension line is omitted. We cannot do both. Therefore, we should update our code to omit lines with no value. Since we can't guarantee that we will always emit all lines that the parse might expect to see, it is safe to use the policy of omitting lines with no value.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158184150
The command was set up such that an update without any contact types specified would clear out the list, instead of leaving them unchanged, as it should.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157766429
[] opened up the settings on RdeStaging, in order to make it usable by the nomulus tool. But in retrospect, we think that all we needed to do was support the POST method, not loosen the auth settings, since nomulus invokes RdeStaging via a task queue. Removing the looser auth settings will bring this action into line with other backend actions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157262595
This brings the affected actions into line with the settings on other similar actions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157259842
Our build got broken by the deprecation of goog.structs.Map in [] This is a quick fix.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157236744
It was setting the TLD and watermark date parameters as comma-separated lists. What it really needs to do is set the parameters multiple times, once for each value.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156741051
We will not want to run this under normal circumstances, but in cases (such as PDT testing in sandbox) where it's desirable to generate an escrow deposit even if it isn't technically valid XML, this gives us that option. Manual-mode RDE generation is also changed so that, if no watermark date is specified, it defaults to the previous midnight, to better support running of RDE in sandbox to catch data problems.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156108295
This is needed to delete an invalid host resulting from a previous bug.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154757425
Currently, if the ID is invalid, parentKey is set to null, causing it to return all history entries. Note that there is still a problem that you cannot look up history entries for entities which have been soft deleted, because the foreign key lookup won't work. That is unfortunate, but at least this simple fix makes things better than they were.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154564262
Previously, GenerateEscrowDepositCommand generated the deposit itself. Channeling it through the existing deposit generation code make things more maintainable.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152847950
RdeStagingAction always processed all RDE and BRDA deposits currently outstanding, updating the cursors appropriately and kicking off the upload job. Sometimes we don't want all that. We just want to create a specific deposit by hand, without modifying the cursors or uploading. This CL adds parameters to support that.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152415959
The code to authenticate and authorize incoming requests (including via OAuth) has been in the system. This CL actually turns it on, since we are satisfied from logging information that it is not unjustly denying access.
Auth settings are also updated on a few commands missed earlier.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152381820
This CL defines metrics for both premium and reserved lists, but actually uses only the reserved list metrics. The premium list metrics will be used in a future CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149982639
A test has been added to RequestHandlerTest, making sure that, while we merely log errors for the time being, the correct dummy AuthResult is being created.
Most actions use the default settings, which have been changed to INTERNAL / APP / IGNORED. Actions with non-default settings are:
INTERNAL/NONE/PUBLIC (non-auth public endpoints)
CheckApiAction
WhoisHttpServer
Rdap*Action
INTERNAL,API/APP/ADMIN (things currently protected by web.xml)
EppTlsAction
EppToolAction
CreateGroupsAction
CreatePremiumListAction
DeleteEntityAction
List*sAction
UpdatePremiumListAction
VerifyOteAction
WhoisServer
INTERNAL,API,LEGACY/USER/PUBLIC (registrar console)
RegistrarPaymentAction
RegistrarPaymentSetupAction
RegistrarSettingsAction
EppConsoleAction
INTERNAL,API,LEGACY/NONE/PUBLIC (registrar console main page)
ConsoleUiAction
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149761652
mountfordandjianglai
