Assume you will publish somewhere publicly when you get it?
I think the city will publish, on something related to innovation.nyc, perhaps.
I think its not too late to make the dotNYC TLD the digital equivalent of Manhattan’s Grid Plan.
And once again, NYC could lead the way. Before the Grid, streets were developed haphazardly as evidenced by the warren of streets we have below Houston. But starting from Houston up, the Grid enabled all kinds of then unforeseen benefits - being able to accommodate future innovations like Central Park, the inherent ease of navigation, increased walkability, and of course, ManhattanHenge.
What if the dotNYC TLD was not only a land grab, with domain squatters registering domains speculatively? What if a reserved URI was automatically assigned to every NYC entity?
Regarding the expected release of data on .nyc domain name registrants, I received the following message last night:
Instead we’re likely to get:
an announcement going out at the end of this week with aggregate data from Whois (borough distribution, registration numbers by zip code, percentage of individuals v. organizations, etc).
Better than nothing, but a bit disappointing.
And there’s a lesson: City contracts must be written with an eye toward public access to generated data. In this instance the city truncated the public review process in 2012 by claiming an emergency situation existed.
Sometimes promises are kept. Other times not. I’m afraid we’re faced with the later in the case of the data on .nyc registrations.
Last Friday the mayor’s office issued a press release entitled “Mayor de Blasio Announces .nyc is Fastest Growing City Domain Six Months After Launch” available at http://www1.nyc.gov/office-of-the-mayor/news/314-15/mayor-de-blasio-nyc-fastest-growing-city-domain-six-months-after-launch.
It’s not been confirmed, but I suspect this was tied in with our request for the raw data. If you start looking at the few stats released, one begins to see why. Here are the stats:
75,000 names have been registered
53 percent of .nyc domains have been registered to individuals;
47 percent have been registered to small businesses and organizations
The average registrant owns between 2-3 domains, and 97 percent of registrants own 10 or fewer domains
With these we can surmise:
- 39,750 names have been registered to “small businesses and organizations” - but I suspect this included large businesses as well.
- 35,250 were registered to individuals
- Let’s presume the average registrant has registered 2.5 names (“between 2-3 domains”) and if 75,000 names have been registered, there are 30,000 registrants.
- 97% of the registrants have 10 or fewer names registered, i.e., 29,100 registrants
- 3% of registrants have 11 or more names registered = 900 registrants.
- Less than 1% (.003%) of New Yorkers have registered a domain name.
What else can we gather from these few stats?
As one ponders these numbers, a desire for more data arises. How many per borough? How many per census district, per street, building, etc. How many of the 3% are parked?
It appears a FOIL request will be required. I’ve been told by a city official that “We have all the aggregate information on the number of registrations by zip code or can break these down further if you’d like.” and would like to fashion the request to get as much info as possible. I’d appreciate hearing from someone who has experience making data requests and can assist me in formulating the request.
How can I get a copy of the story you published on this?
With the mayor having issued a press release - http://www1.nyc.gov/office-of-the-mayor/news/314-15/mayor-de-blasio-nyc-fastest-growing-city-domain-six-months-after-launch - about registrants for .nyc domain names and not the data we were seeking, I’ve been forced to file a FOIL request.
I filed the request through the official form available on the DoITT site. Is this an effective way to approach this task? Or is a certified letter a better way to go? (I say this as DoITT’s digital FOIL system did not provide a receipt of any sort, so I fear waiting several days, receiving a “What FOIL request?” to an update inquiry and having to start all over.)
Guidance and experience appreciated.
Update #1 - On June 5 I received acknowledgement that DoITT had received the FOIL request. As the law states that a response should have been received within 5 business days, we’re facing a lag of a few days. Also, the notification did not provide an estimated date when we might expect to receive the requested data, as required by FOIL. We’re following up.
Update #2 - Clarification resulted in the following: “Thank you for your email. If possible, DoITT will grant or deny your request by July 6, 2015, which marks the 20th business day from the date of acknowledgement. I am looking to locate the records that you have requested. At this time, I am unsure as to whether DoITT maintains WHOIS data of registrants of .nyc domain names.” While this seem outside the FOIL guidelines, activities here make it acceptable. Stay tuned.
Thanks for the update. We’re looking forward to getting this data!
July 6th arrived and DoITT responded to the FOIL request.
What we received was a cvs file listing:
- the domain names registered through March 15th.
- registration dates
- and if the registrant claimed to be a individual or an organization.
With our goal being to determine if there was geographic balance or equity in the name distribution, the data provided is inadequate.
- Small point: we did not request the name of the registrants.
- BIG POINT: one can gain access to the name, home or business address of all applicants by using the WHOIS system: see http://whois.nic.nyc/whoismtld/whois/.
Be that as it may, I’m going to again try for the zip code (or maybe census tract) data. If anyone can assist with this next step I’d appreciate it. Come August I’ll have some law students who’ll be assigned to this, but I’d like to get at the data before the next semester begins.
Finally, can anyone think of something useful to do with the data provided - domain name, date registered, and individual or organization? (I’d be happy to post the data - just tell me where.)
@TomLowenhaupt, you should post the data at data.beta.nyc.
Perhaps a member of the community can then write a small script to fetch the whois details
Indeed, that shouldn’t be too difficult. WHOIS is quite easy to parse, and the .NYC tld does not permit private/masked registrations.
However, Neustar & NYC has terms & conditions which limit automated WHOIS queries.
By submitting a WHOIS query, you agree that you will not use this data: … (ii) to enable high volume, automated, electronic processes that apply to the registry (or its systems ); …
And then there’s this gem:
…it is explicitly forbidden to extract, copy and/or use or re-utilize in any form and by any means (electronically or not) the whole or a quantitatively or qualitatively substantial part of the contents of the WHOIS database without prior and explicit permission by Neustar or the City of New York, nor in any attempt hereof, or to apply automated, electronic processes to Neustar (or its systems).
My favorite is this one, which I saw only after running a WHOIS query:
By executing a query, in any manner whatsoever, you agree to abide by these Terms and Conditions.
Point of information: This is a 5 year contract and we’re more than half-way through it. Let’s make sure prohibitions like these are not in the new agreement.
P.S. There are “back-end” providers out there who deliver the registry services provided by Neustar for far far less than the current 60% of revenue. Interested parties should let the city know of their interest in participating in an open bid contracting process. (Keep in mind that Neustar, through it former parent Lockheed Martin, honed its government relations skills winning military contracts in D.C. It knows that process. Won’t be easy.)
Here’s a quick Tableau viz that I pulled together to allow you to explore the dotNYC registration data.
I pulled it together so its within limits of the Terms and Conditions since we’re not scraping the data…
I’m surprised you got the records at all. Some of this is out of NYC & Neustar’s control. Instead, it is required by ICANN, which certifies Neustar as a registrar. ICANN specifies what information is required for domain registrations, and also specifies restrictions on how it may be used. The Neustar terms of service which I cited above appear to be consistent with ICANN’s restrictions. (I’m not a lawyer, though.)
On the upside, ICANN rules also say that registrars like Neustar can make the data available in bulk, are permitted to charge for it, and must ensure that the data isn’t used for marketing/spam.
Tweaked the viz a bit:
- WHOIS info is displayed in the dashboard rather than opening a new tab which didn’t happen consistently across browsers
- made the registrations/day line chart logarithmic so you can see the registration long tail better. Added trendline.
If I’ve not said this before, the dotNYC Explorer is wonderful. It demonstrates what can be done with just a little open data. Thanks.
Over the past weeks I’ve held my breath hoping that our appeal of DoITT’s denial of the key data we sought (zip code of registrants) would be successful. However, I recently received notice that our appeal had been denied. DoITT concluded it denial saying “You have the right to seek review of this determination pursuant to article 78 of the New York Civil Practice Law and Rules and POL section 89(4)(b).” (See the full letter and correspondence file on this here: https://docs.google.com/document/d/1sSs3CoMCrgqMRh4oGiUDyNzgUamblIopFmJGaYIkju4/edit.)
What next? Here’s a little background to help paint that picture.
When we made the request for zip code info in March, we did so for two reasons:
- to help establish the equity of the name allocation process with the requested data,
- and to test the data request process.
We expected the city to collaborate with us in developing this important allocation map.
After establishing the process and relationship we anticipated a second data request, one that would facilitate a community accountability program to enforce the city Nexus Policy. [Nexus policy says who qualifies for owning a .nyc domain name.]
Most important was to be a third request providing us with the ability to contact individual owners of domain names. This capacity is vital to our role as an At-Large structure of ICANN. As such we are bound to support:
“individual Internet users’ informed participation in ICANN by distributing to individual constituents/members information on relevant ICANN activities and issues, offering Internet-based mechanisms that enable discussions of one or more of these activities and issues among individual constituents/members, and involving individual constituents/members in relevant ICANN policy development, discussions and decisions.”
If we’re to effectively serve as an ICANN At-Large Structure, and more important, a force for civic good, we need access to this data.
We’ll detail our next actions to release this data soon.
Update Report To Beta-NY on .nyc FOIL situation
Our WhoIs data request and Why We’re Seeking Access.
With DoITT’s denial of our request for selective data of registrants of .nyc domain names, we’re left to decide on the next step. DoITT said we needed to go to NYS Supreme Court and begin an Article 78 proceeding. If necessary we’ll do this, but are considering options. We’ve received an unofficial opinion from the NYS Department of State (The NYS entity overseeing FOIL) that DoITT’s Response was unfathomable. As well, they offered to assist with an our next step.
Today I want to explain why we’re looking to gain access to this information. I’ll first explain the available information (What’s WhoIS), second, the info we’ve asked for, and then raise some reasons, why it would be helpful to have this data - possibly generating additional thoughts on how we might use the data.
First, What’s WhoIS?
When you purchase/register a domain name an entry is made in a WhoIS file maintained by the registry. Here’s the full WhoIS file for QueensMuseum.nyc domain name. (Why do I own QueensMuseum.nyc? The Museum was in the process of changing its name during the name allocation process thus it was not reserved. I took it upon myself to purchase the name, to hold in in trust, and to make it available to the Queens Museum when they are ready. I hope they call before October 16 and the renewal $$ date.)
Here’s a typical WhoIs file… Note there are four categories of registrant data collected: the Registrant (the one who registers and “owns” the name), Administrative, Billing, Technical, and nyc contacts. In this instance all are the same. But with other domain names each is different.
Sponsoring Registrar:GoDaddy.com, Inc.
Sponsoring Registrar IANA ID:146
Registrar URL(registration services):whois.godaddy.com
Registrant Contact ID:
Registrant Contact Name:Thomas Lowenhaupt
Registrant Contact Organization:
Registrant Contact Address1:35-35 75 Street
Registrant Contact Address2:Jackson Hts.
Registrant Contact Address3:
Registrant Contact City:NYC
Registrant Contact State/Province:New York
Registrant Contact Postal Code:11372
Registrant Contact Country:United States
Registrant Contact Country Code:US
Registrant Contact Phone Number:+1.7186394222
Registrant Contact Facsimile Number:
Registrant Contact Email:email@example.com
Administrative Contact ID:
Administrative Contact Name:Thomas Lowenhaupt
Administrative Contact Organization:
Administrative Contact Address1:35-35 75 Street
Administrative Contact Address2:Jackson Hts.
Administrative Contact Address3:
Administrative Contact City:NYC
Administrative Contact State/Province:New York
Administrative Contact Postal Code:11372
Administrative Contact Country:United States
Administrative Contact Country Code:US
Administrative Contact Phone Number:+1.7186394222
Administrative Contact Facsimile Number:
Administrative Contact Email:firstname.lastname@example.org
Billing Contact ID:
Billing Contact Name:Thomas Lowenhaupt
Billing Contact Organization:
Billing Contact Address1:35-35 75 Street
Billing Contact Address2:Jackson Hts.
Billing Contact Address3:
Billing Contact City:NYC
Billing Contact State/Province:New York
Billing Contact Postal Code:11372
Billing Contact Country:United States
Billing Contact Country Code:US
Billing Contact Phone Number:+1.7186394222
Billing Contact Facsimile Number:
Billing Contact Email:email@example.com
Technical Contact ID:
Technical Contact Name:Thomas Lowenhaupt
Technical Contact Organization:
Technical Contact Address1:35-35 75 Street
Technical Contact Address2:Jackson Hts.
Technical Contact Address3:
Technical Contact City:NYC
Technical Contact State/Province:New York
Technical Contact Postal Code:11372
Technical Contact Country:United States
Technical Contact Country Code:US
Technical Contact Phone Number:+1.7186394222
Technical Contact Facsimile Number:
Technical Contact Email:firstname.lastname@example.org
nyc Contact ID:
nyc Contact Name:Thomas Lowenhaupt
nyc Contact Address1:3535 75TH ST
nyc Contact Address2:
nyc Contact Address3:
nyc Contact City:Jackson Heights
nyc Contact State/Province:NY
nyc Contact Postal Code:11372
nyc Contact Country:United States
nyc Contact Country Code:US
nyc Contact Nexus Category:INDIV
Created by Registrar:GoDaddy.com, Inc.
Last Updated by Registrar:GoDaddy.com, Inc.
Domain Registration Date:2014-10-17T16:19:53Z
Domain Expiration Date:2015-10-16T23:59:59Z
Domain Last Updated Date:2014-10-17T16:19:54Z
Second, What did we request the data?
Our original request sought info about registrants and their zip codes. We wondered if there was equity in the allocation process. For example, were all the names being bought by those in upscale neighborhoods? If so, perhaps some awareness campaigns and educational efforts in less affluent / digital neighborhoods could address inequities.
I submitted about 10 different requests for this info. The first several were as emails addressed to the city officials seemingly in charge. Then, seeing nothing positive coming forth, the last few were FOIL requests. The July 23 denial of my FOIL request was the last contact I’ve had with DoITT.
Finally, Why should anyone care?
There are several reasons.
Equity - As I mentioned above, the the initial thrust was to look at inequities in the allocation process Was the domain name allocation process just another instance of the rich getting richer? Opportunity for all has been one of the themes of our effort. Our plan was to enter the zip codes of registrants into GIS software and see how the map looked. Were domain names being registered evenly throughout the city. If no, what could be done to improve the situation. (Following this “equity” theme, we recently have been looking for ways to avoid the high-bid auction of Premium Domain Names - news.nyc, sports.nyc, hotels.nyc, etc.)
Nexus Violations - To own a .nyc name you’ve got to have nexus - a connection to the city (think of sticky fruit nectar). According to the agreement with contractor, its up to the residents to self regulate this. If you find a violation, report it. It would be lot easier if we had access to the data in searchable digital format.
Active vs. Inactive - The NTLDstats site reports 20% of .nyc domains are parked. (Last week it reported that 70% were parked - I’ve no explanation for the discrepancy.) Are these owned by a small group of individuals and should the city have a no-parking policy? Data access would help us find this out.
Collaboration - The city and our TLD are all about collaboration. If there were easier ways for .nyc owners to communicate with one another, we’d see more activity. For example, if owners could communicate with one another they might organize and petition for DNSSEC - a security upgrade to the .nyc TLD. Currently the city has the owners atomized and unable to communicate. We for example, would like to fulfill our role as an ICANN At-Large Structure and facilitate communication between individual Internet users and ICANN. Can’t do that under the current veil of secrecy.
There are probably many more uses of this information. What are your thoughts?