POLITICO NY Article - City Record to be (more) available online

Tags: #<Tag:0x00007fca7203e0c8> #<Tag:0x00007fca7203e000> #<Tag:0x00007fca7203df38>

This is an article from August 2015 but still worth sharing since it’s behind a paywall so some may have missed it.


City Record to be (more) available online

The City Record. (The City Record)

By MIRANDA NEUBAUER 5:09 a.m. | Aug. 10, 2015

The city will officially announce today that the City Record, which has published procurement, hearing and other public notices weekdays since 1873, now has a fully searchable, expanded and more accessible online presence.

The new platform is the result of legislation signed into law last year that required the publication, produced by the Department of Citywide Administrative Services, to be available online in a searchable and machine-readable format, and which was introduced by Councilman Ben Kallos.

“The City Record newspaper has carried the important notices of city business for more than a century. Now New Yorkers will have full access to this information at their fingertips - making it easy to learn about contracting opportunities, public hearings and more,” Mayor Bill de Blasio said in a statement. “We are not only making more information accessible to the public, we’re also making it more user-friendly,” Stacey Cumberbatch, DCAS Commissioner, said in the statement. Until now, only a portion of the publication’s content was searchable online.

The expanded database will include public hearings and meetings on subjects such as zoning changes, liquor licenses and sidewalk cafe applications, regulatory changes, hiring notices and other city personnel changes. The platform also allows users to download attached documents such as agendas and requests for proposals when necessary. The print edition will continue to be available. Users will be able to sign-up to receive e-mail notifications for areas of interest, and the information will also be available through the city’s open data portal.

The nearly year-long work involved the equivalent of one full-time DCAS employee and a short-term consultant hired as a developer through the Department of Informational Technology and Telecommunications at a total cost of $350,000. DCAS staff included an I.T. developer, and I.T. user experience designer, a quality assurance staff member as well as City Record editors.

Dahan (Jamal) Abdo, director of .NET Application Development, said in an interview that the work drew on a software development concept known as Scrum to allow for a more collaborative, flexible, feedback-driven approach that involved demonstrating early versions and meeting regularly to review issues. “Our goal entirely was to make sure that the data is presented in a very clear and easy to digest way,” he said. “We’ll be able to search not just the text of the notices but also the documents as well,” he said. “The challenges were mostly how to convey a large amount of data in a very simple and straightforward manner.”

He said the team had reached out to agencies that use the City Record the most to get feedback on mockups of the site, and also had support from Jeff Merritt, the city’s director of innovation as well as civic developers in the City Record Online Working Group, led by civic technology group BetaNYC

Kallos praised an “unprecedented level of cooperation” on the project between the Council, the administration, as well as the developer community. He drew on his own background as a software developer to assist with work on developing schemas, normalizing the publication’s data input, and engaging with DCAS to go over their front-end and back-end designs and recommend additional features.

Beyond the first step of simply making the data available online in a machine-readable format, he worked with DCAS on adding geographic information. “A lot of the information has previously been just in one text block,” he noted. “[With the] new launch, the location of the hearings will actually be parsed out in address format…I’m hoping that in our next release, the geographic information relating to the items being heard will be the next frontier.”

Improved access to the City Record will help provide notice “so that people in communities can give their feedback [on neighborhood proposals] at the right time before it’s too late,” Kallos said. “The City Record is the most important newspaper that no one has ever heard of and hopefully it will become the dataset that everyone uses even if they don’t even know it because they’re using it through an app that helps make the information more useful,” he added, saying that he would be listening for feedback on future improvements.

The civic technologists’ involvement began when the Mayor’s Office reached out to BetaNYC about converting older PDF versions of the City Record going back to around 1998 into machine-readable data, Noel Hidalgo executive director of BetaNYC, recalled.

“As we got access to the PDFs, we realized that we actually have a much larger project ahead of us, it’s not just being able to scrape PDFs and turn them into some magical data,” he said. As the group worked on developing tools that would be able to extract information both from data already digitized and the older PDFs, a more complex process, Hidalgo said one challenge was the lack of an overall glossary of all the different kinds of information in the City Record. The process of extracting data in a structured from the publication by is like “teaching a robot how to walk,” he said.

In over 20 meetings with DCAS, developers highlighted the functionality of an existing civic notification tool called Citygram to demonstrate the importance of adding date and address fields to the meeting notices.

“This was the first time that the city invited technologists to come in and work intimately with them,” he said. “Even though we may have not had all the resources on either side to fully execute the intended vision [in the timeframe of a year], we were able to accomplish a significant amount of work,” he said, pointing to creation of several tools, such as a parser that the developers can start using to create online hearing notices. “That’s a huge, huge win,” Hidalgo said. “Now that the pressure of the anniversary of this law has passed, we are looking very much forward to the city making a greater commitment to … making the City Record more accessible.”