California Leaders on Open Data: Step-by-Step Progress, But No Master Plan


Last week the Data Coalition held our second annual California Data Demo Day in Sacramento. Over 100 open data supporters joined us for two panel discussions and a roomful of tech demos.

IMG_20160301_112322366

From left: State Assembly Members Ling Ling Chang and Brian Maienschein answer questions from the Data Coalition’s Hudson Hollister. (StateScoop)

Several California agencies are making steady, step-by-step progress toward open data. Agencies like the Department of Justice and the State Treasurer’s office have started to adopt consistent standards to make their data searchable, then publishing data for all to scrutinize and use. And many California municipalities are going further, publishing comprehensive open data sets on topics like spending and public safety.

But the state doesn’t have an overarching plan for how these separate efforts all fit together, which makes it hard for agencies and municipalities to plan and discourages open data companies from building new products.

Last week’s event was hosted by member company Socrata, and the Department of Better Technology chipped in for refreshments. Thanks, Socrata and DoBT!

Our other exhibitors, showing how their solutions handle the challenges of compiling, structuring, and visualizing public sector data, included Esri, Information Builders, Open Gov, Tableau, Workiva, and Xcential.

Here are four things we learned at the California Data Demo Day.

  1. Some California agencies are doing (or are about to do) amazing things with open data!

Moderating the a panel of state agency leaders, Robb Korinke of GrassrootsLab introduced the challenges of standardizing and publishing data in an outdated legal context. The state’s public records laws were written when IT couldn’t do what it can do now.

Korinke lamented that California, though it’s home to Silicon Valley, has been burned by large public sector IT failures, breeding hesitancy in moving forward on ambitious tech reinvestments.

Korinke then kicked it over to an energetic panel introducing four groundbreaking open data projects. We heard from Deputy Treasurer for Technology and Innovation Jan Ross (representing Treasurer John Chiang), Special Assistant Attorney General Justin Erlich (representing Attorney General Kamala Harris), the Legislative and External Affairs Director Phillip Ung of the Fair Political Practices Commission (FPPC), and state legislative counsel Diane Boyer-Vine.

Phillip Ung introduced the massive public oversight potential of the FPPC’s Form 700, California’s main disclosure form for public officials’ economic interests and financial holdings. Last year, the FPPC kicked off a project to transform the Form 700 from an outdated document into open data.

Ung’s, and the FPPC’s, next challenge will be working with the California Secretary of State to tackle CalAccess, the state’s campaign finance database, which he declared “impossible to navigate.”

If California’s political practices information is still catching up to other governments’, its legislative information is an open data model. In 2004, the state legislature, led by Legislative Counsel Diane Boyer-Vine, adopted an XML-based open data format for bills and amendments.

Boyer-Vine detailed how open data neatly maps the lineage and impact of amendments to bills, and in turn, to existing law. For the public, open data pulls back the curtain on the “very convoluted [process] to look back and see how a bill in the 16th iteration really impacts existing law.” For legal experts, the 2004 deployment of the open data system fostered a much faster process for updating state codes following the passage of new legislation.

Finally, Boyer-Vine noted how the XML structure allows for the more reliable preservation of legislative history by authenticating bills in their proper context (as opposed to the annals of discrete paper copies).

Justin Erlich spoke on Attorney General Kamala Harris’ OpenJustice initiative. OpenJustice aggregates law enforcement information from state, county, and municipal sources for public consumption through interactive maps and charts. He broke the project down into four conceptual challenges; (1) collecting better data statewide, (2) properly analyzing that data (partnering with outside researchers), (3) achieving transparency (posting in raw form), and (4) engaging people with that data.

Introducing Treasurer John Chiang’s DebtWatch, Jan Ross detailed how the platform publishes California municipalities’ $1.5 trillion worth of debt filings as open data. She credited the Treasurer’s belief that it is “really important for citizens to understand every single ballot measure” as key to this initiative. Treasurer Chiang wanted to help voters avoid having to ask, “Didn’t we just vote on this [initiative?]…what has become of that money?”

DebtWatch is incomplete. State law only requires municipalities to report debt refinancings, not how much they have paid off. But it’s a good start.

  1. Open data progress comes from self-interest and leadership. And self-interest!

Like most state governments’, California’s is not flush with resources to invest in new IT projects to standardize and publish data. Although open data advocates believe the investment is repaid in better external transparency and better internal decision-making, open data projects are hard to initiate and harder to finish. Some large open data projects, including the FPPC’s, require legislators to appropriate dedicated funds.

How have the FPPC, the legislature, the Attorney General, and the Treasurer succeeded?

Phillip Ung emphasized that agencies are more likely to undertake open data projects that promise to automate compliance for a system’s users. The FPPC’s modernization of Form 700 will make life easier for the thousands of public officials who must report on it, as well as improving external accountability and empowering internal analytics.

Sticking to his categorical speaking style, Justin Erlich told the crowd that open data success requires three things: (1) political will from the top, (3) a willingness for culture change, and (3) the ability to deliver. Attorney General Harris’ OpenJustice project employed a startup strategy: the team successfully created a Minimal Viable Product (MVP) in one month. This allowed them to make the case for scaling the project.

Addressing the need for political will to force innovation, Erlich said, “Leaders know that they need to adopt the right technology to enable their [millennial] workforce in the field.” And Jan Ross described Treasurer Chiang’s personal political mission: he “feels personally responsible for with his job,” motivating him to pursue open data solutions to public policy challenges.

  1. Legislators are ready to make open data part of the agenda in Sacramento!

We were honored to host Assemblymembers Ling Ling Chang (R-Los Angeles) and Brian Maienschein (R-SD) who both praised open data projects.

Asm. Maienschein told us, “[B]y releasing [open data], the government is able to tap into the ingenuity of its citizens,” who can use open data sets to build useful apps and platforms. “Open data is a relatively new concept and one that is critically important to increasing transparency and making operations more effective and accountable to the public,” he said. Asm. Maienschein also recognized the benefit to local governments’ internal management: open data “enables localities, it helps localities share best practices with each other…it empowers local governments.”

Maienschein’s Assembly Bill 169, which was passed into law last year, set requirements for California local governments’ open data publication.

But both Maienschein and Chang noted that the state has embraced few sweeping open data transformations. Unlike other states, for example, California does not publish detailed, checkbook-level spending information as open data.

Asm. Chang said she would like to “see complete transparency with the budget…I feel like [the state] government is so opaque…[We must dig through] so many layers of complication to get the right information.”

For context, the 2015 U.S. PIRG Education Fund’s annual “Following the Money” report awards CA the lowest score out of the fifty states on how well they provide online access to government spending data. According to the PIRG, “[California] is weighed down primarily by bureaucratic fragmentation of its information. While the state has made some interesting and useful datasets available to the public…California does not succeed in creating a ‘one-stop’ transparency portal” (see page 30).

Addressing the framentation of California spending data, Deputy Treasurer Jan Ross explained that the state’s financial management is jointly run by multiple agencies, including the Department of Finance, Department of General Services, comptroller’s office, and treasurer’s office

But a question from an audience member succinctly brought the focus around full circle: “Are we merely penny counting or are we moving towards data for better decision making?” The Data Coalition recognizes this need to build open data systems in the context of what will be useful for decision makers. Merely adding layers of disclosure or publication, absent an institutional willingness to act on the information, is pointless.

  1. But there’s no plan!

Our California Data Demo Day participants identified six hurdles facing open data transformations in the state.

But most agreed the last one poses the greatest challenge.

  • The availability of funding remained front and center. Deputy Treasurer Ross eloquently summed up the challenge of convincing a legislature to invest in better government projects: “It is difficult to quantify the ROI of an informed citizenry.” However, when asked to describe her ideal “moonshot” reform project, Asm. Chang strongly hinted that budgetary transparency is the primary area she feels legislators would be willing to invest in. “Just to be able to search through the budget would be great.”
  • Speaking to privacy concerns, Assistant Attorney General Erlich described how his team had to work with outside analysts to proactively avoid mosaic effect. “[It is] difficult with the advance of analytics and re-identification techniques” to avoid publishing data sets that implicate the mosaic effect, he said.
  • On the topic of cultural barriers, Asm. Chang observed how the government needs to get comfortable with the idea of “uber transparency.” She added that the emerging Internet of Things (IoT) might serve as a cultural forcing mechanism.
  • Deputy Treasurer Ross observed that agencies see appropriate presentation of open data as a “big lift.” First, if agencies choose to publish visualization tools alongside raw data sets to help the public make sense of the data, they feel responsible for how those visualization tools work. Second, publishing open data risks that quality problems will look stark. Agencies have to simultaneously “show the quality of the data without manipulating it…[and answer the question] is this raw data or curated data,” said Ross.
  • As open data projects age, they must remain technologically relevant. Phillip Ung flatly admitted that the “government will always be the follower of technology.” To survive, systems need to “remain basic and maintain that flexibility…the failed systems are heavy on frills…[You] never want to be in a place where you can’t enact laws because of an [overly complex] computer system.”
  • Participants noted that the state has no comprehensive plan for open data. Without clear expectations on which domains of the state’s and municipalities’ information will be standardized and published, and on what time frame, the technology industry cannot plan investments, said Sloane DellOrto of Digital Deployment, Inc. Last year, Sen. Richard Pan proposed legislation to appoint a chief data officer for California – an official whose job would include creating a plan. But Pan’s bill failed in the Senate.

As a fitting, closing call to action, Asm. Chang challenged the room by saying how it’s “imperative on the industry and the experts to come educate us on all the issues.”

Message received! Representing the emerging open data industry, the Data Coalition will keep up its advocacy in Sacramento.

Two reforms will rank high on the open data policy agenda for California. First, as Asm. Chang emphasized, the state needs reform to fix its fragmented management and publication of spending information. Second, the state needs a comprehensive open data plan. Creating one should be somebody’s job.