'Service registers' on governmental websites

One of the things we’ve done a lot of at Democracy Club is to gather data from all over local government into a single place. Either we’re trawling through 400-odd PDFs looking for candidates, or we’re sending freedom of information requests to councils to get their polling station data.

Let’s be honest, this is boring. Really boring. And it’s not something any of us actually wants to be doing, but there isn’t really any other way.

We’re not alone though: we know of a few other organisations who have to do this sort of work, such as the UK Parliament and the LGA for election results data.

The work we’re all doing falls into two areas:

1. Finding what you’re looking for on every council web site (about 400 of them for general elections).
2. Creating a standard format from each source.

For this post, I’m going to ignore point two. Standards are hard to agree on, and many people have spent many hours arguing about them. We’ll save that for another day.

Even before we have agreed standards, we can cut down the time it takes to harvest data from all these web sites by solving point one.

Our proposal is this. All web sites at the root of each government domain should publish a sort of ‘index’ of all the services and data they offer, with URLs to these services.

For example, Trumpton Council’s website at https://trumpton.gov.uk/ would serve a file at /services.json or /data.json (we can argue about the exact name and format later).

UPDATE: Thanks to @gwire and @theMarkBeech for pointing us to RFC5785. This suggests that the path to the file should start with /.well-known/. The name of the file is still up for debate though.

That file might look something like this:

{
    "version": "0.0.1",
    "schema": "government-service-index",
    "last_updated": "2015-09-03",
    "services": {
        "contact-information": "/contact-us/",
        "bin-collection-days": "/bin-collection/",
        "council-meeting-calendar": "/council-and-democracy/meetings",
    },
    "registers": {
        "500-pound-spend-data": "https://data.trumpton.gov.uk/accounts/spending/500-pound/",
        "gpc-spend-data": "https://data.trumpton.gov.uk/accounts/spending/gpc/",
        "senior-salaries": "https://data.trumpton.gov.uk/accounts/salaries/",
        "authority-land": "https://data.trumpton.gov.uk/gis/land/",
        "democracy": {
            "elections": "https://data.trumpton.gov.uk/democracy/elections/",
            "councillors": "https://data.trumpton.gov.uk/democracy/councillors/",
            "polling-stations": "https://data.trumpton.gov.uk/democracy/polling-stations/",
    “polling-districts”: "https://data.trumpton.gov.uk/democracy/polling-districts/”,
        }
    },
}

The reason that the council should host this file is that it then becomes a part of the digital infrastructure that the council is maintaining, not something that they have to do somewhere else.

With this system, we only need to agree (or enforce):

1. The URL of this index file
2. What the ‘keys’ are (i.e., that “contact-information” is the term that’s used for contact pages).

There is some excellent existing work on the second point. First, there is a taxonomy of local services on local.direct.gov.uk and a central list of URLs to pages (not data) on council websites.

Unfortunately, this is fairly out of date, and it might not be maintained anymore (the domain name is a clue).

The other bit of positive work in this area is from the Local Government Association, who have “inventories” that point to data that the council publishes.

The final piece in this puzzle is the list of all the domain names. This fits nicely within the scope of GOV.UK, who should do two things:

1. Provide a register of all domains in government (this is harder than it might seem at first)
2. Provide a central validator and cache of all indexes.

Once this is done, our job will become a lot easier.

Now, about those standards…

P.S. This is not a new idea – it’s been inspired by a few existing implementations:

1. I got this idea directly from Christopher Gutteridge at Redecentralize Conference. He’s built equipment.data using more or less the same system to make a register of university equipment.

2. The “Office of Management and Budget” in the US use this system to aggregate all sorts of governance data from all over the US government agencies.

3. For years, websites have used ‘robots.txt‘ to communicate to web crawlers how they should interact with their site. The idea in this post is more complex than a robots.txt, but it works on the same basic principle.