Data is all around us, and all but the most cynical of us assume, probably naively, that the organizations we rely on, whether banks, businesses, schools or governments are optimizing the use of the information they have available. But the reality is that in most industries we’re just scraping the surface of what’s available. Is it that we believe that the only organizations that are actively pursuing “data mining” to any degree are secret government departments?

Imagine all the data potentially stored in every email, letter, tweet, photos on Fickr or Facebook, check ins on Foursquare, traffic routing information on Waze, reports on SeeClickFix and then there’s recreation program registrations, as well as operations calls for services and work-orders. What are people saying about the traffic? How many people are using the trail system? How satisfied are people about the garbage collection?

What got me thinking about this was this article on

Consider the following statistic from a recent report from McKinsey & Co.: By 2009, large companies—with 1,000 employees or more—already had 200 terabytes of data stored about all facets of their consumers’ lives… Now add to that mind-boggling amount of data the exponential growth we have seen in social-networking data in the past three years…

In a recent post, I talked about hiring a conductor for your social-media strategy. This person sets in motion the social-media plan and makes certain that all parts of the company are looped in. But how does that conductor know when to change strategies, when to act on “tweets” and when to dismiss them and how to determine what a “like” on Facebook really means for the brand? If we ask consumers on our social-media sites to help us create new products, can we actually analyze that information and digest it into actionable points that will help research and development move forward? Or are these queries destined to end up where so much of data already do? Stored, but not used.

via: Social Media And The Big Data Explosion

“Stored, but not used” resonates with me, even for “old” data in the local government sector, and for social media, I’d suggest that many municipalities are unaware of the stream of information occurring within their boundary, and that most are not capturing it in any sense. The difference for the businesses in the article quoted is that without a deep analysis of social media, it could be extremely hard to prove a return on investment – just because people are talking doesn’t mean they are buying.

For municipalities, just about all of the data relevant to the day to day running is location based, either related to a facility or a location. “Where was that streetlight outage?” “What was the kids program like at the rec centre”. Along with location, date and time is a core element of the data – this allows for trends over time to be analyzed as well as weeding out duplicate issues.

The framework that currently makes most sense for this type of analysis seems to be GIS, but even the most advanced platforms available today are just pieces of the puzzle, not the whole picture. I’m thinking big: document management, calls for service, work orders, SCADA, weather station, stream flow, water meter readings, asset management, pavement management system, crew workflow, GPS vehicle monitoring, subsurface utility engineering, contract performance, social media, assessment authority data, traffic counts, traffic signal data and power usage.

There are three sub-categories of information in this list, which by the way is not definitive in any sense, but is illustrative, giving you an idea of the data that could be collected and analyzed:

  • Standard municipal data, generated from document management, financial, CRM and calls for service software,
  • Social Media data, and
  • Electronic sensor data, from GPS devices, sensors, computerized equipment and data loggers

The challenge for municipalities is not in gathering this data, it is using it and making it useful. The challenge for the software and tech industries is making the process simple and cost effective, without the need to justify the expense with a financial return on investment, although some of this data should already being used to justify levels of service or making decisions based on a risk based assessment. The bigger picture of big data is looking outside of the needs of the organization and determining what opportunities there are to open this data up for regional, national or even global applications. Obviously this is where freedom of information and privacy laws must restrict access to certain data, but there are ways to release data such as water usage in clusters at a level that it becomes difficult or irrelevant to isolate individual properties.

The flipside argument is that local governments are already failing to adequately analyze even 1% of the data they possess, so why would we ever expect them to collect more? And how would they ever use it?

Your thoughts?

Published by Mike Thomas

Mike Thomas P.Eng. ENV SP, is the author of and Director of Engineering at the City of Revelstoke in the Interior of British Columbia, Canada.