Yesterday, I spent the day at a meeting on "city innovation" at Harvard's Kennedy School, with 30 or so CIOs, CTOs, and other technology executives from around the country. I did a short presentation on predictive analytics and cities (slides here) -- thanks so much to everyone who sent in comments and who emailed me with suggestions. The "aha!" moment of the day came during a coffee break conversation with Boston CIO Bill Oates.  Bill was describing how frustrated he felt by the city's procurement process (this is widely known as a problem across government).  He said that he felt like he was "handcuffed" by having to prove -- up front, and before actually doing anything -- that he wasn't being dishonest, wasn't corrupt, and was serving the city's best interests.  What if, he asked, he could instead proceed ahead and prove -- after the fact -- that his actions were pure.  Using transparency, rather than bureaucracy, to establish accountability and ultimately trust. This strikes me as a big idea. What we have now -- in the era of increasing information liquidity -- is an opportunity to re-think the way we establish trust.  This idea has been proven out by web services (think Ebay, Airbnb, StackExchange), and I think it's time we start thinking about how this applies to public sector policy and regulation. After the conversation with Bill, I ran back to my seat and sketched out the idea, then quickly turned it into a slide for my presentation in the following session.  This is what I came up with:

` The idea that the purpose of bureaucracy and (certain forms of ) regulation is to establish trust is perhaps obvious. But something about it struck me as a new way of looking at things. It's an idea that superblogger David Alpert gets at in his coverage of the Uber / DC fight, which he describes as a conflict between the "permission model" and the "innovation model". I understand that it's hard to get past the permission-based way of thinking.  Before information was available in real-time, it was the best way to make sure bad things didn't happen.  But we have a new tool -- real-time information -- that makes a new approach possible.  Yesterday at Harvard, we were discussing this in the context of government procurement.  At USV, we've been talking about it a lot in the context of online privacy (I'm pushing Brad to write about his idea for this soon). Hopefully you'll find this helpful -- I think I'll be coming back to it w/ some frequency now as we continue to work on this stuff.

It's been a big year for predictive analytics.   I've been following Nate Silver's blog on the election, and his deep data analysis cut through the noise, was consistent, and ultimately proved correct.   And to look at another (eerily prescient) example, look at this 2006 prediction of what a major coastal storm could do to the East Coast. We have lots and lots of data about what has happened, and we're just starting to figure out how to use it. Tomorrow, I'm attending a conference on Innovation and Cities at Harvard's Kennedy School, and I'll be speaking on a panel on predictive analytics and cities.  I'll be joined by New York City's Director of Analytics, Michael Flowers and Chicago's (first ever) Chief Data Officer, Brett Goldstein.  Both Brett and Michael are way deeper on this subject than I am, so my hope is to simply ask some provocative questions, and perhaps give some examples from outside the civic sector. A few weeks ago at the Ford Foundation's Wired for Change conference, MIT's Cesar Hidalgo gave a thought provoking talk on the power of big data and predictive analytics.  A big takeaway from his talk was that by looking at how data is connected -- i.e., focusing on a few of data as a network, rather than as sums of numbers -- we can quickly and compellingly start to see new trends, tell new stories, and predict future outcomes. Cesar presented some research that looked at national exports in terms of connections between products and industries.  By creating such a "map" of the ecosystem, using historical data, it actually becomes relatively easy to guess which sectors will continue to grow and how.  For example, here is a look at South Korea's export economy over time: This simple, but profound, change in approach holds tons of potential for us to understand what's going on in our cities and countries, and better prepare (for economic changes, natural disasters, etc.).  You can play with more visualizations of world economic data at MIT's Observatory of Economic Complexity. So, looking ahead to tomorrow's conversation: the specific topic of conversation is:

Predictive analytics cut across issues and datasets. When it comes to potential new forms of analytics, what are the low-hanging fruit? What are ambitious, longer-term ideas of new ways to use predictive analytics to tackle urban issues? What could/should cities do together?

I have some ideas -- for instance, generally taking an open data and open standards approach at the foundational level (to widen the audience of potential data miners).  Looking for data sets that tell us a lot about how the city works, but might not be the first ones we think of (such as taxi drop off locations, long-distance call originations, tweets, supermarket and other consumer spending data, etc.).  I'll keep noodling on it today and tonight. What do you think?

Yesterday, I spent the day at a meeting on "city innovation" at Harvard's Kennedy School, with 30 or so CIOs, CTOs, and other technology executives from around the country. I did a short presentation on predictive analytics and cities (slides here) -- thanks so much to everyone who sent in comments and who emailed me with suggestions. The "aha!" moment of the day came during a coffee break conversation with Boston CIO Bill Oates.  Bill was describing how frustrated he felt by the city's procurement process (this is widely known as a problem across government).  He said that he felt like he was "handcuffed" by having to prove -- up front, and before actually doing anything -- that he wasn't being dishonest, wasn't corrupt, and was serving the city's best interests.  What if, he asked, he could instead proceed ahead and prove -- after the fact -- that his actions were pure.  Using transparency, rather than bureaucracy, to establish accountability and ultimately trust. This strikes me as a big idea. What we have now -- in the era of increasing information liquidity -- is an opportunity to re-think the way we establish trust.  This idea has been proven out by web services (think Ebay, Airbnb, StackExchange), and I think it's time we start thinking about how this applies to public sector policy and regulation. After the conversation with Bill, I ran back to my seat and sketched out the idea, then quickly turned it into a slide for my presentation in the following session.  This is what I came up with:

` The idea that the purpose of bureaucracy and (certain forms of ) regulation is to establish trust is perhaps obvious. But something about it struck me as a new way of looking at things. It's an idea that superblogger David Alpert gets at in his coverage of the Uber / DC fight, which he describes as a conflict between the "permission model" and the "innovation model". I understand that it's hard to get past the permission-based way of thinking.  Before information was available in real-time, it was the best way to make sure bad things didn't happen.  But we have a new tool -- real-time information -- that makes a new approach possible.  Yesterday at Harvard, we were discussing this in the context of government procurement.  At USV, we've been talking about it a lot in the context of online privacy (I'm pushing Brad to write about his idea for this soon). Hopefully you'll find this helpful -- I think I'll be coming back to it w/ some frequency now as we continue to work on this stuff.

It's been a big year for predictive analytics.   I've been following Nate Silver's blog on the election, and his deep data analysis cut through the noise, was consistent, and ultimately proved correct.   And to look at another (eerily prescient) example, look at this 2006 prediction of what a major coastal storm could do to the East Coast. We have lots and lots of data about what has happened, and we're just starting to figure out how to use it. Tomorrow, I'm attending a conference on Innovation and Cities at Harvard's Kennedy School, and I'll be speaking on a panel on predictive analytics and cities.  I'll be joined by New York City's Director of Analytics, Michael Flowers and Chicago's (first ever) Chief Data Officer, Brett Goldstein.  Both Brett and Michael are way deeper on this subject than I am, so my hope is to simply ask some provocative questions, and perhaps give some examples from outside the civic sector. A few weeks ago at the Ford Foundation's Wired for Change conference, MIT's Cesar Hidalgo gave a thought provoking talk on the power of big data and predictive analytics.  A big takeaway from his talk was that by looking at how data is connected -- i.e., focusing on a few of data as a network, rather than as sums of numbers -- we can quickly and compellingly start to see new trends, tell new stories, and predict future outcomes. Cesar presented some research that looked at national exports in terms of connections between products and industries.  By creating such a "map" of the ecosystem, using historical data, it actually becomes relatively easy to guess which sectors will continue to grow and how.  For example, here is a look at South Korea's export economy over time: This simple, but profound, change in approach holds tons of potential for us to understand what's going on in our cities and countries, and better prepare (for economic changes, natural disasters, etc.).  You can play with more visualizations of world economic data at MIT's Observatory of Economic Complexity. So, looking ahead to tomorrow's conversation: the specific topic of conversation is:

Predictive analytics cut across issues and datasets. When it comes to potential new forms of analytics, what are the low-hanging fruit? What are ambitious, longer-term ideas of new ways to use predictive analytics to tackle urban issues? What could/should cities do together?

I have some ideas -- for instance, generally taking an open data and open standards approach at the foundational level (to widen the audience of potential data miners).  Looking for data sets that tell us a lot about how the city works, but might not be the first ones we think of (such as taxi drop off locations, long-distance call originations, tweets, supermarket and other consumer spending data, etc.).  I'll keep noodling on it today and tonight. What do you think?