It's been a big year for predictive analytics.   I've been following Nate Silver's blog on the election, and his deep data analysis cut through the noise, was consistent, and ultimately proved correct.   And to look at another (eerily prescient) example, look at this 2006 prediction of what a major coastal storm could do to the East Coast. We have lots and lots of data about what has happened, and we're just starting to figure out how to use it. Tomorrow, I'm attending a conference on Innovation and Cities at Harvard's Kennedy School, and I'll be speaking on a panel on predictive analytics and cities.  I'll be joined by New York City's Director of Analytics, Michael Flowers and Chicago's (first ever) Chief Data Officer, Brett Goldstein.  Both Brett and Michael are way deeper on this subject than I am, so my hope is to simply ask some provocative questions, and perhaps give some examples from outside the civic sector. A few weeks ago at the Ford Foundation's Wired for Change conference, MIT's Cesar Hidalgo gave a thought provoking talk on the power of big data and predictive analytics.  A big takeaway from his talk was that by looking at how data is connected -- i.e., focusing on a few of data as a network, rather than as sums of numbers -- we can quickly and compellingly start to see new trends, tell new stories, and predict future outcomes. Cesar presented some research that looked at national exports in terms of connections between products and industries.  By creating such a "map" of the ecosystem, using historical data, it actually becomes relatively easy to guess which sectors will continue to grow and how.  For example, here is a look at South Korea's export economy over time: This simple, but profound, change in approach holds tons of potential for us to understand what's going on in our cities and countries, and better prepare (for economic changes, natural disasters, etc.).  You can play with more visualizations of world economic data at MIT's Observatory of Economic Complexity. So, looking ahead to tomorrow's conversation: the specific topic of conversation is:

Predictive analytics cut across issues and datasets. When it comes to potential new forms of analytics, what are the low-hanging fruit? What are ambitious, longer-term ideas of new ways to use predictive analytics to tackle urban issues? What could/should cities do together?

I have some ideas -- for instance, generally taking an open data and open standards approach at the foundational level (to widen the audience of potential data miners).  Looking for data sets that tell us a lot about how the city works, but might not be the first ones we think of (such as taxi drop off locations, long-distance call originations, tweets, supermarket and other consumer spending data, etc.).  I'll keep noodling on it today and tonight. What do you think?

It's been a big year for predictive analytics.   I've been following Nate Silver's blog on the election, and his deep data analysis cut through the noise, was consistent, and ultimately proved correct.   And to look at another (eerily prescient) example, look at this 2006 prediction of what a major coastal storm could do to the East Coast. We have lots and lots of data about what has happened, and we're just starting to figure out how to use it. Tomorrow, I'm attending a conference on Innovation and Cities at Harvard's Kennedy School, and I'll be speaking on a panel on predictive analytics and cities.  I'll be joined by New York City's Director of Analytics, Michael Flowers and Chicago's (first ever) Chief Data Officer, Brett Goldstein.  Both Brett and Michael are way deeper on this subject than I am, so my hope is to simply ask some provocative questions, and perhaps give some examples from outside the civic sector. A few weeks ago at the Ford Foundation's Wired for Change conference, MIT's Cesar Hidalgo gave a thought provoking talk on the power of big data and predictive analytics.  A big takeaway from his talk was that by looking at how data is connected -- i.e., focusing on a few of data as a network, rather than as sums of numbers -- we can quickly and compellingly start to see new trends, tell new stories, and predict future outcomes. Cesar presented some research that looked at national exports in terms of connections between products and industries.  By creating such a "map" of the ecosystem, using historical data, it actually becomes relatively easy to guess which sectors will continue to grow and how.  For example, here is a look at South Korea's export economy over time: This simple, but profound, change in approach holds tons of potential for us to understand what's going on in our cities and countries, and better prepare (for economic changes, natural disasters, etc.).  You can play with more visualizations of world economic data at MIT's Observatory of Economic Complexity. So, looking ahead to tomorrow's conversation: the specific topic of conversation is:

Predictive analytics cut across issues and datasets. When it comes to potential new forms of analytics, what are the low-hanging fruit? What are ambitious, longer-term ideas of new ways to use predictive analytics to tackle urban issues? What could/should cities do together?

I have some ideas -- for instance, generally taking an open data and open standards approach at the foundational level (to widen the audience of potential data miners).  Looking for data sets that tell us a lot about how the city works, but might not be the first ones we think of (such as taxi drop off locations, long-distance call originations, tweets, supermarket and other consumer spending data, etc.).  I'll keep noodling on it today and tonight. What do you think?