Over the years, technological innovation has always caused shifts in employment. New processes and tools replace work previously done by people who must move on to find other productive ways to contribute to society. We are now on the cusp of an revolution that will dramatically change the employment landscape within our lifetimes: Robots, automated computer processing, artificial intelligence - whatever you call it - will reshape tasks from driving trucks to completing your tax return. So what advice do you give to a young person contemplating their career opportunities today? Our hypothesis (or "hackothesis") is that ultimately, innovation drives growth, and the areas (both geographically, and in terms of specialization) where it drives that growth become the most lucrative points for people seeking employment.
We analyzed a range of datasets as proxies for innovation, industry activity and employment to determine whether the data supports our hackothesis. We worked to align this data by location, across time, to identify interesting trends. We used Microsoft Azure, Microsoft Azure SQL Database and Microsoft Power BI for the majority of the data processing, cleansing, storage and analysis. Larger data sets (in particular, the ABN lookup dataset) were pre-processed in Hadoop to extract the necessary attributes.