We're creating a State of BI report and are interviewing data/analytics/BI leaders from large enterprises to explore BI's current state, long-term strategies, and AI's impact.
Since 2010, data created, captured, copied, and consumed globally increased from 1.2 trillion gigabytes to 59 trillion gigabytes, an almost 5,000% growth. The rapidly growing volume and complexity of data are due to increasing mobile data traffic, cloud-computing traffic, and the development and adoption of technologies including IoT and AI, driving big data analytics and more data available. Moore’s law also plays an integral role in data growth as computing power increases yearly. Data will drive the future of innovation, development, and business decisions; however, it does not exist without its challenges. Does more data mean more solutions, or will it create a more challenging data environment?
Imagine you are a 5-star chef at a Michelin restaurant, but all of your ingredients look the same and have no labels at all. Do you think you would be able to prepare a world-class meal? This is what it is like for data analysts and scientists at most companies. They’re highly skilled and educated, but a confusing data environment creates sluggish results, enormous pain points and prevents them from being the Michelin star chefs of data. Data discovery is a growing problem as more data becomes available because data analysts need data available at their fingertips without having to ask around and swim aimlessly in the data lake that looks like it suffered from a recent hurricane.
More data doesn't mean higher quality data. For most enterprises, new data is created every second. Analysts are creating data sources, and automated data creation is powering many applications, but not all of this data is usable. Data quality refers to the overall usability of a dataset and its ability to be easily processed and analyzed by its users. For someone deriving insights, navigating which source is qualified, cleaned, and accurate, quality is a make or break point for any analysis. Too often, low-quality data can lead to costly mistakes and miscalculations, which drive poor business decisions. Data trust and confidence can quickly dwindle without properly validated sources. Companies need to ensure that they are maintaining high-quality data sources.
Data is growing exponentially with time. How will businesses manage what data is worth storing and paying cloud hosting fees for? Which data is just adding to the confusion? While working as a data analyst, it is easy to find sources that haven’t been touched in years and were created by someone that left the company. These useless sources add to the storage bill while also throwing off new and seasoned data users alike. Companies need to find a way to manage what data is useful and what data is wasted space.
Despite these challenges, data is a fantastic resource and will help drive innovation and successful business decisions.
After all, "the world's most valuable resource is no longer oil, but data," according to The Economist.
