What is Big Data?
It is generally defined as high volume, high velocity, or high variety information assets that demand innovative processing to provide business insight and enable enhanced decision making and process optimisation.
Big Data is all around us all the time. Every digital process, every web click, and every social media exchange produces it. It’s transmitted by systems, software, sensors, smart machines, mobile devices, and any other system that transmits information. Location data, social media profile data, emails, images, videos, and analytics data are just some of the more well known types of Big Data.
To understand the scope of the data surrounding us in this digital age, consider that every second:
Instagram photos are uploaded
tweets are tweeted
Google searches are completed
YouTube videos are watched
emails are sent
GB of internet traffic is transmitted
As technology races forward, especially advances in the mobile and cloud computing spectrum, the sheer volume and size of the data that is generated and stored is overwhelming. It is estimated that there will be over 50 billion Internet of Things (IoT) devices, which could collect and exchange data, by 2020. This includes mobile devices, wearable devices, sensors and smart machines. Even more interesting is the fact that only 5% of the data generated by these devices will be the result of direct human interaction.
Due to the sheer volume, velocity and variety of big data, it is not feasible for organisations to analyse and synthesise this information through conventional methods. Investment is generally required. This can deliver huge benefits, but it’s important to ensure your strategy for working with big data is sound, considering whether the considerable investment to engage in a Big Data project will be worthwhile. Big data is an expensive, time consuming, and major undertaking in most situations.
When considering whether Big Data will definitely further your organizational goals, recognise that Big Data is not a new solution or product. It is information that can be used to better understand things, and to allow more effective decision making. It’s important to identify and prioritise the problems that you hope to address with your Big Data strategy, and ensure it will deliver against your business needs.
Analysis of Big Data can be used to identify business opportunities to enhance customer experience, improve process efficiency, perform targeted marketing, reduce cost, develop new products and improve risk management, but there are a number of more specific uses that you should consider before embracing the Big Data movement.
Monitoring and Anomaly Detection
Big data can be useful in its ability to inform organisations when unusual things happen, or possibly, are about to happen. These kinds of notifications can fall into two general categories – monitoring and anomaly detection. Monitoring is helpful when you know what you’re looking for and you need a notification when that thing occurs. Anomaly detection, on the other hand, describes a situation in which you need to know when something unusual happens. You are looking for a notification of unusual activity without necessarily knowing in advance what that something might be.
Case Study
Global payment schemes VISA and MasterCard have exhaustive fraud monitoring and reduction systems that use Big Data to analyse transaction meta data against things such as geography, channel, amounts, market segment and identify if something out of the ordinary happens, enabling detection of potential fraud down to the individual merchant terminal. Furthermore, these models are flexible to allow new aspects to be added within hours. By investing in Big Data programs, these organisations have increased their ability to analyse client transactions and have saved $2 billion annually in the areas of fraud monitoring and detection.
Data mining & analytics
One of the most powerful and common applications of Big Data is data mining, which uses statistical procedures to find unexpected patterns in data, which might include unexpected associations between variables or people who cluster together in unanticipated ways. For example, managers in a supermarket team might find that people who visit their stores in a particular region, on a particular night of the week are generally different from people who come at other times and places. The most common application of data mining, however, lies in the online advertising realm, where the volume of data is so considerable that it’s easy to adapt advertising to suit each user. This gives organisations the ability to tailor services to their customers’ preferences and behaviours.
Case Study
In recent years, Boeing has made significant progress in optimizing maintenance programs by applying statistical analyses of performance data as part of its work with operators and regulatory agencies. This process analyses data from every aspect of airplane maintenance lifecycle and uses a series of algorithms and advanced statistical analysis techniques to identify the optimum maintenance intervals for maintenance inspections. The statistical analysis also calculates the point where the cost of a scheduled maintenance inspection and the cost of schedule interruption are minimized.
Predictive analysis
Predictive analytics is the crystal ball of big data. It represents a range of techniques that can be used to predict future events based on past observations. While people have been trying to predict the future since the advent of man, the size of Big Data and the sophistication of our predictive modelling capabilities have fundamentally changed the way that we look into the future.
Case Study
As shown in the book and the movie, Moneyball, where statistical analysis is used to assist the classification of a baseball player’s scoring ability. In the late 1990’s, Oakland Athletics were near the bottom of the major league in terms of their spend on talent acquisition and needed to recruit talented players at low cost. The standard that had been used by baseball recruiters for years was to look at things like batting averages, RBIs and runs batted in stolen bases. However, utilising the broader baseball data set, Oakland Athletics focussed more on on-base percentage and slugging percentage, two indicators they believed were better predictors of talent. Using this method they were able to secure some incredibly talented players who helped them consistently make the playoffs over a number of subsequent years.
The biggest challenge in implementing Big Data is determining the business value of the investment. Be sure of what you are trying to achieve and the reasons for it. And consider monitoring, statistical and predictive analysis as some key ways you can leverage big data for your business.
