This post was contributed by the Global Knowledge UK blog. For more courses and information, visit Global Knowledge UK.
Although the term ‘big data’ has been one of the buzzwords of 2012 many companies have been well aware of the escalation in the amount of data and information that is being acquired and stored and the potential value of being able to make effective use of this in gaining and maintaining a competitive advantage. A recent survey by Oracle carried out in the USA indicated that:
- Data volume has increased by an average of 86% in the last 2 years
- 60% of executives rate their companies unprepared to leverage the data and cite significant gaps in people, process, and tools
- 97% say they need to make changes to improve
- 93% of the executives believe they are losing revenue at an average rate of 14% annually
The focus on operational data tends to obscure a similar increase in the growth of unstructured information in the form of sales and customer service reports, product development documentation and market intelligence. This information is crucial in providing a context to assess the significance of trends and outliers derived from data analytics. A survey by AIIM suggests that 60% of companies regard it as essential that data and information can be integrated but only 2% are currently able to do so.
Although big data is often defined in terms of volume, velocity, variety and value these are all relative, and even smaller companies may be running into problems delivering business-critical information. The rapid growth in the use of smartphones and tablets by customer and supplier-facing staff adds to technical complexity.
Big data is probably the first time that open-source software applications have been positioned as a scalable solution for an enterprise application, notably Apache Hadoop. Hadoop offers a distributed environment for data-intensive applications, whilst Apache Lucene and Solr provide powerful solutions for searching very large databases of text information. Several of the market leaders providing what are referred to as “unified information access” (UIA) solutions are based on open-source software, including LucidWorks and Attivio. Attivio recently gained a $37M venture capital investment, an indication of the market potential in this sector.
It is now widely accepted that IT technology will not provide a turn-key solution to big data management. Another big data V is veracity. If data is not of sufficient quality by the time it has been integrated with other data and information a false correlation could result in the organisation making an incorrect analysis of a business opportunity. Many studies are now indicating the importance of data analysts in validating and interpreting data but people with these skills are in very short supply. The same is true of people with the skills to manage large enterprise search applications which (unlike big data applications) will be used by most, if not all, the employees in a company.
Obstacles to taking full advantage of big data include making the business case for the investment in technology and additional specialist skills, understanding how to use big data analytics, data quality, management support and recruiting and then training staff with the required skill sets.
Companies are now appreciating for the first time just how important effective information management is to their future success. With the emergence and expanding adoption of big data, organisations worldwide are discovering entirely new ways to compete and win. They are transforming themselves to take advantage of the vast array of information that is available to improve decision-making and performance throughout the enterprise.
The responsibilities of the Chief Information Officer now have to shift from technology management to information management, helping their company take a decisive step into using its vast resources of information to improve performance and competitive advantage. There is a significant first-mover benefit.
The author
Martin White is Managing Director of Intranet Focus Ltd., and acts as a vendor-independent advisor to corporate and public sector organisations on information management strategy. He is the author of the book Enterprise Search, recently published by O’Reilly Media.