Maximizing and managing big data with SOA middleware
A comprehensive collection of articles, videos and more, hand-picked by our editors
More than 50% of businesses were engaged in big data initiatives last year, according to a Tata Consultancy Services (TCS) Ltd. survey of 1,217 big data and analytics, IT management, and sales and marketing professionals. Yet, while many organizations are striving to make better use of big data, most encounter problems in the notorious three Vs: volume, variety and velocity, said Dr. Satya Ramaswamy, TCS vice president. Overcoming these...
hurdles calls for restructuring workgroups, as well making as other technology and organizational adjustments.
Of the companies that undertook big data in 2012, just over 40% predict seeing a return on investment (ROI) of more than 25%, data indicates. Ramaswamy believes such findings indicate more organizations could be giving the technology the green light down the road.
Conquering big data initiative roadblocks
After the survey, the TCS research team was surprised to learn that four of the top five challenges reported were structural, not technical, said Ramaswamy. The top-rated challenge was business units' failure to share information across the organizational silos. Another hurdle is building trust between data scientists and IT managers.
Eliminating separate departments and creating cross-functional teams can address the collaboration challenges. "One of the best practices we saw is companies setting up independent units that are different from traditional units to drive the big data initiative," said Ramaswamy.
Forrester Research Inc. principal analyst Mike Gualtieri advises that two core practices are musts to drive structural change:
- Implementing a big data analytics platform; and,
- Using predictive analytics to determine what data is valuable.
The difference between business intelligence (BI) and predictive analytics is vast. Traditional BI applications are used to determine which data is important, whereas predictive analytics uses machine learning algorithms to obtain the same information. "That's why it is important to store and analyze all your data," Gualtieri said. "You don't know upfront what is the signal from the noise."
Breaking down technical barriers
Handling the three Vs was cited as the top technical issue, largely because analysis calls for considering all the data, not just what is immediately important. "One challenge for the velocity of big data is to capture and store it as fast as it comes at you," said Gualtieri. "Streaming technology is a critical component of a big data platform." Use scale-out architectures and tools like Hadoop to manage fast-growing, large quantities of data, he advised.
Dr. Satya Ramaswamyvice president, TCS
Choosing the appropriate big data platform is critical to overcoming technical issues, Ramaswamy noted. One simple, yet sometimes overlooked, tactic for selecting platforms is for architects to make sure they clearly understand their organization's priority on big data.
"Architects should investigate all specific areas outlined by the enterprise architecture for the big data platform fitment," said Ramaswamy. "Smaller proof of concepts/technology in the specific areas could be done to make sure each and every one of the guidelines is met."
Other challenges and good news
Staffing and budgets are related challenges for the IT managers and big data/analytics pros surveyed. Getting top management to invest in big data initiatives, particularly in training, is a top problem for IT management. Analytics pros have trouble finding and hiring data scientists capable of managing large amounts of structure and unstructured data and providing creative analysis and action.
Both groups need to work together to inform top management that investing in big data practices provides competitive advantages and, as stated above, gains strong ROI, says Ramaswamy. Also, he noted, companies that have created and funded big data analytics centers of excellence have been successful in attracting and retaining data scientists.
The TCS survey showed that almost all respondents who did not have a big data program said they will have one by 2015. "Clearly, one thing I see [is that] the adoption of big data is going to become universal," said Ramaswamy. The obvious conclusion is that companies that don't invest in big data processes will be in the minority in the near future and, thusly, lose competitive advantage.