Organizations may be leveraging Big Data, but many still struggle with the need to realize value from their initiatives. In the edited interview below, Teradata’s Dan Graham provides some insight into what organizations need to achieve this transition.
Dan Graham is well qualified to speak to Big Data opportunity. In the 1980s, he was an application developer and DBA with a major California bank. From 1989 to 1993, Graham served as senior product manager for Teradata Corporation’s parallel database servers and software, and next joined IBM where he was responsible for product planning on the RS/6000 SP parallel server. After successfully launching the SP, he became strategy and operations manager for IBM’s Global Business Intelligence Solutions. During his GBIS tenure, Graham initiated, staffed, and developed Teraplex Centers to test multi-terabyte workloads in data warehousing proof of concept tests. As enterprise systems GM at Teradata, he was responsible for strategy, go-to-market success and competitive differentiation for the company’s Active Enterprise Data Warehouse platform. Graham currently leads Teradata’s technical marketing activities.
Lyndsay Wise: Many organizations are confused and think that Big Data is beyond their reach. How should organizations define their needs in this area?
Dan Graham: A lot of people think that they’re being passed over by Big Data. They think somebody’s doing a great job of it and they’re not, and they can’t find a way to get involved. The reality is that 80 to 90 percent of customers feel this way.
This goes back to the question, what do you consider Big Data to be? Is it the flood of new data and new types of data? We can look at the flood and say it’s so big – how are we going to deal with it? Most people approaching Big Data start with click streams. You can do click stream analysis in-house or you can pay an agency or a third party to do it for you, but click streams aren’t all that new. We were doing these in 2001 in various places. Some people understand Big Data as volume, veracity, velocity, but Big Data is just data and a lot of it. Then there is this concept of data complexities. Some say that big data equals Hadoop.
In general, 80 percent of what people are doing with big data is analyzing web logs; this is not new and you can do it in Teradata, you can do it in Oracle, and you can do it in Hadoop. You don’t necessarily have to be intimidated, but the time spent learning about these things is certainly precious time and people can’t always spend it.
WISE: Are there specific business challenges that make Big Data management necessary?
GRAHAM: Big Data analysis is inevitable. The primary considerations are achieving the agility an organization needs while applying common sense. If your organization is smaller than $1B in revenue, you may have to pay an agency or a SaaS (software-as-a-service) provider to do some big Data work for you instead of bringing it in-house. Indeed, the SMB may be a large beneficiary of SaaS/clouds, which give them mega-company tools for Big Data at cloud prices. But Big Data analysis is inevitable.
Enhancing customer experience is high on the list of Big Data objectives. One manifestation of this is that nearly all organizations have a public website. Optimizing the visitor experience and leading some of them to book an order is no longer an optional project like it was in 2005. Gathering multi-structured data and analyzing it is inevitable. Corporations cannot resist the pull of the millennials, the mobile phone, the iPads and every other Internet connected device as a means of engaging their customers better.
Sensor data also presents Big Data challenges. The data is simple but the volume of it can be enormous. The main challenge is the lack of standards in sensor data formats, which can cause expense in the ETL (extract, transform, load) process. But analyzing sensor data is fairly easy once the ETL issues are resolved.
Another area where Big Data is being used is new product innovation. Sifting through millions of data samples in discovery mode can reveal new opportunities. Whether it’s a unique customer segment, or a collection of tweets and blogs that cry out for a new product, it’s hard for corporations to crawl out of their bunker and see the buyer with new eyes. But some of those ideas lie out there in Big Data years before someone figures it out and takes action.
WISE: What are some of the challenges organizations face when trying to get value out of their Big Data investments?
GRAHAM: The number one challenge we hear from customers is “How do I get value from Big Data?” It’s ironic. First, I would say that most Big Data business cases are the same ones the data warehouse community has being doing for over a decade, but by adding new forms of data there is a justification for analyzing that data in Hadoop. Text data, machine data, and server logs are the new data and these are often, but not always, analyzed in Hadoop. Now the application is split across the data warehouse and Hadoop with both adding value. So an organization may already be “doing” Big Data applications in the data warehouse, but by adding new data may gain additional accuracy.
The number two challenge is skills – finding the right people to help you deploy the data and the processing. First off, you should take advantage of your data warehouse people. They can learn Big Data faster than others can learn ETL, data mining, MDM, governance, query performance, and optimization. You already have Big Data experts in house.
WISE: Do you see organizations adopting Big Data because they think everyone else is doing it or because of broader organizational initiatives? Are they first evaluating the market to see what others are doing or starting with their own business challenges?
GRAHAM: We’ve seen organizations declare they are going 100 percent with Big Data in Hadoop. There is this concept of piling all of the data into a data lake and all of the good things will happen later. I push back on this, and so do many others – the concept of ‘build it and they will come’ is a fallacy. We went through this with the data warehouse too. For a decade people would just throw everything in there and didn’t know why people didn’t want to use it. When it came time for the annual funding discussion, organizations would say, “don’t put any more money there – look at the mess IT made.” So ‘build it and they will come’ is not a strategy, it’s a hope.
We’re seeing a lot of this data lake with Hadoop. Admittedly, the vendors have not provided enough tools in Hadoop, and as a consequence it’s hard for organizations to look at the data stored there and transform it into data value. A good starting point would be to ask questions like, “where’s the business user, what problems are we trying to solve?” and then let’s go find the data. If you want to put it in Hadoop, put it in Hadoop – not a problem. But if you don’t start with a business problem or keep the business user engaged throughout the whole process, you are going to fail every time.
WISE: Do you have any best practices that organizations can follow to help them move beyond simply managing their Big Data projects towards gaining measurable value?
GRAHAM: There are three things:
- Look inside and outside your industry for Big Data initiatives. Find reference stories. Pay extra attention to whether these projects are in production or something less. Note that if you are in retail or manufacturing or banking, you share many of the same business processes. Reference stories from across industries can provide an innovation in your sector that is commonplace in the other industry.
- As in any major data project, follow an agile methodology with the business users engaged from day one. Always ask “Where’s the money? How do we make an ROI?” This does more to keep the project focused than any manager, any edict or bright idea. This isn’t new but the people who are new never learned this.
- Learn that Big Data means data warehousing, clouds, Hadoop, databases and other technologies. A good engineer uses the best-fit objective of matching a task to the tool that fits that problem. Use all the tools at your disposal. Use a screwdriver for screws and hammers for nails.
