Lateral strategy: from innovation to infrastructure

No IT industry concept has engendered as much recent debate as “bimodal IT.” Introduced by Gartner Group in 2014, it describes a future in which IT is bifurcated, with “Mode 1″ focused on “traditional and sequential” practices “emphasizing safety and accuracy,” while “Mode 2” is used to describe “exploratory and nonlinear” approaches that are aligned with “agility and speed.”

Bimodal IT presents a conundrum to IT leaders: it identifies them with relatively low-value Mode 1 practices that are defined more by cost than value, while also legitimizing the ‘shadow’ and other Mode 2 IT activities of line-of-business managers who may view IT, and its desire for “safety” oriented objectives like security and auditability, as a bottleneck that is better to avoid than heed. Commentators from many quarters have painted bimodal IT as a label for a one-way road that leads away from the CIO and towards independent IT authority across the enterprise. The most common response has been for traditional IT leaders to look for ways to edge their departments down the path towards Mode 2 operating norms, embracing (in various degrees) DevOps, Agile and/or cloud in the pursuit of alignment with emerging business technology requirements.

This is certainly an understandable approach to bimodal: it is evident that the IT department’s starting point is in well-established Mode 1 activities, and that much of the future of IT is in still-evolving Mode 2 applications of advanced technologies, including cloud, Big Data analytics and IoT, to business objectives that could not be addressed with Mode 1 systems. But is it the right approach? In an interview with InsightaaS, Jake Frievald, vice president of marketing at Information Builders, presented a ‘lateral thinking’ alternative that reverses this logic, connecting Mode 2 objectives back to Mode 1 benefits.

The interview arose from InsightaaS’s interest in the February introduction of iWay Big Data Integrator, “a modern, native approach to Hadoop-based data integration and management that ensures high levels of capability, compatibility, and flexibility.” The product addresses key issues that constrain adoption and use of Big Data analytics: the skills required to work with Hadoop, particularly with respect to data management, and the challenge of translating insights gained from data exploration into knowledge that can be used throughout the enterprise to support the development of better business practices. But despite what Mode 2 proponents might believe, these challenges and opportunities don’t exist in a vacuum – they mirror the issues that traditional IT departments wrestle with on a daily basis. Frievald connects these points in an unusual way: he sees the iWay Big Data Integrator (and related capabilities delivered via Information Builders’ flagship WebFOCUS analytics platform) as a means of both supporting business users in their data exploration and of establishing the relevance of Mode 1 skills and practices as a means of expanding on the value of these Mode 2 operations.

Frievald’s perspective is rooted in his observation that most Hadoop use is exploratory in nature. In many cases, it starts with senior management reading a report or a business journal article that sparks interest in the technology (“wow, I’ve got to have a Hadoop!”). This creates a requirement that balances uncomfortably across the bimodal divide: Hadoop is IT focused, in that it requires advanced technical skills, but “it tends to be responsive to the business community.” Generally, Hadoop is used to support one-off queries that use a large amount of data to answer a specific question; it is relatively rare that these explorations are translated into production-level applications. But there is the potential for an insight to spawn a requirement for “a report, a chart, a graph, a dashboard” – and even where this is not the end result, there is need for data management and skills in technologies like Sqoop and Flume to support the analyst’s inquiries.

There are different approaches to addressing this requirement. The IT department can attempt to exert control over the Big Data/Hadoop environment, but this approach runs the risk of further abrading IT’s relationship with “an increasingly-anxious, and increasingly frustrated-with-the-old-way-of-doing-business business community.” The best approach, Frievald believes, is to “marry up the people in IT who understand the management of the data with the businesspeople who are trying to get value out of the data” through rapid, iterative investigation.

In Frievald’s view, the correct starting point for this engagement is data. “In most data projects,” he believes, “it’s more important to understand the data from a business perspective than it is to have the direct technical ability to manipulate it.” This defines roles and relationships between IT and business users in what Frievald calls “a really important use case:” if “IT is going to be put in the position of being the ‘data pump’ or the ‘data service provider’ for analysis…then IT needs to be responsive enough to be valuable to the business.” Merely exerting Mode 1-type control won’t work in this situation; “right now, the business has already figured out that they can displace certain types of business intelligence by uploading spreadsheets to [an external] tool that lets them do anything they want in an uncontrolled way. Business has figured out that they can break certain [IT] bottlenecks by doing that.” This opens the door to additional use of shadow IT to address Big Data objectives. Frievald sees this as a challenge for traditional IT departments, saying that businesspeople are “looking at Big Data and saying, okay, I need to break bottlenecks here, too” – begging the question, “how is IT going to help them achieve their goals, as opposed to being seen as the bottleneck within Big Data?”

The iWay Big Data Integrator is designed to help IT leaders to work back from this point to a position where IT enables rather than impedes data exploration. It is a toolset that acts as a “veneer” over Big Data, responding to the tension inherent in the bimodal construct by providing “an interface that allows [experienced data managers] to make the right choices for the businesspeople very rapidly, and deploy to Hadoop very rapidly, so that the businesspeople can take advantage of all the capabilities of Hadoop very rapidly, and get the responses that they want very rapidly.”

This description is apt for support of the iterative, one-off use of Hadoop to answer focused questions (the most common use of Hadoop today), but by coupling iWay Big Data Integrator with WebFOCUS, Information Builders can also support cases where the initial exploration leads to more formal system requirements. Frievald offers the example of a case where insights gained from an initial Mode 2 exploration leads to a need for a more formal Mode 1-type application to capture, present and distribute the information. It may be that “an analyst is pulling information from multiple streams – maybe social media, sales and fulfillment systems. The data has all been dumped into Hadoop; using iWay Big Data Integrator, the data specialist selects some of the information, integrates it together, and dumps it into a data store so that the analyst can do rapid exploratory work. The analyst sees that the resulting insight needs to be sent out to lots of people; but they may also see a need to integrate additional information that wasn’t in the data store. Hadoop can augment the data that was in the data store inside Hadoop, allowing it to be part of the application that is widely distributed.” This demonstrates, in Frievald’s view, that “it’s important to have the flexibility to support all different kinds of applications – and from the WebFOCUS perspective, it’s important to have the ability to support both Mode 1 and Mode 2 inside Hadoop, using the same infrastructure.”

iWay Big Data Integrator

Although examples like this serve to whet the imagination with visions of the Big Data future, Frievald finds that the most common current business objective served by Hadoop isn’t so much scope as speed. He talks about a project with a hospital that “used to try to deal with the batch window by doing an extraction of just the data they needed, followed by a transform and a load.” With Hadoop, the process changes entirely: “we’re finding that Hadoop gives us, for the first time, a really solid ELT [extract-load-transform] process instead of an ETL [extract-transform-load, the traditional approach to BI] process.” This shift has several implications. One is that it is much faster, in part because the data is “easier to manage once it’s inside the [Hadoop] platform,” which eliminates the constraint of the batch window. Another is that once the data is inside Hadoop, “you can do lots of fun stuff” – such as changed data capture that greatly enhances the ability to understand patterns and issues in the operation. And of course, there is the potential for Big Data extensions. The hospital, for example, needs to respond to payers who are focusing more on outcomes than on claims payments. As a result, Frievald says, “hospitals and insurance companies alike have to assemble clinical data plus claims data plus outcomes data, plus they have to onboard information from community care organizations that they’re not directly affiliated with, in order to assemble a full picture of their entire populations – both individual patients and also the community as a whole. And in order to do that, you’re assembling huge amounts of information that you’ve never had before.”

The hospital example helps to demonstrate that the Big Data future is rooted in current business imperatives. Pressed to expand on the point, Frievald quickly rattles off several other areas where the future of Big Data is apparent in current industry practices: businesses like Montreal’s Yellow Pages Group are updating traditional products (a listing in a business directory) by wrapping them in analytics (digital ads with built-in calculators that help small businesses to understand the impact of advertising), police departments that connect internal data with external sources like weather feeds and school calendars to implement predictive analytics, and similar examples of areas where vastly-increased information sources, coupled with analytics tools that empower analysts to explore data rather than wrestling with Sqoop code, unlock opportunities for improved safety, better diagnoses and increased business success.

In the end, Frievald is clear in his opinion of the bimodal conundrum – and the role that Big Data can play in helping traditional IT practitioners to stay relevant in the bifurcated technology world. “Mode 1 is not going away,” he states, and “Mode 1 to Mode 2 is not a one-way street.” Frievald says plainly that the marriage of Mode 1 and Mode 2 is “the biggest advantage that we’re seeing as likely to come out of Hadoop.” It may not have the same imaginative appeal as data services that transform everyday products, but this marriage is a key issue in enterprise IT operations: as Frievald says of traditional platforms and innovative solutions, “these are two modes that IT will continue to need to run in for the foreseeable future.”