EVOLUTION VERSUS REVOLUTION
The big data storm has rocked the current analytics infrastructure for many marketers. In most cases, the analytics infrastructure was not intended to deal with the volume, variety, or velocity of data anticipated from these new sources. Most marketing organizations were not equipped for handling the volumes of data, engaging in collaborative influence, and orchestrating across many organizations within and outside the corporate boundaries. The organizations may not have the right measures in place to track progress at this fine-tuned level of collaboration. The success of campaigns may be defined in silos, making it hard to collaborate across channels. The chosen tools for data integration, storage, or data mining were unable to scale to these new requirements. How does a marketing organization upgrade their current environment? The upgrade involves process, people, and technology. While it is easy for a technologist to offer a greenfield analytics environment, it may require a massive transformation of the business processes, measures, KPIs, skills, and organizational relationships. How do we deal with change at this magnitude without seriously disrupting a well-functioning organization, which may not be optimally running today, but is ill-equipped to handle the extent of change?
Earlier in this chapter, I discussed the extent of changes in the business processes, measures, collaborative objectives, external relationships, and skills. Fortunately, these changes are already being witnessed by the marketing organization today. As I studied the organizations in a number of industries, I found that marketing organizations are at different levels of maturity and that most leaders are not just at the receiving end. They are driving these changes and often leading the charge to other organizations, which support them with support processes, data, or technologies. Even where the maturity is low, marketing organizations have been able to use external services to drive significant change.
Do we start from people and process changes, or use the major shift in technology as a catalyst for organizational change? We can either start from the current marketing function and evolve into the new marketing function, making incremental changes in people, process, and technology, or make a radical departure from the past and create a new marketing analytics platform for a pilot organization and use the experiment to choreograph major changes in the organization. Both approaches have obvious pros and cons. In this section, I describe the three alternatives and discuss what would tilt us in one direction or another for a specific implementation.
With the serious investment in IT organizations, the well-organized Business Intelligence (BI) environment is the hardest to change. A typical big data analytics environment requires three significant advancements in the IT system. First, it must reduce latency by an order of magnitude, providing accessibility to data in minutes or seconds as opposed to hours or days. Second, it must increase the capacity to store data by an order of magnitude, moving from terabytes to petabytes. Third, it should have the ability to ingest external data and align it to its customers and products, and participate in external communications using the insights gathered from the analytics platform.
The big data technologies come with a significant cost advantage. The software cost is much lower because of the crowdsourced open- source components, which have also reduced the costs for commercial offerings. Because the architecture is typically built on commodity hardware and requires fewer administrators, the cost, too, is reduced by an order of magnitude. So, the good news is that we can change the IT into a self-funded model. That is, savings pay for the cost of change. However, these implementations require a commitment to big data analytics and a strong desire to migrate from the current platform. What if we have already invested a large IT budget in conventional BI? How far do we go in the first phase? Do we replace the current data warehouse architecture or augment it with big data analytics tools?
Automation is often the biggest catalyst for change. It can also be the most serious inhibitor to change. In a typical “traditional” architecture, there are a set of components for ingesting data, a set of components for storing the data, and a set of components for analyzing the data and then feeding the results into a set of actions or reports. Since all the data must be routed via a storage medium using a data warehouse, the storage, organization, and retrieval of data creates a bottleneck. Typically, the traditional approach requires a reorientation of the data from the data source to a system of record and then into a set of models conducive to analytical processing—which typically requires a number of data modelers, database administrators, and extract, transform, and load (ETL) analysts to maintain the various data models and associated keys. Changes to the business environment require changes to models, which cascade into changes across each component and require large maintenance organizations. These maintenance organizations are distributed between marketing and IT groups and must be reoriented to deal with big data. Many IT architects have already started to break away from this traditional model. Today’s analytics engines do not strictly follow this paradigm, and they significantly reduce the model maintenance costs by reducing the need for representation and key-driven performance tuning.
As I study a marketing organization’s plans for radical transformations, there are tea leaves available to assess the organization’s maturity and will to change. Most leading organizations have adapted big data at the strategic level. How do we sense that? Here are a couple of important signs:
- 1. Has the organization declared data to be a strategic asset and decided on investment in data to develop a competitive marketing position? Most leaders have recognized that the “bit buckets” are full of meaningful insights, which must be stored and harvested irrespective of the cost.
- 2. Has the organization started to engage in a new set of business partners for the strategic alignment of marketing programs to drive the use of big data? Most leaders are using the term “monetization” as a way to define these programs. Using business partners, they are able to move rapidly toward their monetization goals, which engage customers in novel ways, as described in chapter 4.
- 3. Has the marketing organization developed a strong link with an IT organization? Classically, the CIOs reported to the CFO or COO. Marketing used to be a secondary objective of the CIO, while the big jobs were the revenue or Enterprise Resource Planning (ERP) implementations and operations. Very often the relationships were adversarial. By their very nature, marketers drove change in information definitions, while information technologists fought for governance and control. This is changing. In many leading organizations, marketing is the most important customer for the IT organization. The CIO may even be reporting to the CMO.
The revolutionary approach involves creating a brand-new big data analytics-driven organization. Typically, it starts with a forward-looking marketing organization that has decided to use information as a competitive strategy. The marketing organization is seeded with analytics- driven individuals and has adopted a series of KPIs to measure their performance using the power of big data.
The marketing data lake in these organizations is in the new environment, which naturally scales to the velocity and volumes of big data.
This approach has been adopted by many greenfield analytics-driven organizations. They place their large storage in the Hadoop environment and build custom analytics engines (often created using custom hardware and software) on the top of that environment to perform orchestration. The conversation layer uses the orchestration layer and integrates the results with customer-facing processes. The stored data can be analyzed using big data tools. This approach has provided stunning performance.
If an existing IT organization must be transformed to create the big data analytics environment, the cost in skill and technology transformation is substantial. It radically changes the roles and skills for the IT organization and places many more technical activities in the marketing organization. Most of the marketing organizations where this approach has worked were analytics-driven high-tech or electronic commerce companies. Analytics in these companies is not an afterthought, but a competitive edge.
In a typical evolutionary approach, big data becomes an input to the current BI platform. The data is accumulated and analyzed using structured and unstructured tools, and the results are sent to the data warehouse. Standard modeling and reporting tools now have access to social media sentiments, usage records, and other processed big data items. Typically, this approach requires sampling and processing big data to shelve the warehouse from the massive volumes. The evolutionary approach has been adopted by mature BI organizations. The architecture has a low-cost entry threshold as well as a minimal impact on the marketing and IT organizations, but it is not able to provide the significant enhancements seen by the greenfield operators. In most cases, the BI environment limits the type of analysis and the overall end- to-end velocity. All the big data flows through the new platform, while conventional sources continue to provide data to the data warehouse. We establish a couple of integration points to bring data from the warehouse into the analytics engine, which would be viewed by the data warehouse as a data mart. A sample of the new data stream data would be abstracted to the data warehouse, while most of the data would be stored using a Hadoop storage platform for discovery.
The hybrid approach provides the best of both worlds; it enables the current BI environment to function as before, while siphoning the data to the advanced analytics architecture for low-latency analytics. Depending on the transition success and the ability to evolve skills, the hybrid approach provides a valuable transition to full conversion.
This chapter summarized how the marketing organization is changing to reflect the changes in the marketing function. I started with the changes to marketing research, media planning, and related metrics and key process indicators. I then discussed the changing nature of advertising and its external relationship with advertising agencies. Then, I discussed the changes to product management, and how product marketing and product engineering are coming together, driven by a need to deal with mass customization. I described the changes in skills and the increasing emphasis on data science. I discussed the new role for marketing communication in engaging and monitoring customer interactions in external media, such as social media. Finally, I discussed the changes in IT and how it can either be an inhibitor or a catalyst in forcing change.