Showing posts with label customer data. Show all posts
Showing posts with label customer data. Show all posts

Monday, December 15, 2025

What Comes After the Composable CDP?

For a bunch of smart people, the technology community can be awfully slow learners. One mistake they keep repeating is believing that the latest technology is a “silver bullet” that solves all problems. This despite years of experience finding that (a) no technology solves all problems and (b) every new technology is eventually replaced by something else.

The Customer Data Platform industry is as prone to these errors as everyone else. The current silver bullet is composability, which is touted as the ultimate solution to problems of CDP cost and complexity. Just hearing the claim should lead anyone to realize it’s not true: composability solves some problems in some situations (i.e., unnecessary data movement at firms with sophisticated technical resources) but it won’t help companies that lack the necessary data infrastructure or staff resources. I suspect that even the most ardent composable enthusiasts would agree with this more nuanced view if you pinned them to a wall.

It's a step in the right direction to move past “composability is great” to discussing “when composability is the best solution.” But it’s also important to address the second flaw in the “silver bullet” fantasy, which is to assume that the best current technology is the best future technology. In other words, it’s worth asking, what comes next?

Remember, composability isn’t a solution: it’s an architecture. The fundamental argument for composable CDPs is that it’s better to have an architecture that uses the enterprise data warehouse as the primary store for customer data, rather than an architecture where customer data is assembled, stored and accessed in a separate CDP database. So the question about what’s next is really a question of what other architectures are possible.

I can think of two candidates.

  • Read customer data directly from the source systems, without assembling it in any persistent database (i.e., neither a data warehouse nor a CDP). This vision of real-time, on-demand customer profile assembly is what many people thought a CDP should be back in the early days of the industry. At that time, ten to fifteen years ago, the existing technology simply couldn’t make it work. Many source systems didn’t allow real-time access, internal networks couldn’t handle the volume, and the processing costs to assemble profiles in real-time were too high. The problem pre-dates CDPs, of course: the need to make data accessible by pulling it from source systems into a separate database is why data warehouses were first created in the 1980s. There has certainly been progress in recent years, although I can’t say that the necessary technology is fully available. Still, the scope of what can be done in real time continues to expand, which diminishes the share of data that must be assembled in a warehouse. This, in turn, diminishes the relevance of the warehouse-centric composable architecture.

  • Create customer digital twins. I’m highly skeptical that AI can accurately mimic customer behavior: this seems most likely to reinforce mediocrity by failing to predict unexpected behaviors (the most interesting kind) or reactions to novel situations (which includes the most important  innovations). But I can still imagine creating a digital twin for each customer, updated in real time with data captured across all company systems. The twins would be self-contained objects (or maybe agents) that can be queried any time a company system wants to find the best action for a particular customer in a given situation. While their role might resemble the customer profiles assembled in today’s data warehouses, I’m sure the technology will be quite different. I can't predict the specifics of that technology but am sure that clever people somewhere are already working on it. My only point here is that, again, the architecture won’t look like the warehouse-based composable CDP.

There are surely other possibilities that haven’t occurred to me. In fact, I’d offer better than 50/50 odds that the next CDP silver bullet will be something I haven’t listed. Placing the right bet matters a lot to investors and developers but not so much to users, who can wait to see what becomes available. 

What does matter to users is identifying their requirements.  This is something that debates about architecture only distract them from doing. So my fundamental recommendation is that CDP users think less about chasing the silver bullet and more about building the golden record – that is, how to create the accurate, unified, accessible customer data that a CDP using any architecture is intended to provide.

Saturday, May 24, 2025

Mapping AI-Based Martech Using the STIB Innovation Diffusion Model




TL;DR: This post fleshes out the STIB model of innovation diffusion, tests it by classifying recent customer management product announcements into STIB categories, and draws some insights about the model and customer management trends.  As a bonus, it proposes a future of "hyper-personalized, financially-optimized treatment of each customer during each interaction across all channels."
 
The Substitution-Transformation-Infrastructure-Business Model (STIB) model of innovation diffusion that I presented in my last blog post is intriguing, but does it reflect reality? One measure is whether it provides a useful classification of industry products. To test this, I looked at the past ten weeks of product news in the CDP Institute Daily Newsletter, which came to about 50 items. Results were quite interesting.

But before we get to that, let’s flesh out the model a bit.  It describes two core elements: 

  • Innovation: the development whose impact is being measured.  This might be electric motors or internal combustion engines.  (More formally, innovation is a change in technology, where technology is defined as the tools, methods and knowledge used to perform a task.  "Technology" in this sense isn't always "technical": a new marketing methods and financial tools can be important innovations.  But I digress.)
  • Application: the process or product changed by the innovation being studied. This might be manufacturing in factories (a process) or transportation vehicles (a product).  The application is assembled from components, which might be physical objects in a product or steps in a process workflow.  The same innovation might affect many different applications but it's often more useful to study just one.

The model describes how an application changes in response to an innovation.  This change comes in three stages:

  • Substitution: the innovation is deployed as an exact replacement for one component in the applications, without changing anything else.  Electric motors were hooked to turn factory shaftwork instead of water wheels; internal combustion engines replaced horses for powering carriages.
  • Intermediate States: this is a period of experimentation. Multiple components are changed as the industry explores how to best take advantage of the innovation. The innovation itself continues to mature during this period, opening up additional possibilities. 
  • Transformation: the industry settles on an optimal design incorporating the innovation.  This "final" design is fundamentally stable although incremental improvements continue. Factory tools incorporate built-in electric motors; motor vehicles locate the engine and drive train most efficiently.

The model also tracks changes over time to technologies beyond the application itself:  

  • Infrastructure: this includes products, suppliers, and processes that support the transformed application or are changed as a result.  Electric motors require generating stations, power grids, and new machine tools, while they result in more efficient factory designs. Parts and materials manufacturers support motor-driven vehicles while auto ownership spawns new roads, filling stations, employment opportunities, legal frameworks, and more.
  • Business model: this describes business methods such as revenue sources, pricing models, marketing, distribution, customer service, funding, and ownership models. Larger factories lead to scale economies that favor large, national brands.  Complex auto technology favors large organizations with deep resources for research, mass production, national advertising, and dealer networks. 

The graphic illustrates relationships among these elements.* 

  • In the initial state, the application, infrastructure and business model are all in their baseline (i.e., pre-innovation) configuration.  
  • This is followed by substitution, where the innovation is applied to make a slight change in the application while infrastructure and business model are unaffected.  
  • In the third stage, intermediate states, developers try variety of changes to the application and the infrastructure starts to mature in response to growing demands.  Some will be a dead end (direct current electrical systems, steam powered cars). Others may hit on the transformed design before infrastructure is available to support them.  Vendors may also fail if they don't change quickly enough to keep up with competition.  
  • In the final stage, developers converge on a transformed configuration and application change comes more slowly.  Some companies drop out or merge as the market consolidates. The infrastructure continues to evolve and new business models start to emerge.  

The Future of Customer Management

Applying the STIB model requires specifying the application being changed and the innovation that is changing it. This might be tricky in some situations, but it’s pretty simple in the current context: the application is customer management while the innovation is artificial intelligence.

What’s not simple is predicting the form of the transformed application. Hindsight about factory equipment or auto design is easy, but we don’t know how customers will be managed in the future.  This prediction isn't strictly necessary: we can simply observe changes as they occur without assessing their relation to the final state.  But that assessment helps to organize the analysis and gives buyers and developers decide where to invest their resources.

Predicting the transformed state is a thought experiment: identify the constraints imposed by the current technology and imagine what the ideal design would look like if those constraints were removed (and replaced by whatever constraints the innovation imposes.)

As I see it, the dominant feature of customer management today is the number of separate steps that are performed by different people. There’s a creative team with a multi-step process to develop content, a targeting team with a multi-step process to select audience segments, a media team with a process to buy advertising, and multiple operations teams with multi-step processes to deliver messages via websites, email, social media, connected TV, digital video, games, out-of-home, podcasts, and elsewhere. The reason we have so many teams with so many steps is the limited capability of individual humans: each can be expert in only a narrow area, so many must work together to deliver a complete result. And, because they’re human, each person can only deliver a relatively small number of outputs over time.  Thus, each output must apply to many people to deliver messages to everyone.

AI removes this limit. At least in theory, a single AI agent can expertly execute all steps in the messaging process, combining content creation, audience selection, media buying, and delivery. In practice, a single super-agent is less likely than a master agent that calls on specialized subordinate agents.  But so long as those agents complete their work almost instantaneously, the process will still function as a single step. Ideally, these agents would view all available data and collaborate with each other: content development would be informed by audience characteristics; media buying would take into consideration other channel opportunities; and so on. All these decisions would be coordinated to produce the highest total value: for a given advertising impression, the system(s) might simulate the results of several different creative treatments for several different offers for several different products, and then select the best creative/offer/product combination – or conclude that even the highest possible value for that impression is too low to justify the investment. This requires a sophisticated value prediction algorithm which would consider long-term as well as immediate results.

In addition to speed and coordination, the process would be completed at near-zero incremental cost.


In short, I see the transformed state as hyper-personalized, financially-optimized treatment of each customer during each interaction across all channels.
  
This is a radical change.  (That's why they call it a transformation.)  There are no more campaigns (defined as a predefined sequence of messages), no more audiences (defined as a group of customers who receive the same messages), no more content (defined as a fixed combination of text, images, and offers), and no more media buys (defined as group of impressions purchased together). What we have instead is a master agent orchestrating customer interactions across all channels to optimize use of budgets, data, customer attention, products, staff, and computing power. That agent would call internal and external data to guide its actions and would deliver messages across both internal (owned) and external (paid) media.

In the transformed world, today's creative, analytics, media, and operations departments have largely vanished, apart from a few human(?) experts left behind to monitor the AI. On the other hand, there might be more need for humans to do strategy and product development.

This vision also points to new infrastructure and business models:

  • Supporting infrastructure would include vendors building the base AI systems, which are too complicated for most companies to build for themselves (although non-technical users might use no/low-code tools to tune them); data tools to prepare, integrate, and expose data from all sources in real time; and analytical tools to measure interaction value.

  • Surrounding infrastructure would include new media that provide real-time access to their customers; low-friction integration methods to connect marketers with these systems; new billing and analytical methods; and social/legal frameworks to govern customer data collection, sharing and privacy.

  • New business models would be needed for companies and the systems, data, and media providers that support and surround them. With each interaction managed separately, providers might become on-demand services that charge for value provided in each case rather than billing based on labor, system use, or impressions.

I’m not sure this will happen. At most, I’d bet one of my later-born children on it. I’m laying it here out because the STIB model works best if you measure against a transformed state (and because it’s fun to think about.)

Mapping against the STIB Framework

That said, let’s try mapping recent product news against the STIB framework. 

Application

The application we’re analyzing is customer management, which is roughly what we cover in the CDP Institute daily newsletter. This gives us a reasonable collection of announcements to work with, although it’s just a small sample based on items that happened to appear during a relatively brief period.

If the STIB model is correct, we should find clusters of products on a spectrum from business as usual, to substitution of AI within current processes, to complete transformation. Because transformation doesn’t happen all at once, we would expect to find several intermediate stages. Indeed, that’s the case.

Current State: The first cluster would hold products that execute the existing process without any change. AI-powered co-pilots might fit here. But co-pilots are so common that we don’t bother to write about them in the newsletter. Nor do we usually cover products that aren’t doing anything new. So this cluster is empty apart from one item about agentic helpers.

Substitution: The second cluster is simple substitution: products that replace a discrete task with an AI-generated equivalent.  We see plenty of these, often offered as collections of (separate) AI tools for multiple tasks.  We also see toolkits for users to build their own.  So long as each AI tool executes one task separately from the others, this is still substitution.  Recent examples are:

Intermediate Products: Now we move into products that change the underlying process but don’t reach the fully transformed state.  The news items seem to fall into three clusters.  It’s important to note that we didn’t define these in advance: they have emerged from the data itself. 
  • Content Creation: This is a popular task to automate.  It’s a single task but more than substitution because most vendors connect their content generator to response data and use this to automatically optimize content over time.

  • Goal-Driven Workflow: this cluster holds systems that have automated development and execution of a multi-step workflow, such as audience segmentation, journey design, or media buying.  Like content creation, these usually let users specify a goal for the workflow and collect data to help measure results.  
  • Customer Management: these products collect data to support optimization, help users to create messages, and select and deliver messages across customer touchpoints.  They come the closest to the fully transformed state but don’t support all channels or create hyper-personalized messages in real time. 

  • Transformed Products: This cluster would hold products that deliver the fully transformed process.  There’s a good chance that some vendors have this mind, but we haven’t seen any products that deliver it.  So the cluster is empty.

Infrastructure

Infrastructure and Business Models won’t take their final shapes until the transformed process is fully deployed, but they do co-evolve with the application changes.  This applies especially to supporting infrastructure, parts of which must be in place for some intermediate product to function effectively.

Supporting infrastructures: The key supporting infrastructures for transformed customer management are inputs (data access and quality), internal processes (agent cooperation and analytics), and outputs (media integration).  We see announcements in all these areas.  

  • Data Access: most data access announcements describe accessing data in real time without loading it into a separate database.  So far, these developments focus primarily on reading data in the company’s internal systems.  But remember that the full vision for transformation includes access to third-party data such as compiled customer behaviors and local weather.  We do see some of that, although none is in the current sample.

  • Data Quality: these tools prepare data for AI use.  Many data quality vendors have added features to support AI use and have added AI-powered offerings.  These apply to customer data but we don’t usually write about them, so this cluster is fairly sparse.  
  • Agent Deployment: this describes technology for building agents and helping agents work together.  It’s another field with extensive activity that is largely beyond the scope of the daily newsletter.  It will be critically important if the transformed state involves teams of agents that cooperate closely with each other.

  • Analytics: any goal-seeking agent will need internal analytics to guide its decisions.  Still, there may turn out to be a market for independent agents that make their results available as a shared resource for teams of specialist agents.   This would apply especially to customer value analytics, which are needed to compare opportunities across different channels.

  • Media Integration: this is the output infrastructure that deliver messages created by the hyper-personalization.  The tools are steadily encompassing more channels. 

Surrounding Infrastructures: changes to the surrounding infrastructure happen after the transformed application is in place.  This makes them harder to predict than supporting infrastructure, which develops sooner.  We do have some current developments that are likely to become more important as hyper-personalization matures.  There will no doubt be others.

  • Integrated Commerce (Sales Agents): this describes shopping in non-traditional channels, including retail media, search engines, social media, video, connected TV, and mobile apps.  These can work without hyper-personalization but are vastly more effective when offers are tailored to the individual and context.  In many cases, the interface will be a chat-style, AI-based sales agent that has a real-time dialog with the customer.

  • AI-Based Search (Buyer Agents): this describes marketing that is targeted at AI agents rather than humans.  Today's most common implementation is AI search overviews, which do research for buyers.  This changes the goal of search marketing from attracting traffic to appearing in genAI summaries. Other agents are evolving for other types of research and other stages in the sales cycle including making purchases on the customer's behalf.
Interacting with these buyer agents at scale and cost-effectively will require AI-based sales agents.   In some ways, it won’t matter whether the sales agent is interacting with a person or a bot: to the sales agent, both are collections of data that must be analyzed and responded to appropriately.  That said, the behaviors of humans and bots will be significantly different, so the sales bots will no doubt develop separate approaches to each.  Ultimately, marketing systems may give buyer agents direct access to (some of) the product and promotional information they provide to sales agents, bypassing the sales agents altogether.
 
Because this is a relatively new development, our news coverage includes more research reports than product announcements.

Business Models

Few companies make announcements about changes in their business model, especially when those changes involve firing people.  As a result, we have few newsletter items on AI-based business models.  One change we do see is movement towards pricing based on value created rather than resources consumed.   The transformed process will surely create other opportunities, perhaps including “no staff” companies run almost entirely by AI or “no product” companies that source products in real time as they find interested customers. It's likely the new business models will include ones we haven't even imagined.

What We’ve Learned

The main purpose of this blog post is to determine whether the STIB model provides a useful way to think about technology innovations.  Of course I’m biased, but I believe it does. Not only was it reasonably easy to slot products into different categories, but the process generated several helpful insights:

  • Products that substitute AI for one step within an existing workflow are significantly different from products change the workflow.

  • It helps to envision the final, fully transformed product so we can assess intermediate products, identify the supporting capabilities and infrastructures it requires, and predict the surrounding infrastructure and business models it is likely to generate. 
  • We should acknowledge that our prediction could be wrong.  It may help to build scenarios around alternative outcomes.  

  • Intermediate products appear before the final transformation. These become possible as new capabilities, infrastructures and business models appear. Observing the intermediate products helps to assess whether the changes are moving in the direction we expect or to adjust our prediction of the transformed state..

  • A minimum set of capabilities, infrastructures, and business models must be in place before the final design can succeed. Companies will fail if they offer the transformed design before supporting infrastructures are in place.

  • Capabilities, infrastructure, and business models continue to evolve after the (successful) transformed design is introduced. These developments make the product more effective and exploit the opportunities it creates.

  • Continued evolution creates rapid growth and expands the value of the final design, giving it an increasing advantage over alternatives. This advantage ultimately locks other configurations out of the market, even though some may be technically superior.

Applying the STIB model to customer management offers additional insights. Our sample of product news is enough to show:

  • Many current products do simple substitution. These are the easiest to deploy and can show immediate improvements in cost and quality. (There are also many products that support the existing workflow without any changes, but those don’t show up in our sample.)

  • Some intermediate products are already available. Most of these automate a single stream of tasks within the customer management workflow, such as content creation, campaign design, media buying or analytics. 
  • Intermediate products usually work within a single department. This makes them easier to drop into the larger workflow and reduces the number of users whose work is disrupted.

  • Some products aim to automate an entire workflow, such as campaign design, development, and execution. Doing this with autonomous AI agents is the leading edge of the industry today.
  • Some companies offer components that support the final vision.  These include application capabilities such as content optimization, response simulation, and advanced attribution, as well as infrastructure and business model changes including agent coordination, cross-company data sharing, touchpoint integration, and performance-based pricing.

  • I haven’t seen any products that promise the predicted final state of “real-time, hyper-personalized, omni-channel messages.”  This has surely occurred to many smart people, so I’ll guess they have decided (correctly) that the capabilities, infrastructure and business models aren’t ready yet.

  • The final state may be delivered by a single integrated product or by multiple agents working in concert. Remember that the final state requires close cooperation between advertisers and media companies and between different departments within the same company.  This makes a multi-agent solution more likely and reinforces the needed for data- and process-sharing technology..

Implications

Here are some practical implications of what we’ve discussed.

  • Substitution products can be valuable, but the benefit won’t last. It’s tempting to argue that substitution is a poor investment because it offers only incremental improvements on today’s current processes and that companies, and software developers should instead focus on more profound solutions. But the plain fact is that substitution creates substantial benefits and is easier to deploy than changes that require process change. Product developers and business users shouldn’t shy away from substitution but they do need to recognize it has a relatively short shelf life. Product developers should also realize that nearly all incumbent vendors will add substitutions to their current products or have already done this.  That makes it hard to convince users to change to a new system on the basis of substitution alone. Attracting new clients will require solutions with more substantial advantages, which is what the intermediate product designs can offer.

  • The goal of "real-time, hyper-personalized, omni-channel messaging" is not yet widely discussed. What leads me to expect this as the transformed state is the growth of direct sales in social media, search results, connected TV, podcasts, and pretty much every other channel. This "instant commerce" collapses the multi-step "awareness, interest, desire, act" cycle into a single moment when the customer is presented with an opportunity to buy. Taking full advantage of this moment requires marketers to connect with all message opportunities, to gather all data so they can assess the potential value of each opportunity, and to deliver the most effective possible message for each opportunity they purchase. Only AI can do this effectively at scale, most likely by presenting intelligent agents to interact with customers who engage.**

  • The transition can be gradual. Instant commerce can be deployed in one channel at a time, can work with limited data, and doesn’t require advanced message optimization. Companies and vendors can move towards the fully transformed state in stages, building experience, product, infrastructure, and business models along the way.

  • Instant commerce depends on long-term relationships. That may seem like a paradox but customers will only engage with companies that they trust. There’s no time to build trust during the interaction moment, so trust must be built in advance. The good news is that the importance of trust is widely recognized and the methods for building trust (among humans) are well understood, if not always well executed. The industry may need new lessons in building trust among AI agents.

  • Infrastructure may offer the greatest opportunities. The biggest gaps between what’s currently available and what’s needed in the transformed state seem to be deep connections to collect external data and interact with touchpoint systems. 

    • Shallow connections already exist, but real-time, hyper-personalized, omni-channel messaging implies real-time queries of external data sources about individual customers to collect up-to-moment information on behaviors. The touchpoints where those behaviors occur must capture, identify, assess, expose, and charge for that data in real time in a privacy-compliant fashion. Bear in mind I’m talking about touchpoints outside the company that wants to use this data. Very little technology exists to do that today. Data clean rooms are a start.

    • Beyond sharing information, those same touchpoints need to receive ad messages and deliver them to their visitors, again in real time and with feedback on response. Most of the messaging technology will reside with the ad buyer or middlemen, who will need to receive notification of contact opportunities, gather data and assess those opportunities, select the appropriate message, bid on delivering the message, transmit the message on bids they win, and measure results. Again, the existing technology to do all this in real time, at scale, and across many channels is limited at best. (Today’s programmatic ad system is a partial model.) In addition to data movement, this process requires sophisticated evaluation models so marketers can accurately bid on the projected value of each interaction.

    • Because all this interaction is happening between different companies, the infrastructure technology must be widely shared. This could imply broadly accepted standards implemented by many developers, or, more likely, proprietary technology built and sold by a few major suppliers. Competition to be one of those suppliers will be fierce but the rewards are likely to be huge. The rewards might diminish over time once the process is well enough understood to create open standards that are a viable, cheaper alternative.

  • Data quality doesn’t get the attention it deserves. Survey after survey shows that data issues are the top roadblocks to marketing, personalization, and AI success. Yet companies rarely make data quality an investment priority. There isn’t much to say about this except that real-time, hyper-personalized, omni-channel messages make data quality more important than ever. This is yet another piece of infrastructure that’s ripe for improvement.

  • Buyer bots might change everything. Buyers have limited attention, which is why gaining attention has always been the first step in successful marketing. This still applies to real-time, hyper-personalized, omni-channel messages -- so long as they're being sent to humans. But if customers delegate their purchasing activities to bots, the fundamental truth is no longer true: buyer attention will no longer be limited.  Developments such as search engine optimization and AI search overviews as early examples of marketing to bots: marketers must target the algorithms, not attract human attention. But search marketing of any kind is still aimed at putting messages in front of eyeballs. Truly automated purchasing will remove humans from the entire process. The change won't happen overnight but it seems plausible to expect a mix of human and bot buyers in the near future. A STIB model with bot buyers as the final transformed state would be quite different from the one I’ve presented here.

Summary

This post has explored whether current developments in customer management technology can be effectively analyzed with the STIB model of innovation diffusion and whether the results provide useful insights into industry trends. I believe the answers are Yes and Yes.  I've also presented a specific vision for the industry future, of "instant commerce" delivered through real-time, hyper-personalized, omni-channel messages.  I can't promise that this is correct, but it's an interesting starting point for discussion.

_________________________________________________________________________________ 

* The graphic is conceptual but could be quantified by counting the number of components in each product that match the final transformed design or that differ from the initial state.  Fun!

**  Things are admittedly a bit more complicated for non-impulse purchases.  But I’d argue you’re still trying to motivate an action in the moment, even if it’s only saving an offer in a wallet for future consideration.

Wednesday, May 01, 2024

CDP Success Depends on More than Marketers

The CDP Institute runs periodic roundtables where our members discuss industry issues.  We had one earlier this week on the topic of helping companies make use of an existing CDP.  We chose that topic because we frequently hear that many users don’t know what to do with a CDP after it’s built.

The initial discussion followed the more or less expected path towards solutions such as developing change champions and publicizing success stories to encourage CDP use.  Participants stressed the need for organizational change to allow applications that the CDP makes possible.  This change includes cooperation between teams that had previously not worked together, new processes, and new metrics for cross-channel marketing programs.  We heard that traditional direct marketing organizations are especially likely to struggle precisely because they have such mature single-channel techniques in place.

On the bright side, the group said that real-time capabilities were the most likely to create new opportunities that would make users excited enough to adopt new approaches.  Good to know.

The discussion then turned in an unexpected direction: the shift in CDP leadership from marketers to IT and data teams.  One participant said in the past two years, he had found that 90% of new projects were initiated by IT.  While that’s certainly a trend I’ve observed, the group added several new insights.

  • One reason for the shift is that CDP deployment requires more technical skill than most marketing teams can muster.  I have generally argued the reason for greater IT and data team involvement is the broader use of customer data beyond marketing departments.  This makes the CDP an enterprise-level project.  But it’s probably also true that CDPs used more widely throughout the organization are more technically complicated than CDPs used only in marketing.  Broader applications imply more data sources, more types of processing, and connections to more applications.  This all requires greater involvement by the company’s IT and data teams.

  • IT and data teams need new skills to succeed with a CDP.  This goes beyond the need to understand specific processes such as identity resolution, which many IT and data teams had not previously needed to support.  It includes the need for those teams to better understand how marketers actually do their jobs.  Without that fundamental understanding, IT and data teams will make poor choices in designing systems to support marketers.  Traditional project-by-project requirements gathering isn't enough.

  • More specifically, IT and data teams need to recognize that marketers have a different attitude towards data.  One participant observed (and I agree) that IT teams want to make sure everything is correct before they release any data, while marketers are willing to accept some errors in return for quicker results.  It’s an astute observation about culture that both sides should recognize and discuss so they can agree on the right balance for any particular application. 

  • A bigger role for IT and data implies a smaller role for marketing technologists.  They’re still involved in selecting martech, of course, but are less likely to lead a CDP project than they once were.  This probably applies to other marketing technology projects as well, especially if it applies to departments outside of marketing.  This isn't a death knell for martech staff.  But it does imply a shift in responsibilities, away from designing the martech stack to system administration, support, and analytics.  

Those final two points – about the different attitudes towards data quality and the changing role for martech staff – both point to a need for more CDP education aimed at IT and data teams, rather than just marketers.  That’s something we’ll consider here at CDP Institute and I suspect will be important for industry vendors, consultants, and practitioners as well.  

Friday, April 26, 2024

CDP Overview: How We Got Here, Where We're Going, and What Could Get in the Way

Has anyone asked you recently about the past, present, and future of Customer Data Platforms?  No?  That's odd; people ask me those questions

all the time.  Here are my current answers. 

It’s eleven years since the term “customer data platform” appeared in 2013.  What stages has it gone through over that time?
 

The first stage was just to recognize that CDP was a separate category of software.  What was new about CDP was that it was packaged software that was building a customer database.  Before then, customer databases were custom projects, like a data warehouse, and the only packaged marketing software was applications like campaign management systems or predictive modeling tools.  The earliest CDPs actually bundled the database building capability with an application.  But, fairly soon, vendors realized that there was more value in the database building features, which were rare, than in the applications themselves, which were fairly common.

Once we had identified the CDP category, the next stage was convincing people that it was important.  The concept itself was easy to grasp (“all customer data in one place”).  But there was initial skepticism about whether it was a legitimately separate category or just another name for existing technologies such as CRM, data warehouses or DMPs.  So we spent most of our time explaining the difference between CDP and other systems that also built customer databases.  In fact, the formal CDP Institute definition – "packaged software that builds a persistent, unified customer database accessible to other systems" – is carefully crafted so that each term identifies a differentiator between CDP and some other type of system.  “Packaged software”, for example, distinguishes CDP from data warehouses and data lakes, which are custom built.  I won’t go through the other elements point-by-point.

Once the concept was reasonably well defined, we faced skepticism about need for separate CDP database, compared to just reading existing source data in real time to build profiles.  That’s what ”persistent” refers to in the CDP definition.  It took the big martech vendors including Salesforce and Adobe several years to accept that.  The reason is  that building profiles on demand by assembling data in real time just takes too long.

Even after category was established, there was great confusion about what really qualified as a CDP.  That’s why the CDP Institute launched its RealCDP certification program, which expanded on our definition by defining five key requirements: assemble all data types, keep all details, retain the data indefinitely (subject to regulatory constraints), build unified profiles, and share the data with other systems.  We later added a sixth point relating to two real time capabilities: access to individual customer profiles, for things like call centers and website personalization, and real time event triggers, for things like responding to dropped shopping carts.  Of course, we are just one voice among many, and are easily drowned out by vendor promotions. So, unfortunately, some of that confusion persists to this day.

Part of the reason for the continuing confusion is a debate over whether CDP just builds profiles or also should include data activation capabilities such as analytics and personalization.  Our view is they’re not essential, but over time, we’ve seen the industry split between CDPs that only build profiles and CDPs that include activation functions.  The majority of CDP vendors provide activation, so that’s clearly what most buyers want.

More recently, we’ve seen interest in using customer data beyond marketing, which makes IT and data teams more interested in CDP projects.  That’s a big change because when CDP was just a marketing tool, IT teams were very happy to let marketers buy the CDP becaise it kept the marketers happy without consuming IT resources.  Now, CDP is too important to be left to marketers.  This also makes the CDPs that just build profiles more appealing in some situations – and we have in fact seen slightly faster growth in that group most recently.

IT involvement in turn brings more interest in companies building their own CDP, and leveraging their existing data lakes and warehouses as the foundation.  This has been called ‘composable CDP’ although it’s not really the right use of the term ‘composable’.  Most people now agree that 'warehouse-based' is a better label.  Semantics aside, the problem is that IT teams often underestimate the requirements for a proper CDP and, thus, the work involved in creating one.  So it’s important to ensure they do a thorough job of scoping the project in advance, so they make the best choice.

The other thing we’ve seen, more or less continuously, is expansion of CDP into new industries.  Originally they were used primarily in retail and media.  Then, they grew more common in financial services, hospitality, and telecom, which are all industries that traditionally had pretty good customer data systems.  Most recently, we’re seeing CDPs in education, healthcare, and government applications.
 

We’re also seeing CDP used more for advertising applications, as companies lean more heavily on first-party data to replace the loss of third-party cookies and replace other targeting methods that become harder as privacy rules become more stringent.  
 

In fact, CDP also supports other privacy-related applications, such as closer control over ensuring that contacts are authorized by consumer consent, and providing data clean rooms for privacy-safe data sharing.  Funneling all customer list creation through a CDP is one way to avoid breaking privacy rules.

Where do you see the industry headed next?  

The biggest issue right now is the movement towards ‘composability’.  We can’t control whether IT and data departments try to build their own CDP-equivalent.  But we can educate them about the actual requirements and we can give them tools that make it easier to meet those requirements.  Many CDP vendors are now breaking apart their systems to provide modules that companies can use as components in building an internal system.  Of course, that helps those vendors to survive a transition to a composable CDP world, but it also helps to maintain the reputation of the industry as a whole.  What we don’t want to see is composable CDP projects fail because that makes people question the value of the CDP concept itself.

Closely related to the composable trend is a trend for ‘no copy’ access to external data by a CDP.  This means the CDP can read data from other systems without copying it into the CDP data store.  That used to be more or less impossible at scale, because finding, reading, and integrating masses of external data took too long.  With today’s technologies, that process can be much faster, so it becomes a more practical alternative.  Some common use cases are reading things that change quickly and are only relevant at the moment you need them, like inventory levels or local weather conditions.  Of course, this ability also blurs the distinction between a traditional CDP, which imported all its data, and a warehouse-based CDP, which works with data stored externally.  That’s okay but it does add still more confusion to the discussion.  

I’ll also make a side note that the original CDP skeptics argued for reading source system data on demand, so it may seem this proves they were right.  But so far I don’t think you can have a CDP that only works by reading external data on demand, because some processes like identity resolution and data aggregation still take too long to do purely on the fly.  So I see the future CDP as having a core of data that it does copy and store in its own database, for things like the identity graph that tells how to combine data from different sources into each customer’s profile.  Whether that database is a CDP or data warehouse doesn't matter: either way, the data will have been copied from the original source system and preprocessed for use by the CDP.  The core data could well be supplemented with data read on-demand from external systems, which could be the original source or a data warehouse or data lake.  At least in theory, this would give you the best of both worlds: minimal data copying but maximum performance.

Putting aside composability, CDPs will need to adjust to new privacy rules by adding features like encryption, advanced privacy policy management, and data clean rooms.  They’ll adjust to new media and new data types, like audio and video, which need to be not just stored but analyzed in ways that are currently close to impossible.  And, of course, they’ll adjust to growing AI capabilities, which will make it easier to perform some functions like adding new data sources, matching identifiers that belong to the same person, building predictive models, and analyzing business results.  AI will also add to the volume and complexity of data that CDPs have to handle, which itself may require new technology to support.  For example, if AI starts to create a huge variety of content personalized for each individual, that makes result analysis vastly more complicated.  Somehow the CDP will need to deal with that.

On a more prosaic level, I expect to see more CDPs that are tailored to the needs of particular industries.  That’s a typical development for a mature software category.  Specialist systems can be more cost-effective to build and deploy, because they use special data structures and features tailored to a particular industry’s needs.  This will bring down the cost of CDPs, which has been an obstacle to expanding the base of CDP purchasers.  

And, as I’ve already mentioned, I expect to see CDPs used more widely beyond marketing.  One thing we’ve learned in recent years is that customers expect a personalized experience every time they interact with a company, whether it’s before they buy a product or after they start using it.  That requires making customer data available at every interaction point.  Beyond direct customer interactions, teams like product development and operations planning also can benefit from using customer data.

Are there any particular obstacles or threats to the CDP’s continued success?

Certainly CDPs will have to adapt to the changes we’ve already discussed in data types and volumes, in the users for customer data, and in whatever technical developments change the best way for CDPs to be built.  Individual vendors may struggle to keep up.  But I think the need for complete, sharable customer profiles is here to stay.  So to the extent that that’s the core definition of a CDP, the future of the industry is secure.

That said, I do see two major threats to the CDP industry as we know it today.  

The first is technical: the cloud databases from specialists like Snowflake and Databricks and general cloud platforms like Google Cloud, AWS, and Microsoft Azure.  All those vendors are increasingly eyeing the giant pile of customer data held in a CDP and wanting it for themselves.  To get it, they’re adding applications to build profiles, such as data quality and identity resolution, and to do analytics and marketing.  Sometimes they make the applications to be features to their own database systems; sometimes they build direct integrations with specialist applications through a marketplace of some sort.  Either way, this cuts out the CDP, which has traditionally been an intermediary between enterprise data stores and business applications.  Again, this comes back to the notion of the warehouse as the primary customer data store, which makes the CDP database unnecessary.  I think threat is limited today because most IT departments don’t have the resources to build those systems for themselves. But as new tools make it easier to build those systems, the threat becomes more important.  There are really just two ways for CDPs to respond to this threat:

  • one is to become platforms themselves, which is to say, to replace the data warehouse itself.  That’s not as crazy as it sounds, since the CDP is really a set of tools that builds the customer database, so it can work on top of a Snowflake or Google BigQuery or whatever database the client wants.  In this world, the CDP evolves from a tool to build customer profiles to a tool to build data warehouses in general.  Difficult but not impossible, at least for the largest CDP vendors who have the resources to compete.
  • The other option is to become applications on top of the warehouse, offering specialized capabilities like cross-channel journey orchestration.  The would build on CDPs’ existing capabilities to share their profiles with activation systems, and to themselves do activation functions that apply across all channels, such as predictive models and best offer selection.  It’s actually an easier path for most CDP vendors, since it continues their role as intermediaries between data sources and business applications.  But it’s also a fairly narrow role and there will be lots of competition.  Specialization by industry, company size, and other variables will be the key to avoiding that competition by becoming the best choice in a particular niche.

The second threat isn’t technical, but the ability of organizations to actually make use of a CDP once it’s built.  We run into this all the time, when companies tell us their staff doesn’t know what to do with a CDP or doesn’t have the skills to do what they’d like.  It’s a problem that will only get worse as customer data grows more complicated and there are more possibilities for using customer data.  Maybe AI will solve the problem, and that’s something CDP vendors need to invest in to make happen.  But there’s no guarantee that AI will be the answer, and my guess is that even AI will need skilled users to take advantage of the possibilities it creates.  

Of course, if the CDP turns out not to be useful, the staff members will blame the CDP, not their own lack of skill.  But it doesn’t really matter who takes the blame: if CDPs don’t create value, companies won’t be willing to invest in them.  So it’s really important for the CDP industry to help train CDP users, not just in the technical details of how to use a CDP, but in the business programs that a CDP can support. 

Thursday, September 14, 2023

Unleashing the Power of Customer Data Platforms (CDPs) and AI: A Game-Changer for Modern Marketing

For some unknown reason, my last three presentations all started as headlines (two created by someone else) which I then then wrote a speech to match. This isn’t my usual way of working. It does add a little suspense to the writing process – can I develop an argument to match the title? It also dawns on me that this is the way generative AI works: start with a prompt and create a supporting text. That’s an unsettling thought: are humans now imitating AI instead of the other way around? Or have I already been replaced by an AI but just don’t realize it? How would I even know?

The latest in this series of prompt-generated presentations started when I noticed that the title shown in a conference agenda didn’t match the speech I had  prepared. When I pointed this out, the conference organizers said I could give any speech I want, but the problem was, I really liked their title: “Unleashing the Power of Customer Data Platforms (CDPs) and AI: A Game-Changer for Modern Marketing”. 

The idea of “unleashing” a CDP to run wild and exercise all its powers, is not something I get to talk about very often, since most of my presentations are about practical topics like defining a CDP, selecting one, or deploying one. And I love how “and AI” is casually tucked into the title like an afterthought: “there’s this little thing called AI, perhaps you’ve heard of it?”

And how about “game changer for modern marketing”? That’s an amazing promise to make: not just will you learn the true meaning of modern marketing, but you'll find how to change its very nature so it’s a game you can win. Who wouldn’t want to learn that?

This was definitely a speech I wanted to hear. The only problem was, the only way for that to happen was for me to write it. So I did. Here’s a lightly modified version.

Let’s start with the goal: winning the modern marketing game. The object of the game is quite simple: deliver the optimal experience for each customer across all interactions. And, when I say all interactions, I mean all interactions: not just marketing interactions, but every interaction from initial advertising impressions through product purchase, use, service, and disposal. I also mean every touch point, from the Internet and email through call centers, repair services, and the product itself.

That’s a broad definition, and you will immediately see the first challenge: all the departments outside of marketing may not want to let marketing take charge of their customer interactions. Nor is your company’s senior management necessarily interested in giving marketing so much authority. So marketing’s role in many interactions may be more of an advisor. The best you can hope for is that marketing is given a seat at the table when those departments set their policies and set up their systems. This requires a cooperative rather than a controlling attitude among marketers.

The second challenge to winning at marketing is the fragmented nature of data and systems. Most marketing departments have a dozen or more systems with customer data; at global organizations, the number can reach into the hundreds.  Expanding the scope to include non-marketing systems that interact with customers adds still more sources such as contact centers and support websites. Again, marketing will rarely control these. At best, they may have an option to insert marketing recommendations directly into the customer experience, such as suggesting next best actions to call center agents.

The third challenge is optimization itself. It’s not always clear what action will result in the best long-term results. A proper answer requires capturing data on how customers are treated and how they later behaved as a result. Some of this will come from non-marketing systems, such as call center records of actions taken and accounting system records of purchases. Again, those systems’ owners may not be eager to share their data, although it’s harder for them to argue against sharing historical information than against sharing control over actual customer interactions.

But the challenge of optimization extends beyond data access. Really understanding the drivers of customer behavior requires deep analysis and no small amount of human insight. Some questions can be answered through formal experiments with test and control groups. But the most important questions often can’t be defined in such narrow terms. Even defining the options to test, such as new offers or marketing messages, takes creative thought beyond what analysis alone can reveal.

And even if you could find the optimal treatment in each situation, the playing field itself keeps shifting. New products, offer structures, and channels change what treatments are available. New systems change the data that be captured for analysis. New tools change the costs of actions such as creating customer-specific content. These all change the optimization equation: actions that once required expensive human labor can now be done cheaply with automation; fluctuating product costs and prices change the value of different actions; evolving customer attitudes towards privacy and service change the appeal of different offers. The optimal customer experience is a moving target, if not an entirely mythical one.

None of this is news, or really even new: marketing has always been hard. The question is how “unleashing” the power of CDPs and AI makes a difference.

Let’s start with a framework. If you think of modern marketing as a game, then the players have three  types of equipment: data systems to collect and organize customer information; decision systems to select customer experiences; and delivery systems to execute those experiences. It’s quite clear that the CDP maps into the data layer and AI maps into the decision layer. This raises the question of what maps into the delivery layer. We’ll return to that later.

First, we have to answer the question: Why would CDP and AI be game changers? To understand that, you have to imagine, or remember, life before CDP and AI. Probably the best word for that is chaos. There are dozens – often hundreds -- of data sources on the data layer, and dozens more systems making choices on the decision layer. The reason there are so many decision systems is that each channel usually makes its own choices. Even when channels share centralized decision systems, those are often specialized services that deal with one problem, whether it’s product recommendations or churn predictions or audience segmentation.

CDP and AI promise to end the chaos by consolidating all those systems into one customer data source and one decision engine. Each of these would be a huge improvement by itself:

  • the CDP makes complete, consistent customer data available to all decision systems, and
  • AI enables a single decision engine to coordinate and optimize custom experiences for each individual.

Yes, we’ve had journey orchestration engines and experience orchestration engines available for quite some time, but those work at the segment level. What’s unique about AI is not simply that it can power unified decisions, but that each decision can each be tailored to the unique individual. 

But we’re talking about more than the advantages of CDP and AI by themselves. We’re talking about the combination, and what makes that a game-changer. The answer is you get a huge leap in what’s possible when that personalized AI decisioning is connected to a unified source of all customer information. 

Remember, AI is only as good as the data you feed it. You won’t get anything near the full value of AI if it’s struggling with partial information, or if different bits of information are made available to different AI functions. Connecting the AI to the CDP solves that problem: once any new bit of information is loaded into the CDP, it’s immediately available to every AI service. This means the AI is always working with complete and up-to-date data, and it’s vastly easier to add new sources of customer data because they only have to be integrated once, into the CDP, to become available everywhere.

To put it in more concrete terms, one unified decision system can coordinate and optimize individual-specific customer experiences across all touch points based on one unified set of customer data. 

That is indeed a game-changer for modern marketing, and it only reaches its full potential if you “unleash” the CDP to consolidate all your customer data, and “unleash” the AI to make all of your experience decisions.

That’s a lot of unleashing. I hope you find it exciting but you should also be a little bit scared. The question to ask is: How can I take advantage of this ‘game changing potential’ without risking everything on technology that is relatively new and, in the case of AI, largely unproven. Here’s what I would suggest:

Let’s start with CDP. So far, I’ve been using the term without defining it. I hope you know that a CDP creates unified customer profiles that can be shared with any other system. No need to get into the technical details here. What’s important is that the CDP collects data from all your source systems, combines it to build complete customer profiles, and makes them available for any and every purpose.

Some people reading this will already have a CDP in place but most probably do not. So my first bit of advice is: Get one. It’s not such easy advice to follow: a CDP is a big project and there are dozens of vendors to choose from, so you have to work carefully to find a system that fits your needs and then you have to convince the rest of your organization that it’s worth the investment. I won't go into how to do that right now. But probably the most important pro tips are: base your selection on requirements that are directly tied to your business needs, and ensure you keep all stakeholders engaged throughout the entire selection process. If you do those two things, you can be pretty sure you’ll buy a CDP that’s useful and actually used.

That said, there are some specific requirements you’ll want to consider that tie directly to ensuring your CDP will support your AI system. One is make sure you buy a system that can handle all data types, retain all details, and handle any data volume. This does NOT mean you should load in every scrap of customer-related data that you can find. That would be a huge waste. You’ll want to start with a core set of customer data that you clearly need, which will be data that your decision and delivery systems are already working with. This lets the CDP become their primary data source. Beyond that, load additional data as the demand arises and when you’re confident the value created is worth the added cost.

The second requirement is to ensure your CDP can support real time access to its data. That’s another complicated topic because there are different kinds of real time processes. What you want as a minimum is the ability to read a single customer profile in real time, for example to support a personalization request. And you want your CDP to be able to respond in real time to events such as a dropped shopping cart. Your AI system will need both of those capabilities to make the best customer experience decisions. What’s not included in that list is updating the customer profiles in real time as new data arrives, or rebuilding the identity graph that connects data from different sources to the same customer. Some CDPs can do those things in real time but most cannot. Only some applications really need them.

The third requirement relates to identity management. The CDP needs to know which identifiers, such as email, telephone, device ID, and postal address, refer to the same customer. At CDP Institute, we don’t feel the CDP itself needs to find the matches between those identifiers. That’s because there’s lots of good outside software to do that. We do feel the CDP needs to be able to maintain a current list of matches, or identity graph, as matches are added or removed over time. That’s what lets the CDP combine data from different sources into unified profiles.

My second cluster of advice relates to AI. I’d be surprised if anyone reading this hasn’t at least tested ChatGPT, Bing Chat Search, or something similar. At the same time, there’s quite a bit of research showing that relatively few companies have moved beyond the testing stage to put the advanced AI tools into production.

And that’s really okay, so my first piece of advice is: don’t be hasty.  You should certainly be testing and probably deploying some initial applications, but don’t feel you must plow full speed ahead or you’ll fall behind. Most of your competitors are moving slowly as well. 

That said, you do need to train your people in using AI. They don’t necessarily need to become expert prompt writers, since the systems will keep getting smarter so specific prompting skills will become obsolete quickly. But they do need to build a basic understanding of what the systems can and can’t do and what it’s like to work with them. That will change less quickly than things like a user interface. The more familiar your associates become with AI, the less likely they are to ask it to do something it doesn’t do well or that creates problems for your company.

Third, pay close attention to AI’s ability to ingest and use your company’s own data. Remember the game-changing marketing application for AI is to create an optimal experience for each individual. The means it must be able to access that individual’s data. This is an ability that was barely available in tools like ChatGPT six months ago, but has now become increasingly common. Still, you can bet there will be huge differences in how different products handle this sort of data loading. Some will be designed with customer experience optimization in mind and some won’t. So be sure to look closely at the capabilities of the systems you consider.

Fourth, and closely related, the industry faces a huge backlog of unresolved issues relating to privacy, intellectual property ownership, security, bias, and accuracy. Again, these are all evolving with phenomenal speed, so it’s hard for anyone to keep up – even including the AI specialists themselves. Unless that’s your full time job, I suggest that you keep a general eye on those developments so you’re aware of issues that might come with any particular application you’re considering. Then, when you do begin to explore an application, you’ll know to bring in the experts to learn the current state of play.

Similarly, keep an eye on the new capabilities that AI systems are adding. This is also evolving at incredible speed. Some of those capabilities may change your opinion of what the systems can do well or poorly. Some will be directly relevant to your needs and may form the basis for powerful new applications. We’re still far away from the “one AI to rule them all” that will be the ultimate game-changer for marketing. But it’s coming, so be on the alert.

This brings us back to the third level of marketing technology: delivery. Will there be yet one more game-changer, a unified delivery system that offers the same simplification advantages as unified data and decision layers? The giant suite vendors like Salesforce and Adobe certainly hope so, as do the unified messaging platforms like Braze and Twilio. The fact that we can list those vendors offers something of an answer: some companies think a unified delivery layer is possible and would argue they already provide one. I’m not so confident because new channels keep popping up and it’s nearly impossible for any one vendor to support them all immediately.

What seems more likely is a hybrid approach where most companies have a core delivery platform that handles basic channels like email, websites, and advertising and supports third-party plug-ins to add channels or features they do not. These platforms are already common, so this is less a prediction of the future than an observation of the present, which seems likely to continue. The core delivery platform offers a single connection to the decision layer run by the AI. This gives the primary benefits gained from a single connection between layers, although I wouldn’t call it a game-changer only because it already exists.

My recommendation here is to adapt the delivery platform approach, seeking a platform that is as open as possible to plug-ins so it can coordinate experiences across as many channels as possible. In this view, the delivery platform is really an orchestration system. Which channels are actually delivered in the delivery platform and which are delivered by third-party plug-ins is relatively unimportant from an architectural point of view.  Of course, vendors, marketers, and tech staff will all care a great deal about which tools your company chooses.

While we’re discussing architectural options, I should also mention that the big suites would argue that data, decision, and delivery layers should all be part of one unified product, reducing integration efforts to a minimum. That may well be appealing, but remember that most of the integrated suites were cobbled together from separate systems that were purchased by the vendors over time. Connecting all the bits can be almost as much work as connecting products from different vendors.

And, of course, relying on a single vendor for everything means accepting all parts of their suite – some of which may not be as good as tools you could purchase elsewhere. The good news is most suite vendors have connectors that enable users to use external systems instead of their own components for important functions. As always, you have to look in detail at the actual capabilities of each system before judging how well it can meet your needs.

So, where does this leave us?

We’ve seen that the object of the modern marketing game is to deliver the optimal experience for each customer. And we’ve seen that challenges to winning that game include organizational conflicts, fragmented data, and fragmented decision and delivery systems.

We’ve also seen that the combination of customer data platforms and AI systems can indeed be game-changing, because CDPs create the unified data that AI systems need to deliver experiences that are optimized with a previously-impossible level of accuracy.

CDPs and AI won’t fix everything. Organization conflicts will still remain, since other departments can’t be expected to turn over all responsibility to marketing. And fragmentation will probably remain a feature of the delivery layer, simply because new opportunities appear too quickly for a single delivery system to provide everything.

In short, the game may change but it never ends. Build strong systems that can meet current requirements and can adapt easily to new ones. But never forget that change is inherently unpredictable, so even the most carefully crafted systems may prove inadequate or unexpectedly become obsolete. Adapting to those situations is where you’ll benefit from investment in the most important resource of all: people who will work to solve whatever problems come up in the interest of your company and its customers.

And remember that no matter how much the game changes, the goal is always the same: the best experience for each customer. Make sure everything you do is has that goal in mind, and success will follow.

Thursday, December 30, 2021

Game of Thrones Meets Big Bang Theory: Welcome to CDP Industry's Next Phase

The CDP Institute just published its latest Industry Update, our semi-annual overview of CDP vendors with data on employment, funding, locations, and more. (Download here.)  There were three pieces of information that stood out:

  • Only four new vendors were added, compared with an average of fifteen in past reports.

  • four companies reported funding rounds over $100 million, compared with one round that size across all past reports

  • nearly all employment growth (85%) came from previously listed vendors, compared with just 36% in past reports


Of course, it makes sense that most growth would come from existing vendors if we added few new ones. But industry growth over-all was in line with past trends, and actually a bit stronger: up 12% over the past six months. This meant that the growth rate of existing vendors was high enough to compensate for the “loss” of new vendors. In fact, the existing vendor growth of 11% was the highest since 2018, when the industry was just taking off.

Connecting these dots reveals a clear picture: an industry that has stopped attracting new entrants but is now growing strongly on its own – with leading vendors stockpiling funds to compete against each other an elimination round where only a few can emerge as winners. Think Survivor meets Game of Thrones with a dash of Big Bang Theory.

It’s a picture that makes a lot of sense. Customer Data Platforms are now widely accepted as an essential component of a modern data architecture, so it’s a market worth fighting for. But the situation facing potential entrants is daunting:

  • the leading independent CDP vendors now have mature products, big customer bases, high brand recognition, and lots of funding. 

  • enterprise software companies, including Salesforce, Adobe, Oracle, Microsoft, and SAP, are chipping away at the market by selling CDPs as part of their packages. 

  • marketing automation, customer support, ecommerce, and other vendors increasingly offer CDP modules baked into their own systems

  • IT teams show growing interest in building their own CDP equivalent, supported by a growing array of components that make the job easier.

Some mid-tier CDP vendors have already given up the fight and been acquired, most often by firms needing a CDP to anchor a multi-channel customer experience suite. The acquisition wave may have peaked, since there were just three acquisitions in the latest report, compared with a dozen over the previous two. 

Among the remaining firms, some may compete successfully as generalists.  But the more promising path in most cases will be to offer specialized products that can be the best in a particular niche. Those niches might be defined by a particular industry, region, company size, or CDP function.

The functional niches are most intriguing because they serve the growing market for CDP components. We’ve seen some movement in that direction, as vendors offer parts of their CDP as stand-alone modules for identity resolution, data collection, data distribution (“reverse ETL”), and campaign management. Those vendors see their modules as a point of entry into clients who will later buy more of their products. They may be right, but I wonder how many companies that buy best-of-breed components will reverse course by favoring components from a single source. What’s certain is that this approach exposes the CDP vendors to competition from point-solution specialists in each area while discarding the advantage of that comes from purchasing a CDP with a full range of pre-integrated functions.  Here's a sampling of that competitive landscape:

I also suspect that companies like Snowflake, Amazon Web Services, and Google Cloud Services, which now position themselves as providing one piece of a “composable” solution, will eventually add features that match what the independent component providers now offer. Actually, that’s already happening, so I don’t get much credit for predicting it. It’s a dynamic we’ve seen repeatedly in other markets: primary vendors expand their features to secure their position with clients by adding more value (hooray!) and increasing the cost of switching (boo!).

Let’s be clear: both the added value and the switching costs are the result of integration cost. No matter how many promises are made about easy integration, the fact remains that any non-trivial connection between two systems takes skill to plan, deploy, and maintain. Integration is often the top-ranked vendor selection criterion in surveys, which some see as showing that problem is well understood. I draw the opposite conclusion: people list integration as a consideration because they know it’s poorly understood.  This forces them to invest time in trying to avoid integration problems and even sacrifice other benefits to achieve it. If integration were really easy, no one would worry about it.

Right now, someone reading this is saying, “Ah, but no-code changes everything”. I don’t think so. No-code works best when automating simple processes with a few users where flaws are acceptable. The more complex, widely-deployed, and mission-critical a process is, the more important it is to deploy professional-grade design and quality control. Prebuilt components doesn’t change this: configuring those components and connecting them to each other still takes great care and understanding.

This isn’t (just) a cranky-old-man digression. CDP functions rank high in complexity, scale, and risk, so they are poor candidates for no-code development. CDPs certainly can have no-code interfaces that empower business users to do things that might otherwise require a developer. But those interfaces will control carefully defined and constrained tasks, not create core functionality. Assembling CDP-equivalent systems from composable functions is a different matter, and, yes, that should become increasingly possible for people with the right integration skills. What I doubt is that selling modules with those functions will be good business for CDP vendors: they are likely to commoditize their products and ultimately to be pushed aside by platform developers who integrate key functions directly.

I'm not saying that CDP vendors who can’t raise several hundred million dollars are doomed.  I am saying that most will have to pick a niche to succeed. One promising option is building customer data profiles, especially for big enterprises. It’s a single function that incorporates enough separate components for CDP vendors to provide value by avoiding integration costs.

The other big niche, or set of niches, is integrated customer experience solutions. Our latest report already shows systems that campaign and delivery CDPs, our name for systems that do this, account for two-thirds of the industry vendor count and funding, and nearly three-quarters of employment. Their actual share may be greater still: immediately after completing the latest report, I happened to glance at G2 Crowd’s list of CDPs and found our reports doesn't include several large, fast-growing retail marketing automation or messaging specialists (Insider, Listrak, SALESmanago, Klaviyo, and Ometria) that offer what looks like CDP-grade multi-source profile building.

Whether those vendors are true CDPs depends on whether they make those profiles available to other systems. Either way, the point is that there’s a large and growing market for cross-channel retail marketing systems with unified customer profiles at their core. There are similar markets outside of retail, where we already see specialist campaign and delivery CDPs in hospitality, financial services, telecommunications, healthcare, education, and elsewhere.

The value of industry-specific systems is, once again, reduced integration cost. In this case, the key integration is with industry-specific operational systems such as airline reservations, core banking, phone billing, health records, and student management. Vendors in these niches compete primarily on the marketing functions they offer, which makes them more departmental than enterprise solutions and pushes them to add marketing features tailored to their particular industry. Systems like this are hard to dislodge once they’re deployed because they are populated with many complex, difficult-to-replicate campaigns, reports, predictive models, and content libraries. This stickiness is what enables many successful vendors to co-exist in each niche, and what makes it hard for non-specialists to enter.

The division of the CDP industry into enterprise-level data CDPs and industry-specific, department-level campaign and delivery CDPs is not a new trend. What is new is the maturity of the competitors within many niches, which will make it increasingly difficult for new entrants to succeed. What’s also new is that building CDP-style customer profiles is increasingly common, making it a standard feature rather than a product differentiator. This encourages vendors to position themselves as something other than a CDP, even though they need to show buyers that their CDP features are first-rate.

My final conclusion is this: the CDP industry will continue grow, and it will remain important for buyers to find the right CDP, even as the CDP itself slips from the spotlight.