Showing posts with label customer data management. Show all posts
Showing posts with label customer data management. Show all posts

Friday, November 14, 2025

Let's Debate CDP Functions, Not Definitions

What’s the definition of a CDP? It's a bad question because it diverts attention from what really matters: What capabilities do CDP users need? Still, buyers keep asking and sellers keep answering, typically in ways that promote their own interests. Looking for an unbiased perspective on the topic, I recently asked ChatGPT what answers it was finding. It came back with a reasonable cluster of responses and particularly interesting details on who was using each one (see below for the full response)*:

  • Unified customer database: 70–90% of analyst pages, trade articles and vendor docs 
  • Marketing activation / audience building platform: 60–85% of vendor docs, blogs and many press releases  
  • Real-time/streaming profile & interaction engine: 30–60% depending on whether the source is vendor marketing (more likely) or neutral analyst articles (less likely to require “real-time”). 
  • Privacy, governance & identity management layer: 15–35%, increasingly present in analyst pieces and vendor positioning  
  • Part of a larger ‘data cloud’ or enterprise data stack: 10–30%, especially in vendor/marketing copy from big cloud vendors

These are categories that ChatGPT identified without me defining them in advance.  So it's particularly interesting that there’s no mention of composability, warehouse-centricity, no-copy, hybrid, embedded, integrated, stand-alone, or other architectural details that have dominated recent industry discussions. In a way, this represents a failure by the CDP Institute to propagate our view that a CDP must build a separate database of its own. But a less parochial response is to be pleased that the main distinctions reflect system functions, which is where the focus belongs.

Of course, the variety of definitions is still problematic. While it’s usually safe to assume that a system labeled as a CDP will provide a unified customer database, it’s less certain that it will also offer marketing activation and downright dicey as to whether it will offer real-time streaming profiles and interactions. This means the label provides little useful guidance: imagine a can of soup labelled “contains tomatoes and maybe chicken and could also have mushrooms, rice or shrimp”. The only way to know what's inside would be to open it -- which defeats the purpose of a label.  

(And, yes, the problem is worse for CDP than other categories. When I ran the same prompt for the term "customer relationship management software," a single answer dominated: 71% defined CRM as “a software/system/platform to manage customer interactions and data.”  The next highest share was just 29% for “integrated suite (sales, marketing, service automation)”. It’s true that the dominant answer is exceptionally broad, but at least most people understand this and won’t expect anything more specific.)

So, although industry understanding has not been entirely destroyed by architectural debates, there is still enough disagreement on the scope of a CDP to limit the term’s utility. (If the CRM example is typical, it may be a natural progression for popular categories to expand their meaning over time. That would be an interesting hypothesis to explore if anyone out there is looking for a thesis topic.)

The industry could fight to restore a more specific CDP definition, but that’s probably a losing battle. It’s more likely productive to shift the discussion away from defining the term "CDP" to defining the functions required to manage customer data. 

Yes, I’m proposing that the solution to our problem is a checklist. Don’t roll your eyes: whole books  have been written on the topic. (Ok, maybe just one book.)  But in an industry that has long been driven more by theory than practical requirements, anything that gets buyers to focus on what they actually need is a win.  

Getting the industry to agree on a shared requirements checklist wouldn’t be easy, since every participant would want to add or remove items depending on whether their products supports them. Indeed, the very notion of a comprehensive requirements list favors broad, integrated products over narrow point solutions. But I’d still invest a few embers of hope in a project to forge a complete customer data framework. The potential benefits, for users and vendors alike, are well worth the effort.

Sunday, May 11, 2025

Predicting the Future of AI-Based Customer Management: Lessons from Machine Tool Motors

As someone who has spent his career tracking new technologies, I’ve long been fascinated by how those technologies spread over time. The standard model of technology diffusion, developed by Everett Rogers and based on New Deal studies of how farmers adopted new crops, offers the familiar segmentation of users into innovators, early adapters, early majority, late majority, and laggards. 

It’s probably best known in the tech world in its Crossing the Chasm incarnation, which adds the all-important chasm between the second and third stages.

But there’s another model I find more helpful.  This looks not at who adopts new technologies but at how they’re deployed over time. There's a consistent pattern:

  • Substitution (the new technology is deployed to exactly replace the old technology without changing the surrounding processes or workflows)
  • Transformation (the process is adapted to take advantage of the capabilities of the new technology) 
  • Infrastructure change (new infrastructure is developed to support the transformed process)
  • Business model change (a new business model emerges to take advantage of the transformed processes and infrastructure)

I haven’t found any single source that presents this model in quite these terms, which may mean I could name it after myself, although for now I'll just use STIB. Some close matches include Ruben Puentedura’s SAMR Model for educational technology (substitution, augmentation, modification, redefinition), Deloitte’s Three Horizons Model (process optimization, process flow and quality, new business models) and  Carlota Perez’s theory of technological revolutions (radical innovation, irruption or substitution, incremental innovation, maturity).  Probably the best match is William Abernathy and James Utterback's Dominant Design Framework (fluid phase, transitional phase, specific phase). 

The paradigmatic example, at least for me, is deployment of electric power in U.S. factories.  This started with connecting electric motors to the shaftworks that were previously driven by waterwheels or steam engines (substitution).  After several intermediate changes, the endpoint was attaching a separate motor directly to each machine tool (transformation). This enabled an infrastructure change: no longer constrained by the limits of mechanical power transfer or proximity to water or coal, factories became cleaner, larger, single story, and located near other resources or markets. The new infrastructure resulted in a new business model featuring centralized mass production and standardized national brands. 

Perhaps you can't relate to machine tools.  Fair enough.  Consider the automobile instead. The first cars were literally horseless carriages, pretty much the same as horse-drawn carriages except that they used a mechanical motor instead of a horse.  That was substitution.  Over time, the design of the cars was modified to take better advantage of having an internal combustion engine.  That was transformation.  Once it became clear the new model was going to be a success, manufacturers developed new, mass-production business models that better suited the new products and processes.  Ultimately, this was supported by a new infrastructures of roads, filling stations, repair shops, traffic rules, licensing requirements, insurance products, and ultimately an entire, auto-based suburban landscape. 

This may seem academic, but it applies directly to something you probably care about: the growth of AI.  Seen through the STIB framework, most of the AI applications we see today are substitutions: an AI copywriter replaces a human copywriter in an unchanged workflow. The omnipresent co-pilots are another, even less disruptive type of substitution: they help humans perform the same tasks more efficiently, again without changing the workflow.

But it’s clear that these substitutions don’t take full advantage of AI’s potential. Again referring to the STIB framework, the question is what the transformed applications will look like.  To answer this, we conduct a thought experiment: what would the process look like if it were redesigned to make best use the new technology? For electric motors in factories, the final form was motors attached directly to machine tools. For most customer data and customer-facing operations, the final form of AI is likely to be a single process that completes all the previously-separate steps at once. This is because AI is not limited by the need to have different specialists perform each step in the workflow, a constraint which results from the inability of mere humans to master more than one specialty, and from the need for experts to check the output of each step before moving on to the next.

But I don't think unified execution is the final form of the AI transformation.  A deeper change is likely to remove discrete units such as customer segments, content pieces, campaign flows, and maybe even standardized products. Those exist because humans could only manage small numbers of segments, messages, campaigns, and products. An AI could handle more-or-less infinite numbers of these, which in practice would mean treating each customer and each interaction individually.

This leads to an end-state of “hyper-personalized” messaging, where content is custom generated on the fly for each customer and context. 

Imagine an all-knowing, all-seeing bot that listens to what’s happening in the market and jumps into action each time it sees an opportunity to do something useful. The action will be optimized using all relevant data, including the company’s own information about the customer; second-party, third-party, and public information about the customer; behaviors of other customers; and market conditions, inventory, and who knows what else. In another dimension, this listening can extend beyond company-owned systems such as websites and contact centers, to include appearance of customers on third-party sites (already available to some degree through programmatic ad bidding) and even in walled gardens (which already receive lists of customers to watch for; the change would be to open a channel that lets the company assess the situation, generate the optimal message, and send it back for the walled garden for delivery.)*

It should immediately be clear that this vision requires infrastructure and business model changes from what’s available today. A much-improved data sharing infrastructure is needed to monitor behavior and access data outside of company-owned systems. This implies new business models to compensate external data owners for access to their information. Perhaps the data owners would charge a fee for letting companies monitor their data streams or query their data stores; or maybe they would only charge for data that a company uses; or perhaps the fees will be based on outcomes such as clicks or sales. Most of these schemes will ultimately require some way to estimate the value contributed by a particular piece of data.

Hyper-personalized message delivery requires more infrastructure and business model innovations. One of the most important developments in marketing today is the emergence of new channels that allow direct customer interaction: these include interactive TV, social commerce, online games, retail media, and even  interactive podcasts and out-of-home advertising. All are alternatives to common web display ads, social media ads, and search ads, which are also becoming more interactive. As with data, the key change made possible by AI is the ability to monitor vastly more opportunities at once, to evaluate the potential of each opportunity in real time, and to take advantage of the opportunities offering the greatest value.

I certainly hope that everyone reading this realizes that what I’m describing is far beyond the capabilities of today’s AI systems. The data access process requires AI to continuously ingest, clean, and integrate data from multiple sources and to automatically adapt as new sources appear and established sources change. Remember that AI is just beginning to address the bottleneck of incorporating new data sources into today’s CDP and warehouse systems. Similarly, we’re just beginning to see AI systems deliver intermediate steps on the way to hyper-personalized messaging. Today’s cutting edge is automated campaign design, which at best (and with much-needed human quality checks) could transform a user’s prompt into a complete campaign package of audience selection, content, and delivery rules and then execute that package. While impressive, this still uses the conventional structure of a few, discrete segments, content pieces, and campaign flows. That makes it closer to substitution than true transformation.

Another way to look at this is that the vision offers a roadmap for future AI development. The current frontiers in AI are goal-seeking agents, access to external data (Anthropic’s Model Context Protocol), and agent cooperation (Google’s Agent-to-Agent). (See this Medium post for a good overview of these.) If my vision is correct (which is by no means certain), steps beyond those frontiers will include proactive data gathering and integration, automated data value assessments, greater situation awareness, and better simulation of human behaviors. (I'd really like to say "understanding" of human behaviors but don't think we can quite attribute that to AI.)  AI will also need more economical processing and reliable guardrails against hallucinations, biases, privacy breaches, and generally bad behavior.

The table below offers a more detailed view of where I think things are headed. It looks at four major customer data processes: customer data management, people issues related to customer data management, customer data activation, and advertising. For each process, it lists the required capabilities for each of the four diffusion stages.  You can think of these as requirements for new, AI-based products.

AI Applications for Customer Data (STIB Model)

Data Management People Activation Advertising
Substitute [execute via co-pilots and agents] Data collection, ID resolution, connectors, metadata Understand applications, requirements, training Segmentation, analytics, prediction, sharing, privacy Audience assembly, media buys, data buys
Transform [execute via unified AI systems] Unified process, data as service Management tools, define goals/prompts, explore opportunities Hyper-personalize messages Deliver best customer, data, channel/media; optimize spend
Infrastructure [required capabilities; many delivered via AI] Automated data access, security, privacy, quality, transforms Learning systems, training systems, process design systems Efficient processing, attribution, instant commerce, buyer agents Secure data sharing, consented data assembly, contextual targeting, marketplaces, fractional billing
Business Model [rely on AI for analytics and operations] Value-based pricing Pay for skill achievement Goal achievement, Sales as a service Goal achievement, audience as a service, value-based pricing

Where does all this lead? Here are the main points I hope you’ll take away:

1. The impact of AI on customer data is just beginning. We can expect AI to be deployed in the same pattern as other technologies.  At first, it will substitute for humans or non-AI systems by performing the same tasks within existing workflows. Over time, it will transform those workflows into new processes that take full advantage of what AI can do. Ultimately, the industry will develop new business models and infrastructures to support the transformed processes.

2. For developers: consider which stage your AI project is targeting. While substitution is low-hanging fruit, bear in mind that current processes will soon be obsolete. Consider developing products that support transformed processes and their related business models and infrastructures.

3. For users: current self-service AI tools may enable you to build your own substitutions with minimal investment. Unlike system developers, you can afford to deploy these now and then discard them when something better comes along. 

4. If you have a vision for a transformed process, you could try to build it.  It won't achieve its full potential because the supporting AI capabilities, infrastructure, and business models are not yet available.  But it might deliver enough value to justify the investment: imagine hyper-personalization based only on one company's own data and deployed only on the company's own customer-facing systems.  

5. The final form of the transformed process will emerge over time.  As companies experiment with different approaches, the industry will converge on an optimal design.  Commercial developers will then build systems with this design and flesh out the supporting AI features, infrastructures and business models.  As with most complex systems, commercial vendors will probably own most of the market because they can afford to invest more in their products than most individual firms.

6. Standardized integration mechanisms will play an important role in this new world.  This assumes I'm right that the transformed process will rely heavily on connecting to external data and external delivery channels. This should make integration a particularly fruitful area for investment if you're a developer, or for building expertise if you're a user.

__________________________________________________________________________________

*As a practical matter, it's unlikely that thousands of companies would separately monitor thousands of data sources and delivery touchpoints.  It's more likely that data sources and delivery systems will broadcast event information which marketers can read and react to as they see fit.  Perhaps the sources and touchpoints will package their events into channels covering different customer groups or event types and let marketers subscribe to the channels they want.  

Friday, April 26, 2024

CDP Overview: How We Got Here, Where We're Going, and What Could Get in the Way

Has anyone asked you recently about the past, present, and future of Customer Data Platforms?  No?  That's odd; people ask me those questions

all the time.  Here are my current answers. 

It’s eleven years since the term “customer data platform” appeared in 2013.  What stages has it gone through over that time?
 

The first stage was just to recognize that CDP was a separate category of software.  What was new about CDP was that it was packaged software that was building a customer database.  Before then, customer databases were custom projects, like a data warehouse, and the only packaged marketing software was applications like campaign management systems or predictive modeling tools.  The earliest CDPs actually bundled the database building capability with an application.  But, fairly soon, vendors realized that there was more value in the database building features, which were rare, than in the applications themselves, which were fairly common.

Once we had identified the CDP category, the next stage was convincing people that it was important.  The concept itself was easy to grasp (“all customer data in one place”).  But there was initial skepticism about whether it was a legitimately separate category or just another name for existing technologies such as CRM, data warehouses or DMPs.  So we spent most of our time explaining the difference between CDP and other systems that also built customer databases.  In fact, the formal CDP Institute definition – "packaged software that builds a persistent, unified customer database accessible to other systems" – is carefully crafted so that each term identifies a differentiator between CDP and some other type of system.  “Packaged software”, for example, distinguishes CDP from data warehouses and data lakes, which are custom built.  I won’t go through the other elements point-by-point.

Once the concept was reasonably well defined, we faced skepticism about need for separate CDP database, compared to just reading existing source data in real time to build profiles.  That’s what ”persistent” refers to in the CDP definition.  It took the big martech vendors including Salesforce and Adobe several years to accept that.  The reason is  that building profiles on demand by assembling data in real time just takes too long.

Even after category was established, there was great confusion about what really qualified as a CDP.  That’s why the CDP Institute launched its RealCDP certification program, which expanded on our definition by defining five key requirements: assemble all data types, keep all details, retain the data indefinitely (subject to regulatory constraints), build unified profiles, and share the data with other systems.  We later added a sixth point relating to two real time capabilities: access to individual customer profiles, for things like call centers and website personalization, and real time event triggers, for things like responding to dropped shopping carts.  Of course, we are just one voice among many, and are easily drowned out by vendor promotions. So, unfortunately, some of that confusion persists to this day.

Part of the reason for the continuing confusion is a debate over whether CDP just builds profiles or also should include data activation capabilities such as analytics and personalization.  Our view is they’re not essential, but over time, we’ve seen the industry split between CDPs that only build profiles and CDPs that include activation functions.  The majority of CDP vendors provide activation, so that’s clearly what most buyers want.

More recently, we’ve seen interest in using customer data beyond marketing, which makes IT and data teams more interested in CDP projects.  That’s a big change because when CDP was just a marketing tool, IT teams were very happy to let marketers buy the CDP becaise it kept the marketers happy without consuming IT resources.  Now, CDP is too important to be left to marketers.  This also makes the CDPs that just build profiles more appealing in some situations – and we have in fact seen slightly faster growth in that group most recently.

IT involvement in turn brings more interest in companies building their own CDP, and leveraging their existing data lakes and warehouses as the foundation.  This has been called ‘composable CDP’ although it’s not really the right use of the term ‘composable’.  Most people now agree that 'warehouse-based' is a better label.  Semantics aside, the problem is that IT teams often underestimate the requirements for a proper CDP and, thus, the work involved in creating one.  So it’s important to ensure they do a thorough job of scoping the project in advance, so they make the best choice.

The other thing we’ve seen, more or less continuously, is expansion of CDP into new industries.  Originally they were used primarily in retail and media.  Then, they grew more common in financial services, hospitality, and telecom, which are all industries that traditionally had pretty good customer data systems.  Most recently, we’re seeing CDPs in education, healthcare, and government applications.
 

We’re also seeing CDP used more for advertising applications, as companies lean more heavily on first-party data to replace the loss of third-party cookies and replace other targeting methods that become harder as privacy rules become more stringent.  
 

In fact, CDP also supports other privacy-related applications, such as closer control over ensuring that contacts are authorized by consumer consent, and providing data clean rooms for privacy-safe data sharing.  Funneling all customer list creation through a CDP is one way to avoid breaking privacy rules.

Where do you see the industry headed next?  

The biggest issue right now is the movement towards ‘composability’.  We can’t control whether IT and data departments try to build their own CDP-equivalent.  But we can educate them about the actual requirements and we can give them tools that make it easier to meet those requirements.  Many CDP vendors are now breaking apart their systems to provide modules that companies can use as components in building an internal system.  Of course, that helps those vendors to survive a transition to a composable CDP world, but it also helps to maintain the reputation of the industry as a whole.  What we don’t want to see is composable CDP projects fail because that makes people question the value of the CDP concept itself.

Closely related to the composable trend is a trend for ‘no copy’ access to external data by a CDP.  This means the CDP can read data from other systems without copying it into the CDP data store.  That used to be more or less impossible at scale, because finding, reading, and integrating masses of external data took too long.  With today’s technologies, that process can be much faster, so it becomes a more practical alternative.  Some common use cases are reading things that change quickly and are only relevant at the moment you need them, like inventory levels or local weather conditions.  Of course, this ability also blurs the distinction between a traditional CDP, which imported all its data, and a warehouse-based CDP, which works with data stored externally.  That’s okay but it does add still more confusion to the discussion.  

I’ll also make a side note that the original CDP skeptics argued for reading source system data on demand, so it may seem this proves they were right.  But so far I don’t think you can have a CDP that only works by reading external data on demand, because some processes like identity resolution and data aggregation still take too long to do purely on the fly.  So I see the future CDP as having a core of data that it does copy and store in its own database, for things like the identity graph that tells how to combine data from different sources into each customer’s profile.  Whether that database is a CDP or data warehouse doesn't matter: either way, the data will have been copied from the original source system and preprocessed for use by the CDP.  The core data could well be supplemented with data read on-demand from external systems, which could be the original source or a data warehouse or data lake.  At least in theory, this would give you the best of both worlds: minimal data copying but maximum performance.

Putting aside composability, CDPs will need to adjust to new privacy rules by adding features like encryption, advanced privacy policy management, and data clean rooms.  They’ll adjust to new media and new data types, like audio and video, which need to be not just stored but analyzed in ways that are currently close to impossible.  And, of course, they’ll adjust to growing AI capabilities, which will make it easier to perform some functions like adding new data sources, matching identifiers that belong to the same person, building predictive models, and analyzing business results.  AI will also add to the volume and complexity of data that CDPs have to handle, which itself may require new technology to support.  For example, if AI starts to create a huge variety of content personalized for each individual, that makes result analysis vastly more complicated.  Somehow the CDP will need to deal with that.

On a more prosaic level, I expect to see more CDPs that are tailored to the needs of particular industries.  That’s a typical development for a mature software category.  Specialist systems can be more cost-effective to build and deploy, because they use special data structures and features tailored to a particular industry’s needs.  This will bring down the cost of CDPs, which has been an obstacle to expanding the base of CDP purchasers.  

And, as I’ve already mentioned, I expect to see CDPs used more widely beyond marketing.  One thing we’ve learned in recent years is that customers expect a personalized experience every time they interact with a company, whether it’s before they buy a product or after they start using it.  That requires making customer data available at every interaction point.  Beyond direct customer interactions, teams like product development and operations planning also can benefit from using customer data.

Are there any particular obstacles or threats to the CDP’s continued success?

Certainly CDPs will have to adapt to the changes we’ve already discussed in data types and volumes, in the users for customer data, and in whatever technical developments change the best way for CDPs to be built.  Individual vendors may struggle to keep up.  But I think the need for complete, sharable customer profiles is here to stay.  So to the extent that that’s the core definition of a CDP, the future of the industry is secure.

That said, I do see two major threats to the CDP industry as we know it today.  

The first is technical: the cloud databases from specialists like Snowflake and Databricks and general cloud platforms like Google Cloud, AWS, and Microsoft Azure.  All those vendors are increasingly eyeing the giant pile of customer data held in a CDP and wanting it for themselves.  To get it, they’re adding applications to build profiles, such as data quality and identity resolution, and to do analytics and marketing.  Sometimes they make the applications to be features to their own database systems; sometimes they build direct integrations with specialist applications through a marketplace of some sort.  Either way, this cuts out the CDP, which has traditionally been an intermediary between enterprise data stores and business applications.  Again, this comes back to the notion of the warehouse as the primary customer data store, which makes the CDP database unnecessary.  I think threat is limited today because most IT departments don’t have the resources to build those systems for themselves. But as new tools make it easier to build those systems, the threat becomes more important.  There are really just two ways for CDPs to respond to this threat:

  • one is to become platforms themselves, which is to say, to replace the data warehouse itself.  That’s not as crazy as it sounds, since the CDP is really a set of tools that builds the customer database, so it can work on top of a Snowflake or Google BigQuery or whatever database the client wants.  In this world, the CDP evolves from a tool to build customer profiles to a tool to build data warehouses in general.  Difficult but not impossible, at least for the largest CDP vendors who have the resources to compete.
  • The other option is to become applications on top of the warehouse, offering specialized capabilities like cross-channel journey orchestration.  The would build on CDPs’ existing capabilities to share their profiles with activation systems, and to themselves do activation functions that apply across all channels, such as predictive models and best offer selection.  It’s actually an easier path for most CDP vendors, since it continues their role as intermediaries between data sources and business applications.  But it’s also a fairly narrow role and there will be lots of competition.  Specialization by industry, company size, and other variables will be the key to avoiding that competition by becoming the best choice in a particular niche.

The second threat isn’t technical, but the ability of organizations to actually make use of a CDP once it’s built.  We run into this all the time, when companies tell us their staff doesn’t know what to do with a CDP or doesn’t have the skills to do what they’d like.  It’s a problem that will only get worse as customer data grows more complicated and there are more possibilities for using customer data.  Maybe AI will solve the problem, and that’s something CDP vendors need to invest in to make happen.  But there’s no guarantee that AI will be the answer, and my guess is that even AI will need skilled users to take advantage of the possibilities it creates.  

Of course, if the CDP turns out not to be useful, the staff members will blame the CDP, not their own lack of skill.  But it doesn’t really matter who takes the blame: if CDPs don’t create value, companies won’t be willing to invest in them.  So it’s really important for the CDP industry to help train CDP users, not just in the technical details of how to use a CDP, but in the business programs that a CDP can support. 

Sunday, October 29, 2023

Does CDP Need a New Definition?

The earliest Customer Data Platform systems were introduced before 2010; the term CDP was coined to describe this emerging class in 2013. My definition had changed very little when we launched the CDP Institute in 2016, and has been the same ever since: “packaged software that builds a unified, persistent customer database accessible to other systems”. The Institute added the RealCDP checklist* in 2019 to attach more specifics to the definition in the hopes of helping buyers ensure a system that called itself a CDP could actually support the use cases they expected a CDP to support. By then, industry analysts were beginning to offer their own definitions which, while worded differently, were broadly consistent with the Institute definition. Even the major marketing suite vendors, who initially argued a separate (“persistent”) database wasn’t necessary, eventually discarded that position and introduced products that matched our criteria.

A successful concept like CDP quickly takes on a life of its own. It soon became apparent that many people were using CDP in a much looser sense to mean any system that built and shared customer profiles. This extended past packaged software to include custom-built systems and included systems whose scope was more limited than a true CDP. This expansion skewed some survey resuls but otherwise seemed relatively harmless; in any case, resistance seemed both pedantic and futile. What really mattered was these systems still gave CDP users access to the unified profiles.

Unfortunately, the evolution of the term didn’t stop there. As CDP became popular, many vendors adopted the label whether or not they actually met even the looser definition. At the same time, legitimate CDP vendors offered additional capabilities to analyze and deploy the data in the profiles. The resulting confusion ultimately led some vendors to avoid the CDP label entirely because it no longer provided a useful way to differentiate their products. Today, vendors seeking the latest label are more likely to call themselves digital experience managers than a CDP, even if their products meet the CDP requirements.

But the greatest challenge to the utility of the CDP label arose in the past few years when a number of vendors chose to claim that the core feature of the CDP – a dedicated database built by importing data from other systems, a.k.a. “persistence” – could be abandoned while still calling the result a CDP.  Their argument was the customer profiles could reside in a general purpose data warehouse, which most companies already had in place. 

The claim gained some plausibility from the fact that modern cloud data warehouse technologies, such as Snowflake, Google Big Query, and AWS Redshift, are in fact used by some conventional CDP vendors. The problem was they implicitly assumed that every company’s data warehouse had organized the data into useable customer profiles. In fact, very few data warehouses perform the specific tasks, most notably identity resolution, customer-level aggregation, and real-time response, needed to support CDP use cases. While it’s technically possible to add those features to an existing data warehouse, it’s usually a major project that often costs more, takes longer, and delivers less useful results than installing a separate, conventional CDP. (As always, the details depend on the situation.)

One positive result of the interest in warehouse-based profiles has been the decision of some CDP vendors to break their systems into modules that let users buy the data preparation functions separately from the rest of the CDP. This lets companies that want a warehouse-based system to still benefit from the mature capabilities those CDP vendors have developed over many years. These vendors have also often added the ability to combine data from a warehouse or other external system with data stored in a conventional, persistent CDP database, without actually loading that external data into the CDP. This gains some advantages of the warehouse-based approach, such as reduced data movement and storage costs, while retaining the benefits of the persistent CDP, such as greater control and flexibility.

You’ll note that all these changes affect the input side of the CDP data flow. It was once a very simple process: data from other systems was copied into the CDP, where it was formatted into profiles and shared with other systems. Now, the input process may be some mix of copying data into the CDP, reading fully-formed profiles from a warehouse, or combining internal and external data. By contrast, the delivery side of the process has remained the same: the CDP shares its profiles with other systems. 

In some cases, this has led to a subtle shift in perception of the CDP’s purpose: from a system that builds customer profiles, to a system that delivers those profiles to other systems. In this view, the core function of the CDP is to convert general purpose profiles into the specific formats needed by analytical, orchestration, and delivery systems (which we can call “activation systems” if you’ll trade some jargon for simplicity). This may still involve some data processing, such as advance calculation of model scores and aggregates to make them available in real time, and new data structures to hold the results of that processing. So there’s more happening here than a simple data transfer or API call.

Some people carry this shift even further and argue the CDP should really be defined as an activation system that reads profiles from somewhere else. Since three-quarters of the CDPs we track actually do have at least some activation capabilities, this isn’t quite as crazy as it may sound.

All that said, I’m not yet ready to redefine CDP. “Packaged software” and “persistent customer database” are meaningful terms that distinguish one configuration from other approaches (“custom software” and “external profiles”) that can also make profiles available to other systems. More important, the packaged/persistent configuration has significant cost and execution advantages over the custom/external alternative. So it’s important to avoid blurring the distinction between the two approaches. 

 At most, I’d offer retronyms that distinguish the “packaged CDP” (packaged/persistent) from the “warehouse CDP” (custom/external). By definition, the “warehouse CDP” doesn’t build its own profiles, so the term seems to ignore the fundamental function of the CDP.  You might call that oxymoronic, if not just plain moronic. But if we slightly redefine the core function of the CDP to be delivery of profiles rather than creating them, we can overcome that objection and, perhaps, give the market terms it intuitively understands.

You’ll also note the shift from profile creation to delivery puts the emphasis on the delivery side of the CDP flow, which we’ve noted has been the most stable and applies to both the packaged and warehouse approaches. This lets us join the trend to redefining CDP as a profile sharing system.

In short, the key distinction is whether the primary customer profiles are built and stored in the company data warehouse or in a separate CDP database. It’s worth maintaining that distinction because one approach relies on the company’s technical staff to assemble the functions needed to build and store the profiles, while the other relies on a packaged CDP to provide those functions. There are variations within each theme, including whether the warehouse uses modules from a CDP vendor to assemble its profiles or to help deliver them, whether real-time data is posted to the warehouse or held separately, and whether the CDP enriches its profiles with external data without importing that data. These are important from a practical standpoint, but do not affect the fundamental architecture of the system. Focusing first on where the profiles reside should help buyers understand the most important choice they have to make. This choice will in turn determine the other decisions they have to consider. 

 

_________________________________________________________

* ingest data from all sources, retain all details, keep the data as long as desired, build unified profiles, share the profiles with any other system, and enable real-time event-triggers and profile access.

Monday, February 03, 2020

Salesforce Buys Evergage But Not For CDP

The CDP Institute published its semi-annual Industry Update report today, which you download here for free. Although every word and image in the report is a jewel, there’s no question that the main story in this edition is CDP industry consolidation. Events in the past six months (stretching a bit to include early January 2020) include seven new funding rounds, three acquisitions of CDP vendors, four acquisitions by CDP vendors, and four asset sales by CDP companies.  Asset sales aside, these are all ways for companies to strengthen their business more quickly than organic growth permits.

What’s particularly intriguing is the industry position of the firms in these deals.  Using our best guess at CDP employment for each vendor, only one of the twelve vendors involved funding or either side of an acquisition is among the industry’s five largest (SessionM, bought by Mastercard). The rest all ranked within the fairly narrow band from number eight to number thirty (of 101 total). That is, they were bigger than most but not the industry's largest.  I interpret this to mean that these vendors were either adding resources for a push to reach the industry top tier or have already decided they need to be part of something else.  

Notably, the firms that engaged in asset sales were much smaller: only IgnitionOne would have fallen within the top thirty and their deal might be considered more of an acquisition, since Zeta Global is apparently still selling the product.


Careful readers will have already noticed that this chart includes one other deal: acquisition of Evergage by Salesforce, announced on Monday.  Evergage fits into the size range of the other deals and the sale can certainly be seen as an escape from the crowded campaign CDP space. But the purchase is otherwise atypical because Salesforce has stressed that they are primarily interested in Everage for real time interaction management and personalization.  Of course, Salesforce is already far along in work on its own CDP, the Customer 360 Audiences component of Customer 360 Truth, which is due for general release around June. So this deal has little to do with Evergage as a CDP.  It's about closing a gap elsewhere in the Salesforce product line, not a sudden acceleration of Salesforce’s entry into the CDP space.

Monday, December 30, 2019

Explaining Martech to a Five Year Old

One piece of paper stands between me and ending the year with a clean desk: a list scribbled on hotel stationery that compares different customer data systems to different types of motor vehicles. Richard Scarry  meets CDP Institute, you might say.

So, just in case your New Year’s plan includes explaining marketing technology to a five year old, here we go:

  • Data Warehouse = School Bus. The seats in a bus are lined up in nice neat rows, designed to carry one type of cargo very efficiently. Similarly, a data warehouse is highly structured environment that is very efficient at dealing with specified data types. (See how this works?)

  • Data Lake = Moving Van. A moving van is a big box that can hold pretty much anything, although it might be jumbled together in no particular order. A data lake can store any type of data but doesn’t do much to organize it.
  • Integration Platform = Delivery Motorcycle. A motorcycle carries small items quickly from one place to another, but doesn’t have any place to store them. An integration platform moves bits of data between systems but doesn’t have its own storage, either.
  • Data Management Platform = Fire Truck. A fire truck is a highly specialized vehicle designed to move very quickly and pour out huge volumes of water. A DMP is a highly specialized system that quickly moves huge volumes of data.
  • CRM System = Taxi. A taxi carries one person and their baggage directly to a specific destination. A CRM system delivers one customer and their history to a single sales person or call center agent.
  • Cloud Platform = Car Carrier. A car carrier is a frame that can hold many unconnected vehicles. A Cloud Platform supports many unrelated systems in the same rack.
Believe it or not, my original list didn’t include an entry for Customer Data Platforms. So let’s give that a moment’s thought and go with…Ice Cream Truck!  They make a lot of noise and everybody loves them.  No, wait, better still...Food Truck!  A food truck collects ingredients from various sources, converts them into delicious meals, and distributes the results to many happy customers.  A CDP combines customer data into profiles that it shares with different systems.  

You’re welcome.

Happy New Year!

Saturday, November 24, 2018

Oracle CX Unity Looks Like a Real Customer Data Platform

I finally caught up with Oracle for a briefing on the CX Unity product they announced in October. Although it was clear at the time that CX Unity offered some version of unified customer data, it was hard to understand exactly what was being delivered. The picture is now much clearer. Here are straight answers to important questions:

It’s a persistent database. CX Unity will ingest all types of data – structured, semi-structured, and unstructured – from Oracle’s own CX systems via prebuilt connectors and from external systems via APIs, batch imports, or Oracle’s integration cloud.  It will store these in well-defined structures defined by marketing operations or similar lightly-technical users.  The structures will include both raw data and derived variables such as predictive model scores. Oracle plans to release a dozen industry-specific data schemas including B2B and B2C verticals.

It does identity resolution. CX Unity will support deterministic matching for known relationships between customer identifiers and will maintain a persistent ID over time. It will link to the Oracle Identity Graph in the Oracle Data Cloud for probabilistic matches using third party data.

It activates data in near real time. CX Unity can ingest data in real time but it takes 15 to 20 minutes or longer to standardize, match, run models, and place it in accessible formats such as data cubes. Oracle expects that real-time interactions and triggers will run outside of CX Unity.

It shares data with all other systems. Oracle has built connectors to expose CX Unity data within its own customer-facing CX Cloud systems. APIs are available to publish data to other systems but it’s up to partners and clients to use them.

It integrates machine learning. CX Unity includes machine learning for predictive models and recommendations. Results are exposed to customer-facing systems. This capability seems to be what Oracle has in mind when it contrasts CX Unity with other customer data management solutions that it calls merely “database centric”.

It’s not live yet. The B2C customer segmentation features of CX Unity are available now. The full system is slated to be available in mid-2019.

These answers mean that CX Unity meets the definition of a Customer Data Platform: packaged software that creates a unified persistent customer database accessible by other systems. The machine learning and recommendation features would put it in the class of “personalization” CDPs I defined earlier this month. This is a sharp contrast with the CDP alternatives from Oracle’s main marketing cloud competitors: Customer 360 from Salesforce (no persistent database) and Open Data Initiative from Adobe, Microsoft and SAP (more of a standard than a packaged system).

It's likely that CX Unity will be bought mostly by current Oracle CX customers, although Oracle would doubtless be happy to sell it elsewhere.  But even if CX Unity sales are limited, its feature list offers a template for buyers to measure other systems against. That will create a broader understanding of what belong in a customer management system and make it more likely that buyers will get a CDP that truly meet their needs.  So its release is a welcome development – especially as Oracle finds ways to present it effectively.

Friday, February 02, 2018

Celebrus CDP Offers In-Memory Profiles

It’s almost ten years to the day since I first wrote about Celebrus, which then called itself speed-trap (a term that presumably has fewer negative connotations in the U.K. than the U.S.). Back then, they were an easy-to-deploy Web site script that captured detailed visitor behaviors. Today, they gather data from all sources, map it to a client-tailored version of a 100+ table data model, and expose the results to analytics and customer engagement systems as in-memory profiles.

Does that make them a Customer Data Platform? Well, Celebrus calls itself one – in fact, they were an early and enthusiastic adopter of the label. More important, they do what CDPs do: gather, unify, and share customer data. But Celebrus does differ in several ways from most CDP products:

- in-memory data. When Celebrus described their product to me, it sounded like they don’t keep a persistent copy of the detailed data they ingest. But after further discussion, I found they really meant they don’t keep it within those in-memory profiles. They can actually store as much detail as the client chooses and query it to extract information that hasn't been kept in memory.  The queries can run in real time if needed. That’s no different from most other CDPs, which nearly always need to extract and reformat the detailed data to make it available. I’m not sure why Celebrus presents themselves this way; it might be that they have traditionally partnered with companies like Teradata and SAS that themselves provided the data store, or that they partnered with firms like Pega, Salesforce, and Adobe that positioned themselves as the primary repository, or simply to avoid ruffling feathers in IT departments that didn't want another data warehouse or data lake.  In any case, don’t let this confuse you: Celebrus can indeed store all your detailed customer data and will expose whatever parts you need.

- standard data model. Many CDPs load source data without mapping it to a specific schema. This helps to reduce the time and cost of implementation. But mapping is needed later to extract the data in a usable form. In particular, any CDP needs to identify core bits of customer information such as name, address, and identifiers  that connect records related to the same person. Some CDPs do have elaborate data models, especially if they’re loading data from specific source systems or are tailored to a specific industry.  Celebrus does let users add custom fields and tables, so its standard data model doesn’t ultimately restrict what the system can store.

- real-time access.  The in-memory profiles allow external systems to call Celebrus for real-time tasks such as Web site personalization or bidding on impressions..  Celebrus also loads, transforms, and exposes its inputs in real time.  It isn't the only CDP to do this, but it's one of just a few..


Celebrus is also a bit outside the CDP mainstream in other ways. Their clients have been largely concentrated in financial services, while most CDPs have sold primarily to online and offline retailers. While most CDPs run as a cloud-based service, Celebrus supports cloud and on-premise deployments, which are preferred by many financial services companies.  Most CDPs are bought by marketing departments, but Celebrus is often purchased by customer experience, IT, analytics, and digital transformation teams and used for non-marketing applications such as fraud detection and system performance monitoring.

Other Celebrus features are found in some but not most CDPs, so they’re worth noting if they happen to be on your wish list. These include ability to scan for events and issue alerts; handling of offline as well as online identity data; and specialized functions to comply with the European Union’s GDPR privacy rules.

And Celebrus is fairly typical in limiting its focus to data assembly functions, without adding extensive analytics or customer engagement capabilities.  That's particularly common in CDPs that sell to large enterprises, which is  Celebrus' main market.  Similarly, Celebrus is typical in providing only deterministic matching functions to assemble customer data. 



So, yes, Celebrus is a Customer Data Platform.  But, like all CDPs, it has its own particular combination of capabilities that should be understood by buyers who hope to find a system that fits their needs.

As I already mentioned, Celebrus is sold mostly to large enterprises with complex needs.  Pricing reflects this, tending to be "in the six or seven figures" according the company and being based on input volume, types of connected systems, and license model (term or perpetual, SaaS, on-premise, or hybrid).  The company hasn’t released the number of clients but says it gathers data from "tens of thousands" of Web sites, apps, and other digital sources.  Celebrus has been owned since 2011 by D4T4 Solutions  (which looks like the word “data” if you use the right type face), a firm that provides data management services and analytics. 

Wednesday, January 24, 2018

Simple Questions to Screen Customer Data Platform Vendors

I’ve been working for months to find a way to help marketers understand the differences between Customer Data Platform vendors. After several trial balloons and with considerable help from industry friends, I recently published a set of criteria that I think will do the job. You can see the full explanation on the CDP Institute blog. But, since this blog has its own readership I figured I’d post the basics here as well.

The primary goal is give marketers a relatively easy way to decide which CDPs are likely to meet their needs. To do this I’ve come up with a a small list of features that relate directly to working with particular data sources and supporting particular applications. The theory is that marketers know what sources and applications they need to support, even if they're not experts in the fine points of CDP technology.

In other words, read these items as meaning: if you want your CDP to support [this data type or application] then it should have [this feature].

Obviously this list covers just a tiny fraction of all possible CDP features. It’s up to marketers to dig into the details of each system to determine how well it supports their specific needs.  We have detailed lists of CDP features in the Evaluation section of the CDP Institute Library.

The final list also includes a few features that are present in all CDPs (or, more precisely, in all systems that I consider a CDP – we can’t control what vendors say about themselves). These are presented since there’s still some confusion about how CDPs differ from other types of systems.

Now that the list is set, the next step is to research which features are actually present in which vendors and publish the results. That will take a while but when it’s done I’ll certainly announce it here.

Here’s the list:

Shared CDP Features: Every CDP does all of these. Non-CDPs may or may not.
  • Retain original detail. The system stores data with all the detail provided when it was loaded. This means all details associated with purchase transactions, promotion history, Web browsing logs, changes to personal data, etc. Inputs might be physically reformatted when they’re loaded into the CDP but can be reconstructed if needed.
  • Persistent data. The system retains the input data as long as the customer chooses. (This is implied by the previous item but is listed separately to simplify comparison with non-CDP systems.)
  • Individual detail. The system can access all detailed data associated with each person. (This is also implied by the first item but is a critical difference from systems that only store and access segment tags on customer records.)
  • Vendor-neutral access. All stored data can be exposed to any external system, not only components of the vendor’s own suite. Exposing particular items might require some set-up and access is not necessarily a real time query.
  • Manage Personally Identifiable Information (PII). The system manages Personally Identifiable Information such as name, address, email, and phone number. PII is subject to privacy and security regulations that vary based on data type, location, permissions, and other factors.
Differentiating CDP Features: A CDP doesn’t have to do any of these although many do some and some do many. These are divided into three subclasses: data management, analytics, and customer engagement.

Data Management. These are features that gather, assemble, and expose the CDP data.

     Base Features. These apply to all types of data.
  • API/query access. External systems can access CDP data via an API or standard query language such as SQL. It’s just barely acceptable for a CDP to not offer this function and instead provide access through data extracts. But API or query access is much preferred and usually available. API or query access often requires some intermediate configuration, reformatting, or indexing to expose items within the CDP’s primarily data store. Those are important details that buyers must explore separately.
  • Persistent ID. The system assigns each person an internal identifier and maintains it over time despite changes or multiple versions of other identifiers, such as email address or phone number. This allows the CDP to maintain individual history over time, even when source systems might discard old identifiers. CDPs that use a persistent ID applied outside of the system do not meet this requirement.
  • Deterministic match (a.k.a. “identity stitching”). The system can store multiple identifiers known to belong to the same person and link them to a shared ID (usually the persistent ID). This enables the system to connect identifiers indirectly: for example, if an email linked to an account is opened on a particular device, subsequent activity on that device can also be linked to the account.
  • Probabilistic match (a.k.a. “cross device match”). The system can apply statistical methods and rules to identify multiple devices used by the same person, such as computers, tablets, smart phones, and home appliances. While many CDPs rely on third party services for this sort of matching, this item refers only to matching done by the CDP itself.
     Unstructured and Semi-Structured Data. This refers to loading data from unstructured or semi-structured sources such as Web logs, social media comments, voice, video, or mages. These are typically managed with “big data” technologies such as Hadoop. Nearly all CDPs use some version of this technology but it’s only essential if clients have unstructured or semi-structured sources and/or very high data volumes. Some CDPs handle very high data volumes in structured databases such as Amazon Redshift.
  • JSON load. The system can accept and store data through JSON feeds without the user specifying in advance the specific attributes that will be included. Additional configuration may later be required to access this data. There are some alternatives to JSON that offer similar capabilities.
  • Schema-free data store. The system uses a data store that does not require advance specification of the elements to be stored. Examples include Hadoop, Cassanda, MongoDB, and Neo4J.
     Web Site. This refers to interactions with the company’s own Web site, whether on a desktop computer or mobile device.
  • Javascript tag. The system provides a Javascript tag that can be loaded into the client’s Web site and used to capture data about customer behaviors. Some CDP vendors provide full tag management systems but this is not a requirement for this item. This item does require that data captured by the Javascript tag can be associated with a customer record in the CDP database. This is usually done with a Web tracking cookie but sometimes through other methods.
  • Cookie management. The system can deploy and maintain Web browser cookies associated with the client’s own Web site. The cookies can be linked to customer records in the CDP database.
     Mobile Apps. This refers to interactions with mobile apps created by the company.
  • SDK load. The system offers a Software Development Kit (SDK) that can load data from a mobile app into the CDP database. It must be able to associate the data with individual customers in the CDP database. This is usually done through an app ID. Other SDK features such as message delivery are not a requirement for this item.
     Display Ads. This refers to interactions through display advertising networks, including social media networks.
  • Audience API. The system has an API that can send customer lists from the CDP to systems that will use them as advertising audiences. The receiving systems might be Data Management Platforms, Demand Side Platforms, advertising exchanges, social media publishers, or others. Ability to receive information back from the advertising systems is not a requirement for this item.
  • Cookie synch. The CDP can match its own cookie IDs with third party cookie IDs to allow the marketer to enrich profiles with external data or reach users through advertising networks.
     Offline. This refers to interactions managed through offline sources such as direct mail and retail stores, where the customer’s primary identifier is name and postal address.
  • Postal Address. The system can clean, standardize, verify, and otherwise work with postal addresses. This processing is reduces inconsistencies and makes matching more effective. Systems meet this requirement so long as the address processing is built into system process flows, even if they rely on third party software. Systems that send records to external systems in a batch process do not meet this requirement.
  • Name/Address Match. The system can find matches between different postal name/address records despite variations in spelling, missing data elements, and similar differences. As with postal processing, systems can meet this requirement with third party matching software so long as the software is embedded in their processing flows.
     Business to Business. This refers to companies that sell to other businesses rather than to consumers.
  • Account-level data. The system can maintain separate customer records for accounts (i.e., businesses) and for individuals within those accounts. This means account information is stored and updated separately from individual information. It also means that selections, campaigns, reports, analyses, and other system activities can combine data from both levels.
  • Lead to Account Match. The system can determine which individuals should be associated with which account records, using information such as company name, address, email domain, and telephone number. This excludes processing done by sending batch files to external vendors.
Analytics. These are applications that use the CDP data but don’t extend to selecting messages, which is the province of customer engagement.
  • Segmentation. The system lets non-technical users define customer segments and automatically send segment member information to external systems on a user-defined schedule. Ideally, all data would be available to use in the segment definitions and to include in the extract files. In practice, some configuration may be needed to expose particular elements. Systems meet this requirement regardless of whether segments are defined manually or discovered by automated processes such as cluster analysis.
  • Incremental attribution. The system has algorithms to estimate the incremental impact of different marketing activities on specified outcomes such as a purchase or conversion. Attribution is a specialized analytical process that relies on the unified customer data assembled by the CDP. Algorithms vary greatly. To qualify for this item, the algorithm must estimate the contribution of different marketing contacts on the final result. That is, fixed approaches such as “first touch” or “U-shaped distribution” are not included.
  • Automated predictive. The system can generate, deploy, and refresh predictive models without involvement of a technical user such as a data scientist or statistician. This usually employs some form of machine learning. There are many different types of automated predictive; systems meet this requirement if they have any of them.
 Engagement. This refers to applications that select messages for individual customers. It does not include content delivery, which is typically handled outside of the CDP.
  • Content selection. The system can select appropriate marketing or editorial content for individual customers in the current situation, based on the data it stores about them, other information, and user instructions. The instructions may employ fixed rules, predictive models, or a combination. Selections may be made as part of a batch process.
  • Multi-step campaigns. The system can select a series of marketing messages for individual customers over time, based on data and user instructions. The message sequence is defined in advance but may change or be terminated depending on customer behaviors as the sequence is executed.
  • Real-time interactions. The system can select appropriate marketing or editorial content for individual customers during a real-time interaction. This requires accepting input about the customer from a customer-facing system, finding that customer’s data within the CDP, selecting appropriate content, and sending the results back to the customer-facing system for delivery. The results might include the actual message or instructions that enable the customer-facing system to generate the message.

Sunday, March 12, 2017

Should Customer Data Platforms Be "Marketer-Controlled"?

Thomas Wieberneit argues in a thoughtful blog post that companies need one platform for consolidated customer data, but that Customer Data Platform isn’t it because the CDP is “marketer-controlled” by definition, and thus doesn’t support other departments.

This hits a nerve. Many of the CDP vendors have told me their systems are used outside of marketing. Just last week, RedPoint unveiled a “Customer Engagement Hub”* that it defines as extending beyond marketing to all customer touchpoints. When I was recently writing a paper for B2B CDP CaliberMind, they listed sales and customer success teams along with marketing as likely buyers of their product.

As these examples suggest, CDP technology can support all customer-facing departments. It’s true that different departmental applications will probably need the CDP data to be presented slightly differently. But applications within marketing also need different formats, and it’s already standard for CDPs to reformat their data for access by external systems. So there’s no technical reason to limit CDPs to marketing.

Indeed, as I argued in my last blog post, the era of isolated marketing technology systems may be coming to an end as companies return to centralized systems to ensure a seamlessly integrated customer experience. In this world, the CDP is a shared enterprise asset, and, as such, clearly not something that marketers build and control for themselves.

So what’s holding me back from changing the definition? Three things:
  • History. The fact is that CDPs evolved from systems that were built specifically for marketing. This matters because it means marketing needs drove their design. CDPs built to support sales or customer success teams would probably look a bit different, even if they used the same underlying technology. To give a concrete example, systems built for customer success don’t need to advanced identity resolution because almost all the data they care about arrives with a customer ID attached. I know that history by itself isn’t a good reason to retain the marketer-centric definition of CDP since CDPs have outgrown their origins. But it doesn’t hurt to have something remind users outside of marketing that they should look closely at whether any particular CDP can fully support their needs.
  • User control. The primary reason the CDP definition includes “marketer-controlled” is to distinguish CDPs from enterprise data warehouse (or data lake) projects run by corporate IT. That matters because such projects were historically multi-year undertakings that often failed altogether or required so many compromises among enterprise stakeholders that they didn’t meet all of marketing’s particular needs. Moreover, once built, such systems were slow and costly to update, so they didn’t adapt quickly to marketers’ fast-changing environment. This responsiveness is really the biggest change that CDPs introduced into the world of customer data management. As I noted earlier, department-controlled systems may soon be lost in a new round of centralization.  In theory, these new central systems will be vastly more responsive to user needs than the old central systems. But if I were a marketer, I’d be reluctant to give up my own systems until the new ones had proven their agility in practice.
  • Departmental buyers. Whatever the long-term future of centralization, CDPs today are almost always bought by departments.  Even when corporate IT groups are managing the process, they are generally reacting to requests from departmental users rather than executing an enterprise master plan of their own. And when you start looking at which departments are making these purchases, marketing is by far the leader.  It’s true that other departments often find uses for a CDP once it’s deployed. Whether marketers really share control of their CDPs when this happens or simply treat other users as guests, I can’t say. From a seller’s standpoint, I can certainly see companies offering their technology in packages aimed at enterprise buyers, which is exactly what RedPoint just did. But abandoning the large marketing-department segment for the nascent enterprise-buyer segment doesn’t sound like a good idea.
So it seems I’ve backed myself into a corner. CDP technologies work at the enterprise level and, logically, CDPs should be enterprise assets. But the main buyers of CDPs are marketing departments and the CDPs' main attraction is departmental control.  So redefining the CDP as an enterprise system would probably reduce its appeal even though it was technically accurate. Replacing "marketer-controlled" with “user-controlled” or “department-controlled” would broaden the scope but still doesn’t describe an enterprise-wide role. We could drop “control” from the definition and get to the heart of the matter, which is responsiveness – so instead of “marketer-controlled” we could have something like “agile” or, more concretely, “easily changed”. But those seem too vague to be useful.

I’m tempted to offer something truly lame, like “Enterprise CDP” as an alternative to plain or Departmental CDP.  It might come to that, or I might ultimately just drop “marketer-controlled” without replacing it. After all, if enterprise systems really do become flexible enough to meet departmental needs in a way that past enterprise data warehouse projects did not, then the distinction will no longer matter.

Bottom line: I’ll leave “marketer-controlled” in place for now but suspect its days are numbered.


________________________________________________________________

*CEH is a Gartner term, which they define as “an architectural framework that ties multiple systems together to optimally engage the customer. A CEH allows personalized, contextual customer engagement, whether through a human, artificial agent, or sensors, across all interaction channels. It reaches and connects all departments, allowing, for example, the synchronization of marketing, sales and customer service processes.” As the term “hub” implies, this definition doesn’t specify creation of a persistent database, something I believe is required to build an effective customer profile. Come to think of it, I'm not sure that "architectural framework" is a system at all. 

Wednesday, December 14, 2016

BlueVenn Bundles Omnichannel Journey Management, Personalization, and Single Customer View

BlueVenn has only been active in the U.S. market only since March 2016, although many U.S. marketers will recall its previous incarnation as SmartFocus.* The company offers what it calls an omnichannel marketing platform that builds a unified customer database, manages marketing campaigns, and generates personalized Web and email messages.

The Venn in BlueVenn

The unified database process, a.k.a. single customer view, has rich functionality to load data from multiple sources and do standardization, validation, enhancement, hygiene, matching, deduplication, governance and auditing. These were standard functions for traditional marketing databases, which needed them to match direct mail names and addresses, but are not always found in modern customer data platforms. BlueVenn also supports current identity linking techniques such as storing associations among cookies, email addresses, form submits, and devices. This sort of identity resolution is a batch process that runs overnight.  The system can also look up information about a specific customer in real time if an ID is provided. This lets BlueVenn support real time interactions in Web and call center channels.

Users can enhance imported data by defining derived elements with functions similar to Excel formulas. These let non-technical users put data into formats they need without the help of technical staff. Derived fields can be used in queries and reports, embedded in other derived fields, and shared among users. To avoid nasty accidents, BlueVenn blocks changes in a field definition if the field is used elsewhere. Data can be read by Tableau and other third-party tools for analysis and reporting.

BlueVenn offers several options for defining customer segments, including cross tabs, geographic map overlays, and flow charts that merge and split different groups.  But BlueVenn's signature selection tool has always the Venn diagram (intersecting circles).  This is made possible by a columnar database engine that is extremely fast at finding records with shared data elements. Clients could also use other databases including SQL Server, Amazon Redshift (also columnar), or MongoDB, although BlueVenn says nearly all its clients use the BlueVenn engine for its combination of high speed and low cost.

Customer journeys - formerly known as campaigns - are set up by connecting icons on a flow chart. The flow can be split based on yes/no critiera, field values, query results, or random groups. Records in each branch can be sent a communication, assigned to seed lists or control groups, deduplicated, tagged, held for a wait period or until they respond, merged with other branches, or exit the flow. The “merge” feature is especially important because it allows journeys to cycle indefinitely rather than ending after a sequence of steps. Merge also simplifies journey design since paths can be reunified after a split. Even today, most campaign flow charts don’t do merges.

BlueVenn Journey Flow

Tagging is also important because it lets marketers flag customers based on a combination of behaviors and data attributes. Tags can be used to control subsequent flow steps. Because tags are attached to the customer record, they can be used to coordinate journeys: one application cited by BlueVenn is to tag customers for future messages in multiple journeys and then periodically compare the tags to decide which message should actually be delivered.

Communications are handled by something called BlueRelevance. This puts a line of code on client Web sites to gather click stream data, manage first party cookies, and deliver personalized messages. The messages can include different forms of dynamic content including recommendations, coupons, and banners. In addition to Web pages, BlueVenn can send batch and triggered emails, text messages, file transfers, and direct messages in Twitter and Facebook. Next year it will add display ad audiences and Facebook Custom Audiences. The vendor is also integrating with the R statistical system for predictive models and scoring. BlueVenn has 23 API integrations with delivery systems such as specific email providers and builds new integrations as clients need them.

All BlueVenn features are delivered as part of a single package. Pricing is based on the number of sources and contacts, starting at $3,000 per month for two sources and 100,000 contacts. There is a separate fee for setting up the unified database, which can range from $50,000 to $300,000 or more depending on complexity. Clients can purchase the configured database management system if they want to run it for themselves. The company also offers a Software-as-a-Service version or hybrid system that is managed by BlueVenn on the client's own computers.  BluyeVenn has about 400 total clients of which about two dozen run the latest version of its system. It sells primarily to mid-size companies, which it defines as $25 million to $1 billion in revenue.

_____________________________________________________________________________

*The original SmartFocus was purchased in 2011 by Emailvision, which changed its own name to SmartFocus in 2013 and then sold the business (technology, clients, etc.) but kept the name for itself. If you’re really into trivia, SmartFocus began life in 1995 as Brann Viper, and BlueVenn is part of Blue Group Inc. which also owns a database marketing services agency called Blue Sheep. The good news is: this won't be on the final.

Wednesday, November 30, 2016

3 Insights to Help Build Your Unified Customer Database

The Customer Data Platform Institute (which is run by Raab Associates) on Monday published results of a survey we conducted in cooperation with MarTech Advisor. The goal was to assess the current state of customer data unification and, more important, to start exploring management practices that help companies create the rare-but-coveted single customer view.

You can download the full survey report here (registration required) and I’ve already written some analysis on the Institute blog . But it’s a rich set of data so this post will highlight some other helpful insights.

1. All central customer databases are not equal.

We asked several different questions whose answers depended in part on whether the respondent had a unified customer database. The percentage who said they did ranged from 14% to 72%:


I should stress that these answers all came from the same people and we only analyzed responses with answers to all questions.  And, although we didn’t test their mental states, I doubt a significant fraction had multiple personality disorders. One lesson is that the exact question really matters, which makes comparing answers across different surveys quite unreliable. But the more interesting insight is there are real differences in the degree of integration involved with sharing customer data.

You’ll notice the question with the fewest positive answers – “many systems connected through a shared customer database” describes a high level of integration.  It’s not just that data is loaded into a central database, but that systems are actually connected to a shared central database. Since context clearly matters, here is the actual question and other available answers:

 The other questions set a lower bar, referring to a “unified customer database” (33%), “central database (42%) and "central customer database” (57%). Those answers could include systems where data is copied into a central database but then used only for analysis. That is, they don’t imply connections or sharing with operational customer-facing systems. They also could describe situations where one primary system has all the data and thus functions as a central or unified database.

The 72% question covered an even broader set of possibilities because it only described how customer data is combined, not where those combinations take place. That is, the combinations could be happening in operational systems that share data directly: no central database is required or even implied.  Here are the exact options:


The same range of possibilities is reflected in answers about how people would use a single customer view. The most common answers are personalization and customer insights.  Those require little or no integration between operational systems and the central database, since personalization can easily be supported by periodically synchronizing a few data elements. It’s telling that consistent treatments ranks almost dead last – even though consistent experiences are often cited as the reason a central database is urgently required.


This array of options to describe the central customer database suggests a maturity model or deployment sequence.  It would start with limited unification by sharing data directly between systems (the most common approach, based on the stack question shown above), progress to a central database that assembles the data but doesn’t share it with the operational systems, and ultimately achieve the perfect bliss of unity, which, in martech terms, means all operational systems are using the shared database to execute customer interactions.  Purists might be troubled by these shades of gray, but they offer a practical path to salvation. In any case, it’s certainly important to keep these degrees in mind and clarify what anyone means when they talk about shared customer data or that single customer view.

2. You must have faith.

Hmm, a religious theme seems to be emerging.  I hadn’t intended that but maybe it’s appropriate. In any event, I’ve long argued that the real reason technologies like marketing automation and predictive modeling don’t get adopted more quickly are not the practical obstacles or lack of proven value, but lack of belief among managers that they are worthwhile. This doesn’t show up in surveys, which usually show things like budget, organization, and technology as the main obstacles. My logic has been that those are basically excuses: people would find the resources and overcome the organizational barriers if they felt the project were important enough.  So citing budgets and organizational constraints really means they see better uses for their limited resources.

The survey data supports my view nicely. Looking at everyone’s answers to a question about obstacles, the answers are rather muddled: budget is indeed the most commonly cited obstacle (41%), followed closely by the technical barrier of extracting data from source systems (39%). Then there’s a virtual tie among organizational roadblocks (31%), other priorities in IT (29%), other priorities in marketing (29%) and systems can’t use (29%). Not much of a pattern there.

But when you divide the respondents based on whether they think single customer view is important for over-all marketing success, a stark division emerges.  Budget and organization are the top two obstacles for people who don’t think the unified view is needed, while having systems that can extract and use the data are top two obstacles for people who do think it’s necessary for success. In other words, the people committed to unified data are focused on practical obstacles, while those who don’t are using the same objections they apply to everything else.


Not surprisingly, people who classify SCV as extremely important are more likely to actually have a database in place than people who consider it just very important, who in turn have more databases than people who consider it even less important or not important at all.  (In case you're wondering, each group accounts for roughly one-third of the total.)

The same split applies to what people would consider helpful in build in building a single customer view: people who consider the single view important are most interested in best practices, case studies, and planning assumptions – i.e., building a business case.  Those who think it’s unimportant ask for product information, vendor lists, and pricing. I find this particular split a bit puzzling, since you’d think people who don’t much care about a unified database would be least interested in the details of building one. A cynic might say they’re looking for excuses (cost is too high) but maybe they’re actually trying to find an easy solution so they can avoid a major investment.

Jumping ahead just a bit, the idea that SCV doubters are less engaged than believers also shows up in at the management tools they use.  People who rated SCV as extremely important were much more likely to use all the tools we asked about. Interestingly, the biggest gap is in use of value metrics. This could be read to mean that people become believers after they measure the value of a central database, or that people set up measurements after they decide they need to prove their beliefs. My theology is pretty rusty but surely there’s a standard debate about whether faith or action comes first.

Regardless of the exact reasons for the different attitudes, the fundamental insight here is that people who consider a single view important act quite differently from people who don’t. This means that if you’re trying to sell a customer database, either in your own company or as a vendor, you need to understand who falls into which category and address them in appropriate terms. And I guess a little prayer never hurt.

3. Tools matter.

We’ve already seen that believers have more databases and have more tools, so you won’t be surprised that using more tools correlates directly with having or planning a database.


Let's introduce the tools formally.  Here are the exact definitions we used and the percentage of people who said each was present in their organization:


Of course, the really interesting question isn’t which tools are most popular but which actually contribute (or at least correlate) with deploying a database. We looked at tool use for three groups: people with a database, people planning a database, and people with no such plans. 

Over all, results for the different tools were pretty similar: people who used each tool were much more likely to have a database and somewhat more likely to plan to build one. The pattern is a bit jumbled for Centers of Excellence and technology standards, but the numbers are small so the differences may not be significant. But it's still worth noting that Centers of Excellence are really tools to diffuse expertise in using marketing technology and don’t have too much to do with actually creating a customer database.

If you’re looking for a dog that didn’t bark, you might have expected companies using agile to be exceptionally likely to either have a database or be planning one. All quiet on that front: the numbers for agile look like numbers for long term planning and value metrics, adjusting for relative popularity. So agile is helpful but not a magic bullet.

What have we learned here? 

Clearly, we've learned that management tools are important and that long term planning in particular both the most common and the best predictor of success.

We also found that tools aren’t enough: managers need to be convinced that a unified customer view is important before they’ll invest in a database or tools to build it.

And, going back to the beginning, we saw that there are many forms of unified data, varying in how data is shared, where it’s stored, how it’s unified, and how it’s used. While it’s easy enough to assume that tight, real-time integration is needed to provide unified omni-channel customer experiences, many marketers would be satisfied with much less. I’d personally hope to see more but, as every good missionary knows, people move towards enlightenment in many small steps.