Explaining Martech to a Five Year Old

One piece of paper stands between me and ending the year with a clean desk: a list scribbled on hotel stationery that compares different customer data systems to different types of motor vehicles. Richard Scarry  meets CDP Institute, you might say.

So, just in case your New Year’s plan includes explaining marketing technology to a five year old, here we go:

• Data Warehouse = School Bus. The seats in a bus are lined up in nice neat rows, designed to carry one type of cargo very efficiently. Similarly, a data warehouse is highly structured environment that is very efficient at dealing with specified data types. (See how this works?)

• Data Lake = Moving Van. A moving van is a big box that can hold pretty much anything, although it might be jumbled together in no particular order. A data lake can store any type of data but doesn’t do much to organize it.

• Integration Platform = Delivery Motorcycle. A motorcycle carries small items quickly from one place to another, but doesn’t have any place to store them. An integration platform moves bits of data between systems but doesn’t have its own storage, either.

• Data Management Platform = Fire Truck. A fire truck is a highly specialized vehicle designed to move very quickly and pour out huge volumes of water. A DMP is a highly specialized system that quickly moves huge volumes of data.

• CRM System = Taxi. A taxi carries one person and their baggage directly to a specific destination. A CRM system delivers one customer and their history to a single sales person or call center agent.

• Cloud Platform = Car Carrier. A car carrier is a frame that can hold many unconnected vehicles. A Cloud Platform supports many unrelated systems in the same rack.

Believe it or not, my original list didn’t include an entry for Customer Data Platforms. So let’s give that a moment’s thought and go with…Ice Cream Truck!  They make a lot of noise and everybody loves them.  No, wait, better still...Food Truck!  A food truck collects ingredients from various sources, converts them into delicious meals, and distributes the results to many happy customers.  A CDP combines customer data into profiles that it shares with different systems.  

You’re welcome.

Happy New Year!

Acquia Buys AgilOne CDP

Acquia, which is moving past its roots in Web content management to become a multi-channel “digital experience platform” (DXP), took a big step in that direction today with a deal
to buy the AgilOne Customer Data Platform. The deal follows Acquia’s May 2019 purchase of open source marketing automation platform Mautic  and September 2019 purchase of site building tool Cohesion.  Acquia itself was purchased in September by Vista Equity Partners  for $1 billion, which obviously supports their DXP strategy.

The logic behind this deal is so clear that there’s little need for comment. While the exact meaning of DXP is a bit fuzzy, it surely involves coordinating and personalizing customer experiences across channels. This certainly requires the unified customer data that a CDP provides.  Acquia’s heritage in Web content management doesn’t provide deep customer data unification experience, and neither does Mautic.  AgilOne is a particularly good fit because it’s better than many CDPs at identity matching, including offline as well as online data. It also provides lots of connectors to source and delivery systems, as well as advanced machine learning for segmentation and predictions. Acquia and Mautic lacked those, too.

AgilOne rebuilt its core technology fairly recently, giving it a highly flexible and scalable platform that should easily extend beyond the company’s current base in mid-tier retail.  In particular, it will be able to serve Acquia’s clients, who tend to be very large companies with multiple Web multi-sites around the world. At the same time, AgilOne gives Acquia a stronger story in retail and other B2C markets where it has been less active.  AgilOne will also gain by integrating with some of Mautic’s features, notably email and SMS delivery and complex customer journey management. And the deal gives AgilOne much deeper resources to fund growth than it had as an independent company.

What, if anything, does the deal tell us about the larger CDP industry? I’d argue it mostly reinforces the trends I described in October, of independent CDPs being purchased by companies that are not primarily marketing software vendors but need to add customer data capabilities. Nearly all major CDP purchases to date meet this description: Mastercard buying SessionM, Dun & Bradstreet buying Lattice Engines, Arm buying Treasure Data, and Informatica buying Allsight. The only partial exception is Salesforce buying Datorama, but CDP wasn’t the focus of that deal. None of the other companies trying to follow Acquia’s approach of expanding from Web content management to DXP has yet purchased a CDP. But they’ll all need CDP functions so don’t be surprised to see more deals along those lines.

Put in a broader context, adding a CDP as a module inside a DXP is an example of CDP as a component within large marketing or even operational systems, something I refer to as “CDP Inside”. I expect that to be increasingly common and, thus, a potential home for independent CDP systems as the market matures, competition heats up, and the big marketing cloud vendors release their own products.  Selling or merging to become part of a larger system is one escape path for the independent CDPs. Another path is to focus on specific industries or cost-sensitive segments where the big marketing clouds are at a disadvantage.  I expect to see current CDP vendors take both approaches, even as new entrants continue to appear.  The CDP market won't get any simpler but buyers should have increasingly clear choices, so buying a CDP may become a bit less complicated.

What Happens When Everyone Has a CDP?

October 22 was a landmark day for the CDP industry.  We reported three significant announcements:

• Mastercard bought CDP and loyalty vendor SessionM which will power Mastercard services for its merchant clients
• Teradata launched its own CDP offering, Vantage CX, which will anchor Teradata’s marketing software product line
• Tealium partnered with marketing agency Wunderman Thompson, which will use Tealium AudienceStream to manage customer data for agency clients.

Each announcement follows a string of similar previous developments.

SessionM/Mastercard: most CDP acquisitions have been made by companies that are not primarily marketing software vendors. Arm (microprocessor technology) bought Treasure Data; Dun & Bradstreet (B2B data sales) bought Lattice Engines; Kabbage (small business finance) bought Radius; Anaplan (business planning) bought Mintigo; Informatica (data management) bought Allsight; Equifax (credit bureau) bought Datalicious. The exception that proves the rule is Salesforce’s purchase of Datorama, which it uses for marketing performance measurement, not as a CDP.

I believe the reason for these deals is that the buyers want to offer services that depend on unified customer data, but find it’s easier and cheaper to buy the necessary technology than to develop it internally. Note that it's truly a build/buy choice: many of the buyers already have extensive customer data management operations, so they probably could have built the systems for themselves.  They simply realized that buying was the better option.

The implications of this are substantial.  Competitors of the acquiring firms will feel pressure to offer similar services that help their clients deploy customer data.  Such services can be important tools for retaining clients, since switching customer database providers is painful at best. For CDP vendors, these deals are a promising exit path from a crowded industry which will only become more competeitive (see below).  For marketers, these deals mean their companies gain new options for access to a CDP.  This is especially true for small and mid-size businesses that might lack the resources to buy and integrate a CDP on their own.

Teradata: major marketing software vendors have chosen to build their own CDPs rather than buying one. This list includes Salesforce, Adobe, Oracle, Microsoft, IBM, SAP (sort of) and SAS (although they don’t use the term).  Their decisions to build their own CDPs are a bit perplexing, given that most have made many other acquisitions to fill gaps in their product lines.  My best guess is they like to buy companies that give them a substantial position in a new market, and even the largest of the pure-play CDP vendors are too small to catch their fancy. It might also be that building a CDP looked simple to them, so they all decided there’s not much reason to purchase a CDP for technology alone. The time it has taken them to deliver proper CDPs suggests it may have been harder than they thought.

The marketing software vendors’ delay in delivering CDPs has given other vendors opportunities that might not have existed had the marketing software companies moved more quickly. But with the marketing software vendors products now finally reaching the market, that era is ending. This will make life more difficult for the independent CDP vendors.  I still expect many of the independents will survive by developing systems tailored to particular industries, regions, or client sizes.

Wunderman Thompson: many ad agencies have decided to partner with a CDP vendor rather than purchasing one outright. The analysis gets a little confusing here because the big ad holding companies have been purchasing data-based marketing agencies: Dentsu bought Merkle, IPG bought Acxiom, Publicis bought Epsilon. But those agencies have themselves generally resold marketing technology rather than building or buying their own. This is probably a good choice: although they have considerable skill working with customer data, they have limited software development capacity. So it makes more sense for them to rent technology from others.

Assuming they continue to work with other vendors' technologies, the agencies represent a market for CDP vendors that won’t go away. If anything, it’s likely to grow as more agencies offer customer data-based services. But agencies have special needs and are often very cost-sensitive. So only a handful of CDP vendors are likely to get much benefit.

These three lines of development all point in the same direction. The path leads to a world where unified, sharable customer data is available to nearly every organization: that is, a world where every company has a CDP.  Nirvana, you say?  Yes, possibly, for CDP users.  But remember that CDP might be a stand-alone system, part of a marketing software suite, embedded in operational systems, or provided as part of an agency’s service. So it's not necessarily great news for CDP sellers.  

The broad availability of CDP functions affects users in other ways. When CDP functionality was available only from specialist vendors, the choice of a CDP was based on finding the best system (or, more precisely, the system that best fit each buyer’s particular requirements). But when CDPs are baked into larger software and service offerings, the quality of the embedded CDP is one of many considerations in selecting a vendor. In fact, the CDP itself may be invisible, as buyers base their choice on which vendor can best meet their business needs. If the potential vendor can meet those needs, its CDP must be adequate. If the potential vendor isn’t a good fit, it really doesn’t matter whether the fault lies with their CDP or some other component.

Note that there will be exceptions to this new rule. Large enterprises are likely to assemble their own collection of best-of-breed components, including a stand-alone CDP to integrate data from disparate sources. Mid-size firms who don't want to commit to one comprehensive marketing suite may prefer broad-scope CDPs that combine centralized data, analytic and personalization functions while integrating with external delivery systems.

What doesn’t change are the needs for users to define their requirements, to accurately assess which vendors will meet them, and to deploy their choice effectively. These tasks are closely related: you can’t define requirements, assess alternatives, or deploy effectively without understanding what the CDP needs to do. So education and training of CDP users will remain important regardless of how CDPs are purchased or delivered.

Another way to look at it is this: CDP has been cruising through the Gartner Hype Cycle for the past four years. Each hype cycle stage implies a common customer question*.  Here's what we’ve seen for CDP:

• at the start, when the technology is unfamiliar, the question was: What’s a CDP?
• during the peak of inflated expectations, people had some understanding of the concept, so their question became: Do I need a CDP?
• as CDP enters the final stages of disillusionment, enlightenment, and productivity, people accept that they probably need a CDP and ask the next logical question: How do I best use my CDP?

Of course, different markets and individual users are at different stages in their own CDP journey, so we still get all three questions. But it’s clear that the third question – often phrased in requests for use cases, best practices, and maturity models – is becoming the most important. I’ve no doubt that the industry will provide more answers.

____________________________________________________________________________
*This is my interpretation, not Gartner’s.

Privacy Regulations Will Lead to Advertising Innovation

Naysayers doubted that the European Union’s General Data Protection Regulation (GDPR) and ePrivacy rules, California Consumer Privacy Act (CCPA) and related privacy regulations would have any real impact on the flow of consumer data. But it’s now clear that they will, as European regulators show they will interpret the rules to require meaningful consent to data sharing, will treat cookies as personal identifiers, and will assess significant fines for data breaches. The imminent effective date of CCPA, which rather surprisingly survived without being watered down or preempted by weaker federal regulation, confirms that privacy can not longer be ignored by data collectors. Meanwhile, actions by Google, Microsoft, Apple and other major browser are putting still more barriers in the way of data business as usual.

None of this marks the end of the advertising industry or even of targeted online advertising. Advertising was a big and important business before the Internet existed and would remain one if the Internet went dark tomorrow. Pre-Internet advertising wasn’t as targeted or measurable as today’s display and social ads, but it did work. A total end to distribution of third party personal data would still leave Internet marketers able to target based on context, content, and in-session behaviors. This probably wouldn’t be quite as effective as individual-level targeting, although I don’t recall any studies that prove this.  It’s even possible that closer attention to ad targeting methods would reduce fraud enough to more than compensate for the loss of third party data.

In any case, the hunt is on for mass advertising audiences to replace audiences based on third party personal data. The recent merger of content-based advertising leaders Taboola and Outbrain was driven in part by the desire for greater scale in content ads (those click-bait teasers that lead you off-site at the bottom of many Web pages). The rebirth of out-of-home advertising (billboards, wall posters, in-store displays, and clever niche products such as elevator and gas pump ads) as a digital channel opens another mass medium. Above all, the various forms of individually targeted TV advertising promise new levels of personalization and tracking in a medium that already has near-universal reach. Other types of video, available through YouTube, Instagram, TikTok, Snapchat, Twitch, and much else, are delivering new mass.

Taken together, these emerging options promise a new Golden Age of Advertising Creativity, as marketers explore their potential and evolve the most effective approaches to each. It’s true that a vastly more fragmented media landscape will be harder for marketers to manage. But it also means that fears of a Google/Facebook duopoly controlling all access to new customers are clearly overblown. Regulatory pressures, changing consumer attitudes, and new competition should further deflate them every day.

What does all this mean for martech in general and Customer Data Platforms in particular? The most directly affected martech systems are Data Management Platforms (DMPs), whose core function is to manage the third party data-based customer profiles that are quickly becoming extinct. Marketers who had hoped the DMP would manage all their customer data had already lost much of their interest when they discovered how limited DMP capabilities really were.  Many DMPs have repositioned themselves as CDPs, although not all have the technical capabilities required to meet the RealCDP requirements.*

The implications for CDPs are more positive, although not quite as rosy as sometimes suggested. There’s a common argument, which I frequently make myself, that the loss of third party data makes first party data more important, and that’s good for CDPs because they primarily manage first party data. This is true on some level but shouldn’t be overstated. The third party data that populates DMPs is mostly used for acquisition marketing, while the first party data in CDPs is mostly for marketing to current or former customers. So switching focus from DMP to CDP would leave a significant gap in many acquisition marketing programs.

As we’ve just seen, marketers will plenty of other ways to reach new audiences even after today’s third party data-driven advertising is gone. But there is one way that CDPs can directly support acquisition programs: by making it easier for companies to share first party data with each other, creating what is usually called second party data.

Second party data can refer to any first party data that is directly shared with another company (the second party). The CDP supports this by creating an extract file of first party data that can be sent to the second party. If that’s all that happens, it’s no different from any other extract.

But often second party data is created by comparing the customer lists of both parties and finding shared customers. The trick is often that the two parties don’t want to expose information about customers they don’t share.

One way to achieve this is to send a copy of both customer lists to another company that both firms trust. That company compares the two lists, finds matches, and returns whatever information the parties have agreed to share. Alternatively, a common identifier such as email address might be run through a standard “hashing” algorithm that will yield a unique result for each input but not reveal the actual identity information. Both parties use the same algorithm so that all records with the same identifier yield the same hash code and are identified as a match. Records that don’t match will generate different hash codes which are meaningless to the other party. With this sort of processing, there’s no need for another trusted company to do the matching because customer identifiers are not actually shared. Each party keeps a record of the original identifier and the resulting hash code, so it can tie the codes back to its actual customer records.

Here’s a practical example. Suppose a hotel chain and airline agree to identify loyalty program members on each others' lists.  That is, the airline would learn which of its customers were members of the hotel loyalty program and the hotel would learn which of its customers were members of the airline program. They can do this with the type of matching just described, adding a "partner loyalty member" flag to appropriate customer profiles.  The airline could then make a special offer to hotel loyalty members and the hotel could make a special offer to  airline program members. Each party could also send acquisition promotions on behalf of its partner to its own customers who are not  members of the partner's loyalty program. So long as each party sends the promotions to its own members, the other party never learns anything about them.

The minimum role of the CDP in this sort of second party sharing is to generate a customer list with the required data. Some CDPs can also find direct matches (acting as the trusted third company) or generate and match the hashed identifiers. CDPs with these added functions are in a position to benefit as the loss of third party data makes second party data sharing more important to acquisition marketing programs.

In sum, privacy regulations won't kill targeted marketing.  They'll actually lead marketers to expand the methods they use, promoting a healthy diversity and weakening the much-disliked Google/Facebook duopoly.  Customer Data Platforms will benefit as first and second party data play larger roles in the marketing mix. 

_____________________________________________________________________________
*load all types of data, retain all original details, store the data indefinitely, build unified profiles, and expose the profiles to any external system.

It's CDP Time for Marketing Cloud Vendors

Adobe, Salesforce and Oracle all made announcements regarding Customer Data Platform products this week. None are world-changing: Salesforce described a planned extension of Customer 360; Abobe announced triggered journey campaigns that draw on its “real time CDP”, and Oracle described CDP services from systems integrators.  But the fact that all three vendors are addressing the topic raises some interesting questions.

Why does this matter to technology and marketing professionals? Customer Data Platforms have been a hot topic for the past three years. The reason is simple: they promise to solve a pressing problem that has not been solved by anything else. That problem is the need of marketers (and others) to combine data from all sources into easily accessible customer profiles. Those profiles are needed for accurate targeting and consistent, satisfying customer experiences.

The problem is unsolved because alternate solutions fall short in different areas.  Data warehouses are largely limited to structured data.  Data lakes are not unified or easily accessible to non-technical users.  Data Management Platforms are limited to summary data about mostly anonymous individuals.  CRM and marketing automation systems don’t easily combine data from external sources.

By contrast, CDPs promise to ingest all data sources, retain all details, and share the resulting profiles with any system that needs them.  But CDPs have been developed by relatively small specialist vendors, which many enterprise buyers are reluctant to consider. So having Salesforce, Adobe, and Oracle promote CDPs will prompt more enterprise buyers to give the category serious consideration.

It also doesn’t hurt that Forrester issued its first CDP wave this week, that Dun & Bradstreet just bought CDP Lattice Engines, and that Martech Advisor’s Talking Stack podcast devoted an entire session to the topic  (humble-brag disclosure: I’m a panelist on Talking Stack).

So, yeah, CDP is getting a lot of attention right now.

What’s new in these announcements and what’s not? Adobe, Salesforce and Oracle have all previously announced something that they positioned as addressing the needs met by a CDP.

• Adobe’s original approach was to map a single customer ID across all its systems and create transient customer profiles by pulling together data on demand. It changed direction in March 2018 with news that its Experience Platform would create persistent profiles.  It released this a year later and described it as including a "Real-Time CDP".  Nothing about that changed with yesterday’s announcement, which described an Adobe Campaign application using the CDP data.  All this is separate from the Open Database Initiative, a still-undelivered project announced last September to build shared database model with Microsoft and SAP.

• Oracle announced CX Unity last October. It included a persistent data store from the start but is just now starting beta deployments. The latest Oracle announcement describes collaboration with Accenture and Capgemini to provide services around CX Unity projects.  It doesn’t include anything new about the product.


• Salesforce’s Customer 360, also announced last September, was another master customer ID used to access source system data on demand. Salesforce now says they’ll release that product this November. The bigger news is they’re developing a next-generation Customer 360 that will store its own data, bringing them in line with Adobe and Oracle. There’s no release date although they hope to start pilot projects this fall.

It's taken a while but all three vendors now acknowledge that a CDP must store its own data. That will remove some confusion from the CDP market.  Cynics might say it also confirms that software vendors define user requirements based on what their systems currently do, not what users actually need. It’s hard to interpret the CDP story in any other way, since the need for a persistent data store has always been clear to anyone who tried to support core CDP use cases such as attribution, prediction, and journey analysis.

What benchmarks should these CDP products be measured against? The CDP Institute (which I head) believes that buyers expect a CDP to ingest all types of data, capture the full detail of the ingested data, retain that data as long as the user wants, assemble unified customer profiles, and make the profiles available to any external system. We codify those as requirements in our RealCDP^TM certification program. The original Adobe and Salesforce approaches certainly didn’t store the data and had other limits about the types and detail they captured. On the other hand, they did include real-time access to current source data, which is not part of RealCDP but is important for many CDP use cases. Many other CDPs also provide real-time access as well as segmentation, predictive modeling, personalized message creation, and sometimes even message distribution. Oracle, Salesforce, and Adobe have separate products for most of those functions so they are not part of their CDPs.

What should we look for next from these vendors? The main thing to watch is how quickly the announcements turn into released products. Only then will buyers be able to judge the details of how well these vendors deliver on their promises and how their offerings compare with other, more mature CDPs. In particular, pay close attention to how easily each system connects with other products. Because the marketing clouds are built from acquisitions that remain technically separate, integrations for each component are developed and delivered in whatever sequence the vendor thinks best.  Connections to other vendors’ systems are an even lower priority. Buyers will often be left to develop their own connectors using a CDP API – which should itself be examined closely to see what it does and how hard it is to use.

Once the vendors flesh out the core CDP capabilities of their offerings, the focus will shift to improvements in user experience. This includes end-user functions such as segmentation and more technical capabilities for connecting to data sources and destinations. Reducing the technical work required to set up and maintain a CDP can have a significant impact on time to value and operating cost.  More important, it can help the CDP achieve its mission of including all data and sharing it with all other systems. Also expect the vendors to create industry-specific packages with prebuilt data models, connectors, predictive models, workflows, and reporting. These will further ease deployment and help the vendors to compete with industry-specialist CDPs.

In most of the CDP market, vendors are extending their systems by adding more activation functions such as marketing campaigns, personalized message selection, and message delivery. Many users are eager to buy one system that combines as many functions as possible. But because the marketing clouds offer these functions as separate modules, they are unlikely to expand their CDPs in that direction.  This will probably reduce their competitiveness in the mid-market.

Who else might toss their hat into the CDP ring? There have been four significant CDP acquisitions to date: Datalicious/Veda by Equifax (2016); Datorama by Salesforce (2018); Treasure Data by Arm (2018), and Lattice Engines by Dun & Bradstreet (2019). Salesforce is the only marketing cloud vendor on this list and they've positioned Datorama as a campaign analytics product, not a CDP.  Since Oracle and Adobe are far along in their CDP development, neither is likely to purchase an independent CDP vendor – although that might happen if they feel pressure to deliver a mature CDP more quickly. That Equifax and Dun & Bradstreet have bought CDPs suggests other data compilers might do the same, with the goal of adding more value than just their data.

Other potential acquirers include ad agency holding groups and consultancies who want to bulk up their data management capabilities. Moving from the other direction, companies that offer message delivery and interactions, such as email delivery, Web site personalization, mobile app development, call center, and customer support systems, might add a CDP to expand their footprint, make their products harder to replace, and ultimately let them add delivery in other channels. Case in point: unified marketing, sales and support vendor Freshworks recently purchased Natero, a customer success system with CDP capabilities.

But the big clouds looming over the entire customer data ecosystem are, literally, the big clouds: Amazon Web Services, Google Cloud, and Microsoft Azure. So far, none has shown much interest in selling customer data management tools, although they already host plenty of CDP data.  There’s a reasonable chance these vendors will eventually enter the CDP market.  But that would be a few steps up the value chain from their current offerings, which still focus primarily on data storage. The cloud services vendors are starting that climb, adding cloud connectors, data transformations, in-database analytics, machine learning, and reporting tools. Google Cloud’s purchase of Looker is a recent step in this direction. Data quality services such as address standardization and master data management could be next.  There's a way to go before these vendors are ready to add the specialized features needed for a CDP. They may also find themselves constrained by anti-trust and privacy issues. But don’t rule it out.

What do these announcements mean for current CDP vendors? At a fundamental level, these announcements confirm that there’s a real need for CDPs and that the CDP model of building a separate, persistent database is correct. That’s no small thing, given the fear, uncertainty and doubt that the marketing cloud vendors have previously spread about both. The result is to expand the market, since more potential buyers will now accept CDP as a valid option.

This does not mean the cloud vendors have given up their efforts to shape the conversation. They are now trying to redefine CDP as part of a larger integrated package – that is, of systems like their own. This isn’t likely to be successful, given how few enterprises actually limit themselves to one vendor’s products. So the ability of the current CDP vendors to work equally well with systems from any provider is likely to be even more appealing.

Still, it would be unrealistic to ignore the fact that many buyers would rather buy from the marketing cloud vendors. The long lead times between preannouncement of the marketing cloud CDPs and their actual delivery will lead some buyers to delay their purchases, which is exactly the cloud vendors’ intent. But the interval will also give potential buyers more time to think carefully about what they need in a CDP, making them smarter consumers when they do start an acquisition project. A rigorous, requirements-based acquisition process will often favor the existing CDP vendors, whose products are proven and mature.

How do these announcements relate to other developments in the CDP market? The CDP market is evolving rapidly. From its initial base in retail and media, it has spread to new industries including travel, financial services, B2B, telecommunications, healthcare, and education. Industry-specialist vendors are appearing with prebuilt data models, integrations with industry systems (such as point of sale in retail or reservations for hospitality), and staff expertise. Other specialists now focus on mid-size or small businesses and in particular geographic regions.

There’s also an increasingly clear split between CDP vendors who focus on data collection, unification, and access functions and those with marketing functions for analytics and personalization. One result is that some clients deploy one CDP for data unification and another for marketing applications. Some CDPs have extended their marketing features to include delivery systems such as email engines, Web site messaging, and even DMPs.  These were previously beyond the scope of CDP products. Vendors with delivery capabilities still allow clients to use other delivery systems – otherwise they would not meet the definition of a CDP. You can see where this leads: CDPs with a full set of marketing applications become direct competitors of the marketing clouds. It also means that CDP becomes one feature among many in these products -- a change that may lead some to deemphasize their CDP capabilities and instead partner with data unification specialists.  (See this post for a deeper exploration of that scenario.)

On the buyer side, CDPs are increasingly used beyond marketing to support sales, service, and business operations. This often means the CDP becomes a shared resource managed by corporate IT rather than marketing.  Enterprise-wide digital transformation projects and increasing concern over compliance with privacy regulations have further increased involvement of central IT and compliance teams.  In general, greater familiarity with CDPs has meant that buyers have a better understanding of what they do and what to look for, making them more sophisticated consumers and helping them to navigate the growing number of products in the market.

Entry of the big cloud vendors may accelerate the trend to industry-specific CDPs by pushing smaller vendors into niches where they can better compete with general purpose systems. The big cloud vendors may push also independent CDPs to deliver comprehensive marketing functions to the mid-market where they can be more integrated and more economical than the marketing clouds. CDPs that focus primarily on data management will probably face the greatest competition from the marketing cloud CDPs, since they compete for enterprise clients where the big cloud vendors are the strongest. But those CDP vendors also have the greatest technical lead and will find it easiest to support enterprise-wide use cases.

In short, entry of the marketing cloud CDPs will accelerate industry growth and reinforce current industry trends, not change the over-all direction. For CDP vendors and buyers alike, that is good news.

Tableau, Looker, and Origami Logic Acquisitions Show Analytics Is In Fashion

One of the unwritten laws of punditry is that one event is random, two events are interesting, and three events make a trend. By that measure, the purchases of data analytics vendors Looker by Google, Tableau by Salesforce, and Origami Logic by Intuit within a three week span must signify something. Is it that martech suites must now include business intelligence software?

I think not. Even though the acquired products were fairly similar, each of these deals had a different motivation. Conveniently, the buyers all stated their purposes quite clearly in their announcements.

- Intuit bought Origami Logic to advance its strategy to become an “A.I.-driven expert platform”. Specifically, they see Origami Logic as providing a “strong data architecture” that will “accelerate Intuit’s ability to organize, understand, and use data to deliver personalized insights that help customers quickly achieve success and build confidence whenever they use Intuit products.” Reading between the lines, Intuit recognized its existing data architecture can’t support the kinds of analysis needed to generate AI-based recommendations for its clients, and bought Origami Logic to close that technology gap. In other words, Origami Logic will be the foundation of new Intuit products.

- Google bought Looker “to provide customers with a more comprehensive analytics solution — from ingesting and integrating data to gain insights, to embedded analytics and visualizations — enabling enterprises to leverage the power of analytics, machine learning and AI.” That’s somewhat similar to Intuit’s purpose, but Looker is building applications on top of Google Cloud’s existing, extremely powerful data management capabilities rather than providing a new data management foundation. Indeed, Looker already runs on Google Cloud. So Looker is adding another layer of value to Google Cloud, letting it meet more needs of its existing clients.

- Salesforce bought Tableau so it can “play an even greater role in driving digital transformation, enabling companies around the world to tap into data across their entire business and surface deeper insights to make smarter decisions, drive intelligent, connected customer experiences and accelerate innovation”. That’s not exactly pithy, but we’re dealing with Marc Benioff. The key is digital transformation, which lets Salesforce participate in projects beyond its current base in sales and marketing departments. That is, the purpose isn’t to add products for existing customers but to serve entirely new customers. The huge size of Tableau’s customer community – “more than 1 million passionate data enthusiasts” -- clearly a draw for Salesforce. This makes complete sense for Salesforce, which is always straining to maintain its growth rate.

Is there some commonality here? Sure: each of these vendors is striving to offer products based on advanced data management and analytics. Intuit is focused on the data management foundation while Google Cloud and Salesforce are focused more on analytics. All are acknowledging that it’s easier to buy mature technology than to build it from scratch. But of the three buyers, only Salesforce is a martech vendor and their purpose is explicitly to serve customers outside the martech user base. So whatever these deals prove, it’s not that business intelligence is the latest martech must-have.

What's Next for the Customer Data Platform Industry?

Customer success platform Gainsight recently released a Customer Data Platform module for its system. This followed Freshworks’ acquisition of Natero, another customer success platform that builds a unified customer database. Totango, another customer success platform, also positions itself as CDP. 

There's nothing questionable about these claims.  Customer success platforms were among the earliest classes of systems identified as CDPs. Like predictive modeling systems, they originally built a unified database to support their primary application and later recognized that the database had even more value if it was shared with other systems. So a CDP positioning makes sense.

But the Gainsight announcement did prompt an industry colleague to ask me whether classifying products like Gainsight as CDPs would lead to each department building its own CDP, creating “CDP silos” and ultimately requiring a “CDP of CDPs” to unify them all.  That contains enough irony to meet your minimum daily requirement for a week.  But it’s a fair question that’s often posed by CDP buyers who don't want another silo in their server farm. 

My first answer is that not all systems claiming to have a CDP are true CDPs, particularly when it comes to sharing their data with any external system. We’ve been trying clarify that with the RealCDP project, which includes data sharing as one of the five qualification requirements.  But many vendors like Gainsight and Totango pass that test. So it doesn’t really address the question.

My second answer is to respond with another question: If a system meets the definition of a CDP, should we NOT call it one?  That seems silly. Of course, it begs the question of the CDP definition itself: should we define a CDP as a system that only builds the unified database?  This would exclude products that also provide integrated applications such as customer success management, predictive modeling, campaign management, personalization, and more.  The narrow definition would certainly reduce confusion.  But it doesn’t address the real-word situation, which is that many buyers want a CDP that includes some or even all of those applications.  My response has been to suggest the “CDP Inside” concept, which says CDP is a function, not a software category.  This function could be provided by a single-purpose system or by a system that offers other functions as well.  The concept  hasn’t been as widely adopted as I'd like but I think the world is moving in that direction regardless.

My third answer builds on the second.  Companies can buy multiple products with a CDP capability and still not deploy all of them.  This comes down to management: a company with several CDP-capable products can make one its primary CDP and push profiles from that system to the rest.  That would be the best choice in most cases, I think.  But a company might also build several complete CDPs by feeding the same data into each of them.  Or it might build siloed CDPs that only connect to their associated applications. Those sound like bad ideas. But it’s not the CDP's fault if someone deploys it ineffectively.

Let’s game this out. 

• Application vendors are incented to add CDP capabilities to their systems because some clients want them. Foolish clients may build several separate CDPs, especially in organizations where different groups prefer to function independently. So, yes, there’s some potential harm in making that easy by putting a possible CDP inside each system. 

• Smart clients will avoid this mistake by choosing one system as their primary CDP. Each vendor will hope to play that role, since it adds value to their product and makes clients less likely to switch to another system.

• Over time, the smart clients will learn what they need in a CDP and push vendors to compete to build the best product. As this happens, vendors with limited resources will drop out of the competition to focus on their core applications.  They'll still provide some data unification features but won't promote them as a CDP. 

• Other vendors will double down on their CDP investment so they can win the competition. As these vendors develop best-in-class products, they’re likely to offer their CDPs as separate modules. 

• The most demanding buyers will want the best possible CDP and will buy best-in-class products whether they come from specialist CDP vendors or application developers.  As a result, best-in-class CDPs will be increasingly distinguished from lightweight CDPs embedded in application systems.

A reasonable analogy might be reporting systems. In the early days of the packaged software industry, application vendors competed to offer the best custom report builders within their products. Later, general-purpose reporting tools like Tableau came to dominate the market. Most application vendors then stopped presenting their report builder as a differentiator. They still deliver specialized reports as part of their systems but expect users to integrate with third party reporting tools for other needs.

I think the CDP industry will follow a similar trajectory. Many applications need unified customer profiles. Some will rely entirely on an external CDP to create those profiles and share them. Most will offer a lightweight CDP so customers without a separate CDP can still use their application. A few will build best-in-class CDPs that can be sold on their own. These best-in-class CDP modules will often be spun off as separate products.

Some of this is already happening.* CDP vendors themselves are increasingly distinguishing between CDPs that focus on building and sharing the unified database and CDPs that also deliver analytical, personalization, and other application capabilities. Some in the latter group position their CDP features as best-in-class but most will admit (with varying degrees of reluctance) that they are often deployed in combination with a separate CDP that specializes in data unification. So, just as companies often have a shared standard reporting tool that complements the reports built into applications, we can expect to see one shared CDP that complements specialist CDPs within applications.

It will take some time to sort this out, as vendors decide what kind of CDP to offer and buyers decide what they need. But I think the future of the industry is coming into focus.


__________________________________________________
*A little-cited corollary to the famous aphorism that predictions about the future are hard is that predicting the past is easy.

Do Customer Data Platforms Need Real-Time Processing?

Is real-time processing a requirement for a Customer Data Platform? It’s a deceptively simple question that can’t be answered without resolving two additional questions: What do we mean by real-time processing? How do we decide what is and isn’t a requirement?

Let’s tackle the definition of real time first. There are at least four different flavors of real time processing that relate to CDPs.

• Real-time updates. This means that new data flows into the CDP and is added to the exposed customer profiles in a few seconds. The exact speed requirement isn’t clear: although one second is widely considered to be the threshold for real time interactions (based on a study done in 1968), longer lags are acceptable for sharing customer data between systems. In practical terms, customers will not be annoyed if it takes a half-minute before a call center agent learns about her latest Web interaction.  Many CDP vendors offer what they call “near-real-time” updates, where the lag might be one to five minutes. This is obviously too slow to manage an interaction like online chat. But it can support asynchronous activities such as sending order confirmation emails.

Real-time updates are quite difficult. The first step, adding a new piece of data into the CDP data store, isn’t too hard for most CDPs, which simply append each item without touching existing data. But there’s usually additional processing to update customer profile elements, such as lifetime purchase value, predictive model scores, or last purchase date. This requires finding the right profile, reading the existing data, running whatever calculation is needed, and storing the new result. That takes time. And there’s often an additional step to copy data from the profile into a separate data store that’s optimized for real-time access, such as a flat file or in-memory database. This takes time too.

• Real-time identity resolution. This is a special case of real-time updates, where the system needs to do some processing to associate a piece of data with the right customer profile. It's needed when the new data isn’t already associated with a known identifier such as a customer ID.  It may require substantial processing to run matching algorithms that test various combinations of data against existing records. Usually this process is limited to adding the data to an existing profile, but it might also re-examine all profiles to see if the new data uncovers a connection that would allow them to be merged or split. That requires still more processing. This sort of comprehensive reexamination is usually done in a batch process that might run anything from nightly to monthly.

• Real-time access. This means an external system can query the CDP in real time for a copy of an existing customer profile. This might be done with an API call or by accessing a separate data store. Note that the profile itself isn’t necessarily being updated in real time: in fact, companies commonly run a nightly batch process that loads profiles and next-best-actions into an online data store. The data remains unchanged until the next nightly update.

• Real-time interactions. This means an external system can send data to the CDP and receive back a profile that incorporates that data – for example, with an updated model score, product recommendation, or marketing messages. It’s another flavor of real-time updates but is limited to activities within a single system. In practice, real-time interactions often work by moving a copy of the profile into memory at the start of an interaction, updating the in-memory copy as the interaction proceeds, and then copying the final version of the profile back into the main database after the transaction is over. This avoids the need for real-time updates of the main database. Real-time interactions are often integrated with multi-step campaign flows so the user can guide the interaction process.

It’s important to recognize these distinctions and to clarify which are included when you’re discussing a real-time CDP. All systems in the CDP Institute’s Vendor Comparison Report say they do real-time access while only half do real-time interactions. Real-time data loads are almost universally available but the report doesn’t capture which systems can make the data available in real time. Nor do we ask about real-time identity resolution.

This brings us to the second question, of how to decide whether the CDP definition should include real-time capabilities (and, if so, which ones).  This matters because the market is confused by the variety of companies claiming to be a CDP.  A clear, widely understood definition is essential to reducing that confusion.

One approach is to start with current user expectations. CDP Institute research shows that users’ top priorities are data collection and unification. Real-time access and real-time recommendations rank far down their list. We can’t tell whether users include real-time updates in their definition of data collection.  My suspicion is that most do not. But I haven’t seen any reliable research.

A second approach is to look at what’s included in existing CDP systems. As we’ve already seen, all systems in our Vendor Comparison Report offer real-time access and about half offer real-time interactions. While we don’t ask directly about real-time updates, we do know that quite a few offer near-real-time updates, which presumably means they don’t do full-real-time updates. You may notice there’s some circular logic here, since we have to decide in advance which systems to examine.

A third approach is to look at common use cases. The data are again ambiguous because top applications like predictive modeling and personalization can benefit from real-time updates but don’t require it. Some standard CDP use cases do require real-time updates, such as providing call center agents with information about recent Web behaviors. But plenty of others do not, ranging from customer journey analysis to email audience selection.

Since none of these options offers much guidance, how do we decide?  I'll argue that we ultimately want to make the decision that best serves CDP users.  Many users need real-time updates and interactions but many others do not. If we want the CDP category to include systems that meet the needs of both groups, we should include non-real-time systems in the category.

But that's not enough.  We also need to help users understand which CDPs provide real-time capabilities and which specific capabilities are included.  CDP Institute is addressing this need with its RealCDP program, which will gather and present information about real-time capabilities.  We’ll also add real-time updates to the Vendor Comparison report. Stay tuned for more information.

Jamie's Excellent Privacy Adventure

Jamie knew something was wrong when his alarm clock didn’t greet him by name. Weird, he thought.

Things quickly got weirder.

The water in the shower was too hot.  The traffic report mentioned an accident nowhere near his route to work. When the radio played a song he had specifically banned from his play list, he knew something serious was wrong.

“Clock, please run a system check,” Jamie muttered, annoyed.

Silence.

He said it again, louder, this time staring directly at the device on his night table.

Silence.

System must be down, he thought. I’ll deal with it later.

But then he looked out the window. The cars all had human drivers and pedestrians were not staring into their phones. He knew he had a much bigger problem.

A multiverse flip.

He'd read about it on the news.  Result of global warming or wireless signal overload or overlapping artificial realities. No one really knew.  What they did know was that people woke up in a different world from the one they’d fallen asleep in.  It seemed to be happening more often. 

Jamie picked up his phone. It didn’t recognize him but on-screen instructions let him unlock it with a thumbprint. He dialed LOST – League of Strayed Travelers – a service that spanned multiverses with enough cross-traffic to cooperate. He was lucky; the call was answered. By a human, which felt odd. But apparently that was the world he was in.

A quick conversation established that Jamie was indeed lucky: the local LOST staff included someone from his home universe. She would be Jamie’s guide. Soon a car pulled up – this one with both a driver and passenger.  Out hopped Emily, a cheerful young woman who quickly began explaining what was different.

“This is a world where privacy has been taken very seriously. Businesses and government are banned from storing any personal information beyond what’s needed for security purposes. That’s why your smartphone recognized your thumbprint. But look more closely at the phone and you’ll see it doesn’t have a record of your past calls or contacts. In the same way, we still have search engines and social networks and ecommerce, but they don’t personalize their services. It’s annoying but you get used to it. What I miss more is voice recognition, which is entirely forbidden as inherently invasive. Alexa was my best friend!”

She glanced wistfully at Jamie’s alarm clock, which he now realized was no smarter than a rock.

“But it gets much worse,” Emily continued. “Without gathering personal data, companies can’t train AI systems for things like self-driving cars, energy conservation, or personalized medicine. So we have more accidents, more pollution, and a vastly less efficient economy. Everyone has less free time and is poorer. Crime is worse because people have to carry cash and video surveillance is strictly limited. And, ironically, social media is still filled with bullying and misinformation – it turns out those have nothing to do with privacy and everything to do with human nature.”

Emily frowned briefly. Then her face brightened.

“But there’s good news, too. We’ve been researching the multiverse flips and have experimental devices that can move you from one world to another. We can’t guarantee where you’ll end up but have enough return traffic to be reasonably sure it won’t be too terrible. At least, most of a time. So there’s a risk but we can give it a shot if you like.”

On the ride back to the LOST office, Jamie and Emily chatted more about this universe, the flipping device, and their previous lives. Turned out she was from New Jersey. And married. By the time they arrived, Jamie had decided to try the new machine.

Emily gave Jamie one final smile as she closed the lid on the flipper. It was dark and warm and filled with white noise.

Then the lid was up.  Jamie was in a different room. New faces. No Emily.

A serious-looking man reached into the flipper and pulled out a package. “May I have this? It’s how we share information across the multiverses.”

The man opened the package and scanned the top document. “I see you’re Jamie. Welcome. I’m Giovanni.”

Giovanni took a closer look at the documents. “Looks like you come from a place where anything goes, privacy-wise. It’s very different here. We believe that people own their own data. But we respect their liberty, so any use is allowed if they give consent.”

Jamie was a bit groggy as he stepped from the flipper. “How’s that working out for you?”

Pretending not to notice that Jamie looked tired, Giovanni smiled and pointed to a seat. “If you please.”  His face turned serious.

“Mixed results, to tell the truth. Most people consent to pretty much everything. Their experience is much like yours – many free services but little privacy. Data breaches are common and much of the data is inaccurate. But the majority put up with it for the convenience and an occasional discount coupon.”

“And everyone else?” asked Jamie.

“There are two groups. Some people are privacy zealots, pure and simple – they won’t give up their data on principle. They pay extra for services that others get for free, and many services aren’t available to them at all.  They’re often left out of business and social events and have harder lives as a result.  Many live off the grid doing things crafting Faraday cage handbags. At least we don’t hear their complaints.” He did not seem amused at his own joke.

“The others are people with enough power and money that they can easily afford the cost of privacy-enhanced alternatives. They have staff or bots to maintain a social media presence without exposing their own data directly. They use special devices and software that hides their identity, regularly erases their data, and maintains separate personas for different purposes. They get the best of both worlds: privacy for themselves and convenience based on the data of others.”

Jamie was puzzled. “Why is privacy protection expensive? I’d think most of the solutions are based on software.  That should be nearly free to run for everyone once it’s built.”

“I thought so too,” sighed Giovanni. “But it’s a lizard-and-the-egg sort of thing: most people won’t pay even a little extra for privacy, especially if they’re rewarded for giving it up. So the market for privacy-enhanced systems is fairly small, which means manufacturers must charge a higher price per customer, which makes the market still smaller, which drives the prices still higher. It ends up as a luxury good.” Giovanni sighed again.

“Yes, I can see how that works out,” said Jamie. He thought for a moment. “Look, it’s nice to meet you but this isn’t my home universe. Can I go back into the flipper and try again?”

“Of course,” replied Giovanni. “With your permission.”

There was time for a quick lunch while Giovanni prepared another information package. Jamie climbed back into the flipper, relaxed for a moment, and the lid was up again. Another room. More new faces. .

Now a veteran, Jamie sat up and handed the information packet to the nearest person. A badge clipped to his shirt showed his name was Tim.

“Hi Tim. I’m Jamie. Where am I this time?”

Tim opened the package and read the cover sheet. He paused a moment.

“Not where you want to be, I’m afraid. But not a bad place. How much do you want to know?”

Jamie was disappointed but Tim seemed friendly enough. This room somehow seemed more cheerful than the last one.

“Well, I’m here, so I might as well find how things work. You’re the first place where people are wearing name tags. What’s up with that?”

“Happy to explain,” said Tim. “But where are my manners. Would you like a glass of wine?”.

“I prefer beer,” replied Jamie. They walked to the front of the building, where a pleasant café fronted the sidewalk. They sat with their drinks.

“Unlike the last two places you’ve been, our universe believes that some data should be shared with everyone while other data should always be kept private. Deciding where the line falls isn’t easy and I can’t say we always get it exactly right. But we keep making adjustments over time. The good thing is people mostly know what to expect and are treated fairly without taking extraordinary measures to protect themselves.”

Jamie didn’t get it. “How does that work? Everyone is wearing name tags, so clearly that’s something you’re required to share. But the tags just show first names. How do you deal with more sensitive data?”

“Excellent question,” smiled Tim, warming to his subject. “So few people really care. Have another drink.”

Tim drained his own wine glass and ordered another. Jamie was still working on his first beer. Tim continued.

“We apply what people in your universe call ‘privacy by design’.  Our badges do more than show our first name: they contain details that are shared on an as-needed basis. For example, when we entered the cafe, a sensor queried my badge and told the server was that I’m allowed to order a drink. But that’s all he learned; it didn’t tell him my age, let alone the name, address, birthdate, and biometrics he’d get from your driver’s license. And if there were some other reason I wasn’t allowed to order a drink – say I was already drunk – it wouldn’t have said that, either. It would just have told the server not to serve me. So my privacy is protected even while drinking restrictions are enforced.”

Jamie pushed away his beer. “So that badge knows that you’ve been drinking? I don’t think I’d want anyone keeping a log of my alcohol consumption.”

Tim looked at his own glass. “Neither would I. But the badge doesn’t keep a log; it just monitors my blood alcohol level. And it doesn’t share that unless there’s reason, like determining whether I can legally order a drink.

Jamie relaxed a bit.  Time continued. 

“There’s lots more information that the badge or other devices do log. My smartphone knows my location history, the Web sites I’ve visited, search queries I’ve made and much more. It uses those to make my life easier, same as in your universe. But, unlike your universe, the data never leaves my device. That way no one can use it in ways I don’t control. When someone wants to serve an ad to people who like red wine” – he lifted his glass – “they just query devices until they find a profile that fits the description. The profiles aren't stored outside the devices and there's no record of which device an ad was served to. That makes things a little harder for advertisers, who can’t control how often one person sees the same ad or connect ad views to subsequent purchases. But it still allows most kinds of behavior- and profile-based personalization.  The economy manages to function.”

Tim stopped short. “Sometimes I repeat myself. I hope I’m not boring you.”

He was, a little. But Jamie could see he loved the topic. “Well, I do want to try to get home. But maybe there’s something here I can bring back that would be useful. What else do you think I should know?”

Tim gathered his thoughts. “To quickly flesh out the picture, the same principles apply to other types of data. So, my phone knows where I live and where I am now, which lets it connect with navigation software to tell me how to get home. But it only asks the central navigation system for a route, without telling it who’s asking. So the navigation system doesn’t know anything about my movements over time. And we do let people view ads for payment, but there are strict rules against trading away personal data.”

“Who makes these rules?” asked Jamie. “In my world we’re pretty skeptical of regulators.”

Tim gave him a sharp look. “So are we. The rules come from a mix of legislators and agency staff.  There's plenty of lobbying from all sides. As I said before, there’s lots of disagreement and they don’t always get things right. But everyone starts from the premise of putting individual interests first and business interests second. Turns out that’s a good guide for many decisions."  Tim paused for a breath.

"And, yes, social interests like public safety come into play. So I can't drive a car if my badge says I'm legally drunk, although the badge doesn't give the car a reason.  I can actually override that rule in an emergency, but then the car also notifies the authorities and turns on special tracking devices.  So, yes, it's complicated.  But just because it’s hard doesn’t mean we shouldn’t try to make it work. You’ve already seen how poorly things turn out in worlds that apply simple solutions instead.”

“Indeed I have,” said Jamie. “I certainly don’t think my universe has it right.” He paused. “But home is home and that’s where I belong. Can we try the flipper again?”

“Of course,” replied Tim. They left the café without paying. Tim winked at Jamie. “Don’t worry. They'll charge my badge. Anonymously.”

Once more into the flipper.

Jamie’s phone buzzed to life before the lid was raised. He knew he was home.

“Good morning, Jamie,” the phone said. “You’re late for work. Shall I call you a car and let the office know you’re on your way?”

Jamie shut it off and opened the lid.

Privacy-Protecting Systems Are The New Green

Let’s take a break from Customer Data Platforms to do some trend-spotting. I spy with my little eye…privacy systems!

Specifically, there's a crop of systems that are privacy-safe alternatives to dominant social, search, email and other common consumer technologies. One well known example is DuckDuckGo, which positions itself as “a search engine that doesn’t track you”.  But there are plenty of others.  Some that have recently caught my attention include:

• Brave, a browser that lets users decide which ads they’ll see and blocks advertisers from seeing behavioral details
• Anagog, which let mobile apps track behaviors and make predictions while keeping all data on the device
• ProtonMail, an encrypted email service  (it's one of a dozen alternatives in that market)
• Vero, an ad-free social network
• Chatterbox, a privacy-safe smart speaker for kids
• Aegis One “mini-computer” for anonymous Web browsing, from a company so privacy-conscious that they apparently don’t publish their contact information (which may take things a bit too far)

A thorough search would surely turn up more examples. You could also add add products whose purpose is privacy, like ad blockers or proxy servers; the gazillion contenders in the pay-people-to-watch ads industry; privacy-enhancing extensions to standard products such as Google Chrome and Firefox; and, perhaps most prominent, the privacy-centered positioning of Apple.

In other words, privacy-protecting systems are a big and growing business.  Privacy is the new green: the cool virtue signifier for consumers and businesses alike.

This is worth noting because industry conventional wisdom has long held that consumers don’t really care about privacy, despite claims to the contrary. The core evidence has been that even people who say they care about privacy are willing to give up their personal data for the tiniest of incentives, whether monetary discounts or convenience. There’s still plenty of data along those lines, such as this Mulesoft study, which found that 49% of consumers would share personal data to get personalized service.  But the same surveys show a substantial minority don’t want their data tracked at all and many have stopped using big social media platforms due to privacy concerns. (See also this Harris Poll report for another set of similar statistics.)

It’s hard to find consistent data over time but it’s a safe bet that this GlobalWebIndex report is correct that consumers' privacy concerns have grown sharply in recent years.

The implications of this are intriguing. There’s a reasonable possibility that we’ll soon gain access to an alternative universe of online systems that protect rather than destroy consumer privacy. If government regulators finally step up their protection of consumers – as is already happening with laws like GDPR and the California Consumer Privacy Act – these systems will have a significant head start over existing products, not to mention vastly more credibility. The result could be a tipping point when network effects kick in and privacy-centric systems suddenly pull a mass audience away from the current, data-fueled incumbents.

That’s still a long shot, if only because the incumbent firms have huge revenues that give them the resources to fight back. But a fundamental change in consumer attitudes could make their brands so toxic that no amount of investment would save them once consumers recognize there are viable, privacy-safe alternatives. 

How will you know if this is actually happening? Keep your eyes out for three things:

• funding announcements from venture capital firms that specifically cite the privacy-preserving features of their investments
• evidence that consumers are paying real attention to privacy, based not just on surveys about attitudes but on actual behaviors (such as the decline in Facebook use, which is already happening)
• a Scott Brinker/Terry Kawaja style logoscape of privacy-enhancing versions of standard consumer technologies

If you’re looking for a much deeper analysis of Internet privacy and other trends, take a look at the Mozilla Foundation’s recent Internet Health Report.

Informatica Buys AllSight and What It Means for the CDP Industry

Informatica announced last week that has purchased AllSight, a Toronto-based Customer Data Platorm vendor. This is interesting on several levels:

• A CDP vendor is being bought by an established enterprise data management firm. This is a normal event in the development of a software category: products are developed as single-purpose "point solutions" and then, after the category is validated and requirements are standardized, the capability is assimilated into larger suites by acquisition (usually) or internal development. The most surprising thing about the CDP industry so far has been that this hasn’t happened very often.  There have been some deals but the two biggest each came with an asterisk: Salesforce bought Datorama but positions it primary for campaign measurement and Arm Ltd., which purchased Treasure Data, is not a typical enterprise software vendor. By contrast, Informatica is very much a leader in enterprise data management, with a major presence in integration, Master Data Management (MDM), governance, and security. The purchase confirms what we already knew: business managers recognize they need unified customer data and enterprise software vendors will offer them solutions.

• It illustrates the difference between MDM and CDP. This is one of those questions we still get fairly often. Informatica lays out the contrast quite nicely: they characterize MDM as limited to highly governed, structured data that delivers the “best version of the truth” about master objects (customers, products, supplier, etc.), while AllSight stores transaction and interactions in addition to the master objects, supports unstructured as well as structured inputs, and allows multiple data views and matching options. AllSight is an exceptionally powerful CDP, with features including machine learning-based identity matching, natural language-based information extraction from unstructured data, a graph data store for complex relationships among objects, ability to create derived variables, and high scalability. These are not found in all CDPs, which is probably one reason Informatica selected AllSight in particular. But all CDPs share the core CDP capability of storing all types of customer information without losing any of the details, which is beyond the scope of MDM.

• It confirms the importance of a persistent data store. This, like storing all information, is part of the core CDP definition. Some enterprise software vendors still argue it’s enough to connect personal identifiers in different source systems and then to pull data from those systems as needed. That is how Informatica has been doing things, apparently. But they’ve now described a progression from “distributed” customer views (disconnected systems) to “federated” views (data remains in transactional systems) to “consolidated” views (data is stored centrally and synchronized with source systems). Reasons they cite for doing this include scalability and real time performance; I’ll add greater consistency, easier control, visibility, and ensured access to historical data.

• It shows use of CDP beyond marketing. Informatica sells primarily to IT staff, which in turn supports all corporate departments, not just marketing. In their briefing on the topic, Informatica made clear that a major reason for buying AllSight is to access the funds controlled by marketers and analytics teams. But they also argued that AllSight will give them more to sell to IT staff, which will use it to better manage large volumes of unstructured data, prepare analytical data sets, add insights based on artificial intelligence and machine learning, and support specific applications such as risk and compliance. Growth to support new use cases and departments is another natural business development for any category.

• It gives a literal example of “CDP Inside”. This is my shorthand for recognizing that CDP functions are increasingly available within larger systems, in addition to being stand-alone products. This is arguably the most important trend in today’s CDP industry.  Let's explore it in more depth.  

“CDP Inside” matters because it’s inevitable as the market matures. So far, the main reaction among vendors and analysts has been to debate which products are properly labeled as CDP systems. Not surprisingly, each CDP vendor wants a definition that matches its own configuration. But those configurations range from products that only build a unified, sharable customer database to products that build the database, analyze its contents, and use it to create personalized marketing messages. The resulting definitions are thus incompatible.  That's a bit of a problem.

The bigger issue is that even the broadest definition excludes other products that include CDP capabilities. Informatica is good example: AllSight didn’t stop being a CDP just because Informatica bought it, but it’s hard to argue that Informatica should now call itself a CDP system as a result. The same question will appear when Salesforce, Oracle, and Adobe finally deliver proper CDP capabilities – which will doubtless happen, sooner or later. Rather than calling every system with CDP capabilities a CDP, I think it's more useful to define core CDP capabilities and then to judge every system’s CDP features against that list. This is another normal market development: companies choose all the time between buying “best of breed” point solutions and buying larger suites that include similar functions. Companies with the most demanding requirements and most sophisticated users often find themselves purchasing “best of breed” products while those with simpler needs often find the product embedded in a suite is good enough.

CDPs are slightly different from other suite components because many suites are connected internally but isolated from external systems.  CDPs are by definition required to connect with external systems.  This means a suite with CDP functions wouldn’t qualify as having a “CDP inside” if those functions couldn’t also connect with external data sources, analytical, decision, and delivery systems. But there’s no technical reason that suite vendors can’t provide systems capable of such connections and their importance is probably well enough understood that most suites will provide them.  Conversely, suite vendors are under increasing pressure to open their systems to integration with external products.  This may make it easier for “best of breed” CDPs to survive than it has been for other “best of breed” products in the past.

Of course, even accepting the notion of “CDP inside” doesn’t resolve the debate over what scope of functions should be included on the CDP requirements list. I lean to a narrower definition, limited largely to building the unified, sharable database, based on the fact that analytics, decisions, and distribution systems can be purchased separately. To me, this indicates that each of those is a separate software category.  In concrete terms, it means that buyers face a separate “suite vs best of breed” decision for each of those components. The counter argument is that some analytics, decision, and distribution capabilities are substantially more effective when they are directly integrated with the unified customer database, so they should be considered part of the core CDP requirements. I suppose the easiest solution is to include the broader requirements on any list and let each buyer decide for herself which ones matter.

The more inclusive approach is in fact what we’re currently taking at the CDP Institute, where I’ve been distinguishing between data, analytical, and personalization CDPs. This seems to reduce confusion, so I’m reluctant to abandon it. But I've drawn the line at systems with delivery capabilities, such as sending email or managing a Web site.  My theory is that those are demanding operational features which are, or at least should be, the focus of the developers' attention.  So calling such systems a CDP is misleading, even though they might have a perfectly adequate "CDP inside".

Our current projects at the Institute include improved evaluation criteria for the data, analytical, and personalization functions.  These will make it easier for buyers to determine which CDP vendors fall into which category or, more precisely, how well each vendor meets the requirements for each category. The good news is that the same evaluation criteria can be applied to the CDP components inside of broader products. So the work will help marketers to make sound choices across all their options regardless of how we define a CDP.  And making good choices, not defining software categories, is ultimately what matters.

Stride Offers a Next-Generation Customer Data Platform

It seems a bit soon to wax nostalgic about the good old days of the Customer Data Platform industry. But the industry has become so bewilderingly diverse in recent years that it is tempting to recall simpler times when all a new product needed to promise was making it easier to unify their customer data.  That was way back in 2016.

Stride is clearly a next-generation CDP, on the market for under two years and including the analytic and orchestration tools common to new CDP systems. But its primary focus on data aggregation and access makes it feel something like a throwback.  How did that happen?

The tale begins with RelateIQ, an AI startup purchased by Salesforce in 2014. Stride’s co-founders came along with the deal and became Salesforce employees. They soon discovered that existing Salesforce tools couldn’t collect and present all the customer data they wanted – Web site browsing history in particular.  They left Salesforce in early 2016 to build a product to fill this gap. Stride, the result of their efforts, reached the market in mid-2017.

The design goal for Stride was a system that could ingest and make available nearly any type of data with little technical effort and then activate that data by sharing customer lists with delivery systems. That's what they delivered.  

It all starts with data. Stride can ingest nearly any type of data with minimal preparation, loading batch files, SQL transactions, streaming data, Web tags feeds, and other sources into SQL tables or S3 buckets. Set-up requires little more than connections to source systems: the ingested data is flattened and then stored in pretty much its original form in what the company describes as a “semi-structured, flexible schema” that can accommodate any source and set of objects. Initial deployment can be completed in a couple of weeks after the source data is made available.

Users can enrich the inputs by adding custom objects, attributes, and events. These are built in structured interfaces designed for non-technical users. Stride doesn’t have its own identity matching processes, although users can map relationships between personal identifiers in different systems and the system will store links between identifiers when it finds them. Stride can perform simple deterministic matching, such as connecting emails to Web cookies on the devices used to open them.

Results are placed in the Snowflake relational database, an increasingly popular tool for cloud-based data warehouse applications. Users can decide which items be exposed for analysis and segmentation. Stride provides extensive tools to explore the loaded data, including a detailed view of all data for an individual customer.

External systems can query the Stride data through APIs.  In addition, Stride provides two sets of integrated capabilties: Audiences and Programs.

Audiences lets users build customer segments using the same drop-down interface that creates custom attributes and events. An audience can be defined as a single segment or as a waterfall sequence of segments, where each is customer assigned to the first segment they match. Segment membership is updated within minutes as new data enters the system.

Audience reports can show movement of customers between segments, can compare different segments against each other, and show segment member statistics such as average order value and number of messages received. Users can analyze or extract subsets within a segment, such as new entrants. Segments can be shared with external systems, including Facebook, Google, and Twitter advertising products. Shared segments are updated automatically as members are added or removed.

Programs assign actions to execute in external systems. Each program has eligibility criteria, behaviors that define entry conditions, actions to take when a behavior occurs, and behaviors that remove customers from the program.  Eligibility criteria can look across all events to do things such as limit contact frequency within a specified time period. The system can’t yet prioritize programs to ensure the most important messages are sent first, although Stride is working on it.

Each program can include a sequence of actions which occur over time.  Actions can also have their own qualification conditions and be chosen in a waterfall sequence.  Actions send instructions and data to external delivery systems. Instructions can trigger a single message or assign a customer to a campaign or journey. The system has standard connectors for major email platforms and marketing clouds as well as advertising systems. Behaviors can include events that trigger actions in near-real-time.

Program reports describe program-generated activities and dropout rates with reasons for each step in a multi-step program. Users can also create split tests within a program as well as global control groups to provide a baseline for measuring program impact.

One feature that marks Strike as a next-generation CDP is pricing.  Most early CDPs were targeted at enterprise buyers and rarely sold for less than $250,000 per year.  Stride pricing is based on number of customer profiles and can be as low as $50,000 per year for a company with under 100,000 contacts. The largest client is over 20 million. Clients are spread across multiple industries including retail, travel, media, financial services, and healthcare.