Software Review: BigID for Privacy Data Discovery

Until recently, most marketers were content to leave privacy compliance in the hands of data and legal teams. But laws like GDPR and CCPA now require increasingly prominent consent notifications and impose increasingly stringent limits on data use. This means marketers must become increasingly involved with the privacy systems to ensure a positive customer experience, gain access to the data they need, and ensure they use the data appropriately. 

I feel your pain: it’s another chore for your already-full agenda.  But no one else can represent marketers’ perspectives as companies decide how to implement expanded privacy programs.  If you want to see what happens when marketers are not involved, just check out the customer-hostile consent notices and privacy policies on most Web sites.

To ease the burden a bit, I’m going to start reviewing privacy systems in this blog. The first step is to define a framework of the functions required for a privacy solution.   This gives a checklist of components so you know when you have a complete set. Of course, you’ll also need a more detailed checklist for each component so you can judge whether a particular system is adequate for the task. But let’s not get ahead of ourselves. 

At the highest level, the components of a privacy solution are:

• Data discovery.  This is searching company systems to build a catalog of sensitive data, including the type and location of each item. Discovery borders on data governance, quality, and identity resolution, although these are generally outside the scope of a privacy system. Identity resolution is on the border because responding to data subject requests (see next section) requires assembling all data belonging to the same person. Some privacy systems include identity resolution to make this possible, but others rely on external systems to provide a personal ID to use as a link.
  
  
• Data subject interactions.  These are interactions between the system and the people whose data it holds (“data subjects”).  The main interactions are to gather consent when the data is collected and to respond to subsequent “data subject access requests” (DSARs) to view, update, export, or delete their data. Consent collection and request processing are distinct processes.  But they are certainly related and both require customer interactions.  So it makes sense to consider them together. They are also where marketers are most likely to be directly involved in privacy programs.
  
  
• Policy definition.  This specifies how each data type can be used.  There are often different rules based on location (usually where the data subject resides or is a citizen, but sometimes where the data is captured, where it’s stored, etc.), consent status, purpose, person or organization using the data, and other variables. Since regulations and company policies change frequently, this component includes processes to identify changes and either automatically adjust rules to reflect them or alert managers that adjustments may be needed.
  
  
• Policy application.  This monitors how data is actually used to ensure it complies with policies, send alerts if something is not compliant, and keep records of what’s done. Marketers may be heavily involved here but more as system users than system managers. Policy application is often limited to assessing data requests that are executed in other systems but it sometimes includes actions such as generating lists for marketing campaigns. It also includes security functions related specifically to data privacy, such as rules for masking of sensitive data or practices to prevent and react to data breaches. Again, security features may be limited to checking that rules are followed or include running the processes themselves. Security features in the privacy system are likely to work with corporate security systems in at least some areas, such as user access management. If general security systems are adequate, there may be no need for separate privacy security features. 

Bear in mind that one system need not provide all these functions.  Companies may prefer to stitch together several “best of breed” components or to find a privacy solution within a larger system. They might even use different privacy components from several larger systems, for example using a consent manager built into a Customer Data Platform and a data access manager built into a database’s core security functions. 

Whew.

Now that we have a framework, let's apply it to a specific product.  We'll start with BigID.

Data Discovery

BigID is a specialist in data discovery. The system applies a particularly robust set of automated tools to examine and classify all types of data – structured, semi-structured, and unstructured; cloud and on-premise; in any language. For identified items, it builds a list showing the application, object name, data type, server, geographic location, and other details. 

Of course, an item list is table stakes for data discovery.  BigID goes beyond this to organize the items into clusters related to particular purposes, such as medical claims, invoices, and employee information. It also draws maps of relations across data sources, such as how the transaction ID in one table connects to the transaction ID in another table (even if the field names are not the same). Other features highlight data sources holding sensitive information, alert users if these are not properly secured from unauthorized access, and calculate privacy risk scores. 

The relationship maps provide a foundation for identity resolution, since BigID can compare values across systems to find matches and use the results to stitch together related records. The system supports fuzzy as well as exact matches and can compare combinations of items (such as street, city, and zip) in one rule.  But the matching is done by reading data from source systems for one person at a time, usually in response to an access request. This means that BigID could assemble a profile of an individual customer but won’t create the persistent profiles you’d see in a Customer Data Platform or other type of customer database. It also can’t pull the data together quickly enough to support real-time Web site personalization, although it might be fast enough for a call center. 

In fact, BigID doesn’t store any data outside of the source systems except for metadata.  So there's no reason to confuse it with a data lake, data warehouse, CRM, or CDP.

Data Subject Interactions

BigID doesn’t offer interfaces to capture consent but does provide applications that let data subjects view, edit, and delete their data and update preferences. When a data access request is submitted, the system creates a case that is sent to other systems or people to execute. BigID provides a workflow to track the status of these cases but won’t directly change data in source systems. 

Policy Definition 

BigID doesn’t have an integrated policy management system that lets users define and enforce data privacy rules. But it does have several components to support the process:

• "Agreements" let users document the consent terms and conditions associated with specific items. This does not extend to checking the status of consent for a particular individual but does create a way to check whether a consent-gathering option is available for an item.
  
  
• “Business flows” map the movement of data through business processes such as reviewing a resume or onboarding a new customer. Users can document flows manually or let the system discover them in the data it collects during its scan of company systems. Users specify which items are used within a flow and the legal justification for using sensitive items. The system will compare this with the list of consent agreements and alert users if an item is not properly authorized. BigID will also alert process owners if a scan uncovers a sensitive new data item in a source system.  The owner can then indicate whether the business flow uses the new item and attach a justification. BigID also uses the business flows to create reports, required by some regulations, on how personal data is used and with whom it is shared. 
  
  
• “Policies” let users define queries to find data in specified situations, such as EU citizen data stored outside the EU. The system runs these automatically each time it scans the company systems. Query results can create an alert or task for someone to investigate. Policies are not connected to agreements or business flows, although this may change in the future. 

Policy Enforcement

BigID doesn’t directly control any data processing, so it can’t enforce privacy rules. But the alerts issued by the policy, agreement, and business flow components do help users to identify violations. Alerts can create tasks in workflow systems to ensure they are examined and resolved. The system also lets users define workflows to assess and manage a data breach should one occur. 

Technology 

 As previously mentioned, BigID reads data from source systems without making its own copies or changes any data in those systems. Clients can run it in the cloud or on-premises. System functions are exposed via APIs which let the company, clients, or third parties build apps on top of the core product. In fact, the data subject access request and preference portal functions are among the applications that BigID created for itself. It recently launched an app marketplace to make its own and third party apps more easily available to its clients. 

Business 

BigID has raised $146 million in venture funding and reports nearly 200 employees. Pricing is based on the number of data sources: the company doesn’t release details but it’s not cheap. It also doesn’t release the number of clients but says the count is “substantial” and that most are large enterprises.

Data Security is a Problem Marketers Must Help Fix

Everything you need to know about 2020 is covered by the fact that “apocalypse bingo” is already an over-used cliché. So I doubt many marketers have found spare time to worry about data security – which most would consider someone else’s problem. But bear in mind that 92% of consumers say they would avoid a company after a data breach. So, like it or not, security is a marketer’s problem too. 

Unfortunately, the problem is a big one. I recently took a quick scan of research on the issue, prompted in particular by a headline that nearly half of companies release software they know contains security flaws.  Sounds irresponsible, don't you think?  The main culprit in that case is pressure to meet deadlines, compounded by poor training in security procedures. If there’s any good news, it’s that the most-used applications have fewer unresolved security flaws than average, suggesting that developers pay more attention when they know it’s most important. 

The research is not reassuring. It may be a self-fulfilling prophecy, but most security professionals see data breaches as inevitable. Indeed, many think a breach is good for their career, presumably because the experience makes them better at handling the next one. Let’s just be grateful they're not airline pilots. 

Still, the professionals have a point. Nearly every company reports a business-impacting cyberattack in the past twelve months. Even before COVID-19, fewer than half of IT experts were confident their organizations can stop data breaches with current resources.

The problems are legion. In addition to deadline pressures and poor training, researchers cite poorly vetted third-party code libraries, charmingly described as “shadow code”; compromised employee accounts, insecure cloud configurations, and attacks on Internet of Things devices.

Insecure work-from-home practices during the pandemic only add new risk. One bit of good news is that CIOs are spending more on security,  prioritizing access management and remote enablement. 

What’s a marketer to do?  One choice is to just shift your attention to something less stressful, like fire tornados and murder hornets. It’s been a tough year: I won’t judge. 

But you can also address the problem. System security in general is managed outside of most marketing departments. But marketers can still ensure their own teams are careful when handling customer data (see this handy list of tips from the CDP Institute). 

Marketers can also take a closer look at privacy compliance projects, which often require tighter controls on access to customer data. Here’s an overview of what that stack looks like.  CDP Institute also has a growing library of papers on the the topic.

Vendors like TrustArc, BigID, OneTrust, Privitar, and many others, offer packaged solutions to address these issues. So do many CDP vendors. Those solutions involve customer interactions, such as consent gathering and response to Data Subject Access Requests.  Marketers should help design those interactions, which are critical in convincing consumers to share personal data that marketers need for success. The policies and processes underlying those interfaces are even more important for delivering on the promises the interfaces make. 

In short, while privacy and security are not the same thing, any privacy solution includes a major security component. Marketers can play a major role in ensuring their company builds solid solutions for both. 

Or you can worry about locusts. 

 

Don't Misuse Proof of Concept in System Selection

Call me a cock-eyed optimist, but marketers may actually be getting better at buying software. Our research has long shown that the most satisfied buyers base their selection on features, not cost or ease of use. But feature lists alone are never enough: even if buyers had the knowledge and patience to precisely define their actual requirements, no set of checkboxes could capture the nuance of what it’s actually like to use a piece of software for a specific task. This is why experts like Tony Byrne at Real Story Group argue instead for defining key use cases (a.k.a. user stories) and having vendors demonstrate those. (If you really want to be trendy, you can call this a Clayton Christensen-style “job to be done”.)

In fact, use cases have become something of an obsession in their own right. This is partly because they are a way of getting concrete answers about the value of a system: when someone asks, “What’s the use case for system X”, they’re really asking, “How will I benefit from buying it?” That’s quite different from the classic definition of a use case as a series of steps to achieve a task. It’s this traditional definition that matters when you apply use cases to system selection, since you want the use case to specify the features to be demonstrated. You can download the CDP Institute’s use case template here.

But I suspect the real reason use cases have become so popular is that they offer a shortcut past the swamp of defining comprehensive system requirements. Buyers in general, and marketers in particular, lack the time and resources to create complete requirements lists based on their actual needs (although they're perfectly capable of copying huge, generic lists that apply to no one).  Many buyers are convinced it’s not necessary and perhaps not even possible to build meaningful requirements lists: they point to the old-school “waterfall” approach used in systems design, which routinely takes too long and produces unsatisfactory results. Instead, buyers correctly see use cases as part of an agile methodology that evolves a solution by solving a sequence of concrete, near-term objectives.

Of course, any agile expert will freely admit that chasing random enhancements is not enough.  There also needs to be an underlying framework to ensure the product can mature without extensive rework. The same applies to software selection: a collection of use cases will not necessarily test all the features you’ll ultimately need. There’s an unstated but, I think, implicit assumption that use cases are a type of sampling technique: that is, a system that meets the requirements of the selected use cases will also meet other, untested requirements.   It’s a dangerous assumption. (To be clear: a system that can’t support the selected use cases is proven inadequate. So sample use cases do provide a valuable screening function.)

Consciously or subconsciously, smart buyers know that sample use cases are not enough. This may be why I’ve recently noticed a sharp rise in the use of proof of concept (POC) tests. Those go beyond watching a demonstration of selected use cases to actually instal a trial version of a system and seeihow it runs. This is more work than use case demonstrations but gives much more complete information.

Proof of concept engagements used to be fairly rare. Only big companies could afford to run them because they cost quite a bit in both cash (most vendors required some payment) and staff time (to set up and evaluate the results). Even big companies would deploy POCs only to resolve specific uncertainties that couldn’t be settled without a live deployment.

The barriers to POCs have fallen dramatically with cloud systems and Software-as-a-Service. Today, buyers can often set up a test system with a just a few mouse clicks (although it may take several days of preparation before those clicks will work). As a result, POCs are now so common that they can almost be considered a standard part of the buying process.

Like the broader application of use cases, having more POCs is generally a good thing. But, also like use cases, POCs can be applied incorrectly.

In particular, I’ve recently seen several situations where POCs were used as an alternative to basic information gathering. The most frightening was a company that told me they had selected half a dozen wildly different systems and were going to do a POC with each of them to figure out what kind of system they really needed.

The grimace they didn’t see when I heard this is why I keep my camera off during Zoom meetings. Even if the vendors do the POCs for free, this is still a major commitment of staff time that won’t actually answer the question. At best, they’ll learn about the scope of the different products. But that won’t tell them what scope is right for them.

Anther company told me they ran five different POCs, taking more than six months to complete the process, only to later discover that they couldn’t load the data sources they expected (but hadn’t included in their POCs). Yet another company let their technical staff manage a POC and declare it successful, only later to learn the system had been configured in a way that didn’t meet actual user needs.

You’re probably noticing a dreary theme here: there’s no shortcut for defining your requirements. You’re right about that, and you’re also right that I’m not much fun at parties. As to POCs, they do have an important role but it’s the same one they played when they were harder to do: they resolve uncertainties that can’t be resolved any other way.

For Customer Data Platforms, the most common uncertainty is probably the ability to integrate different data sources.  Technical nuances and data quality are almost impossible to assess without actually trying to load each system.  Since these issues have more to do with the data source than the CDP, this type of POC is more about CDP feasibility in general than CDP system selection. That means you can probably defer your POC until you’ve narrowed your selection to one or two options – something that will reduce the total effort, encourage the vendor to learn more about your situation, and help you to learn about the system you’re most likely to use.

The situation may be different with other types of software. For example, you might to test q wide variety of predictive modeling systems if the key uncertainty is how well their models will perform. That’s closer to the classic multi-vendor “bake-off”.  But beware of such situations: the more products you test, the less likely your staff is to learn each product well.

With a predictive modeling tool, it’s obvious that user skill can have a major impact on results. With other tools, the impact of user training on outcomes may not be obvious. But users who are assessing system power or usability may still misjudge a product if they haven’t invested enough time in learning it.  Training wheels are good for beginners but get in the way of an expert. Remember that your users will soon be experts, so don’t judge a system by the quality of its training wheels.

This brings us back to my original claim.  Are marketers really getting better at buying software?  I’ll stand by that and point to broader use of tools like use cases and proof of concepts as evidence. But I’ll repeat my caution that use cases and POCs must be used to develop and supplement requirements, not to replace them. Otherwise they become an alternate route to poor decisions rather than
guideposts on the road to success.

Here's a Game about Building Your Martech Stack

TL;DR: you can play the game here.

I’ve recently been running workshops to help companies plan deployment of their Customer Data Platforms. Much of the discussion revolves around defining use cases and, in particular, deciding which to deliver first. This requires balancing the desire to include many data sources in the first release of the system against the desire to deliver value quickly. The challenge is to find an optimal deployment sequence that starts with the minimum number of sources needed for an important use case and then incrementally adds new sources that support new use cases. I’ve always found that an intriguing problem although I’ll admit few others have shared my fascination.

As coronavirus forces most marketers to work from home, I’ve also been pondering ways to deliver information that are more engaging than traditional Webinars and, ahem, blog posts. The explosion of interest in games in particular seems to offer an opportunity for creative solutions.

So it was fairly natural to conceive of a game that addresses the deployment sequence puzzle. The problem seems like a good candidate: governed by a few simple dynamics that become interestingly complex when they interact. The core dynamic is that one new data source may support new multiple use cases, while different combinations of sources support different use cases. This means you could calculate the impact of different sequences to compare their value.

Of course, some use cases are worth more than others and some sources cost more to integrate than others; you also have to consider the availability of the CDP itself, of central analytical and campaign systems, and of delivery system that can use the outputs. But for game purposes, you could simplify matters to assume that each system costs the same and each use case has the same value. This still leaves in place the core dynamic of balancing the cost of adding one system with the value of enabling multiple use cases with that system.

To make things even more interesting and realistic, you could add the fact that some use cases are possible with a few systems but become more valuable as new systems come online.  It might be that their data adds value – say, by making predictions more accurate – or because they enable delivery of messages in more channels.

In the end, then, you end up with a matrix that crosses a list of systems (data sources, CDPs, analytics, campaigns management, and delivery systems) against a list of use cases. Each cell in the matrix indicates whether a particular system is essential or optional for a particular use case. Value for any given period would include: the one-time cost of adding a new system; the recurring cost of operating active systems, and the value generated by each active use case.  That use case value would include a base value earned by running the use case plus incremental value from each optional system. Using red to indicate required systems and grey to indicate optional systems, the matrix looks like this:

The game play would then be to select system one at a time, calculate the value generated as the period revenue, and then repeat until you run out of systems to add.  Sometimes you’d select systems because they made a new use case possible, sometimes you’d select because they added optional value to already-active use cases, and sometimes you’d select a system to make possible more use cases in the future. Fun!

I then showed this to a professional game designer, whose response was “you may have found the least fun form factor imaginable: the giant data-filled spreadsheet. I'm kind of impressed.”

Ouch, but he had a point. I personally found the game to be playable using a computer to do the calculations but others also found it impenetrable. A version using physical playing cards was clearly impossible.

So, after much pondering, I came up with a vastly simplified version that collapsed the 19 systems in the original model into three categories, and only required each use case to have a specified number of systems in each category. I did keep the distinction between required and optional systems, since that has a major impact on the effectiveness of different solutions. I also simplified the value calculations by removing system cost, since that would be the same across all solutions so long as you add one system per period.


The result was a much simpler matrix, with just six columns (required and optional counts for each of the three system types) and the same number of rows per use case (22 in the example). I built this into a spreadsheet that does the scoring calculations and stores results for each period, so the only decision players need to make in any turn is which of the three system types to select. Even my game designer grudgingly allowed that it “made sense pretty quickly” and was “kinda fun”. That’s about all the enthusiasm I can hope for, I suspect.

I’ve put a working version of this in a Google spreadsheet that you can access here.

Go ahead and give it a play – it just takes a few minutes to complete once you grasp how it works (put a ‘1’ in the column for each period to select the class of system to add during that period). Most of the spreadsheet is write-protected but there’s a leaderboard if you can beat my high score of 1,655.

Needless to say, I’m interested in feedback. You can reach me through LinkedIn here.
Although this started as a CDP planning exercise, it’s really a martech stack building game, something I think we can all agree the world desperately needs right now. I also have worked out a physical card game version that would have a number of additional features to make games more interesting and last longer. Who wants to play?

A Dozen Market Research Studies on COVID-19 Business Impact

This sums it up.  From Bank of America via Twitter but I can't find a link to the original.

As marketers finish their initial emergency adjustments to coronavirus lockdowns, they are starting to think about longer-term plans. While the shape of things to come is impossible to guess, reporting on industry changes has become a marketing trend of its own. Here are a dozen-plus studies I’ve seen in the past week, most of which are on-going.

Retail Behavior Data

Adobe this week launched their Digital Economy Index, a long-term project that gained unexpected immediate relevance. The index draws on trillions of Web visits tracked by Adobe systems to construct a digital consumer shopping basket tracking a mix of products including apparel, electronics, home and garden, computers, groceries, and more. The headline finding of the initial report would have been a continuing drop in prices driven by electronics, but this was overshadowed by short-term changes including a 225% increase in ecommerce from March 1-11 to March 13-15. Online groceries, cold medications, fitness equipment and computers surged, as did preordering for in-store pickup. Extreme growth was concentrated in hard-hit areas including California, New Hampshire and Oregon.

Customer Data Platform vendor Amperity reported a less rosy result in its COVID-19 Retail Monitor, which draws data from Amperity’s retail clients. They report that total retail demand fell by 86% by the end of March and even online revenue is down 73%.  Food and health products fell after an initial stock-up surge in mid-March.

Retail foot traffic tracker Placer.ai has packaged its in-store data in a COVID-19 Retail Impact tracker, which not surprisingly shows an end to traffic at shuttered entertainment and clothing outlets, near-total drop at restaurants, and mixed results for grocery stores and pharmacies. Results are reported by day by brand, if you really want to wallow in the gruesome details.

Grocery merchandising experts Symphony RetailAI have also launched a COVID-19 Insights Hub, which reports snippets of information with explanations. These range from obvious (consumers are accepting more product substitutions in the face of stock-outs) to intriguing (canned goods sales rose twice as much in the U.S. than in Europe because of smaller families and less storage space).

Retail Behavior Surveys

Showing just how quickly the world changed, retail consumer research platform First Insight found that the impact of coronavirus on U.S. shopping behavior doubled between surveys on February 28 survey and March 17. In the later survey, 49% of consumers said they were buying less in-store and 34% were shopping more online. Women and baby boomers went from changing their behavior slightly less than average in the first survey to changing slightly more than average in the second.

Ecommerce platform Yotpo ran its own survey on March 17, reaching 2,000 consumers across the U.S., Canada, and United Kingdom. They found consumers evenly split between expecting to spend more or less over-all, with a just 32% expecting to shift purchases online. Food, healthcare, and, yes, toilet paper were high on their shopping lists.

The situation was clearer by the end of March, when Retail Systems Research surveyed 1,200 American consumers for Yottaa. By this time, 90% were hesitant to shop in-store, 94% expected online shopping will be important during the crisis, and their top concerns were unavailable inventory, no free shipping, and slow websites. (Really, no free shipping?) More surprising but prescient, given Amazon's labor troubles: just 42% felt confident that Amazon could get their online orders delivered on time.

Media Consumption

Nobody wins any prizes for figuring out that Web traffic went up when people were locked down. But digital analytics vendor Contentsquare did provide a detailed analysis of which kinds of Web sites attracted more traffic (supermarkets, media, telecom, and tech retail) and which went down the most (luxury goods, tourism, and live entertainment) in the U.S., UK, and France. Week-by-week data since January shows a sharp rise starting March 16. Less easily predictable: supermarket and media conversion rates went down as consumers spent more time searching for something they wanted.

Media tracking company Comscore has also weighed in with an ongoing series of coronavirus analyses. Again, no surprises: streaming video, data, newscasts, and daytime TV viewing are all up. Same for Canada and India, incidentally.


You also won’t be shocked to learn that Upfluence found a 24% viewing increase in the live-streaming game platform Twitch in Europe. Consumption growth tracked national lockdowns, jumping in Italy during the week of March 8-14 and in France and Spain the week after.

Consumer review collector PowerReviews has its own data, based on 1.5 million product pages across 1,200 Web sites. Unlike Contentsquare, they found traffic was fairly flat but conversion rates jumped on March 15 and doubled by March 20. Their explanation is people were buying basic products that took less consideration.  People read many more reviews but submission levels and sentiment were stable. Reviews were shorter as consumers likely had other things on their minds.


Influencer marketing agency Izea got ahead of the game with a March 12 survey, asking social media consumers how they thought they’d behave during a lockdown. More social media consumption was one answer, with Facebook and Youtube heading the list. Izea also predicted that influencer advertising prices would fall as more influencers post more content.


Consumer Attitudes


Researching broader consumer attitudes, ITWP companies Toluna, Harris Interaction and KurunData launched a Global Barometer: Consumer Reactions to COVID-19 series covering the U.S., UK, Australia, India, and Singapore. The first wave of data was collected March 25-27.  People in the U.S. and India were generally more satisfied with how businesses had behaved and more optimistic about how quickly things would return to normal. But U.S. respondents ranked support from the national government considerable lower than anyone else.


Edelman Trust Barometer issued a ten market Special Report on COVID-19, although the data was gathered during the good old days of March 6-10. Even then, most people were following the news closely and 74% worldwide felt there was a lot of false information being spread. Major news outlets were the primary information source everywhere (64%) but the U.S. government was by far less relied upon (25%) than anywhere else (31% to 63%). Interesting, people put more faith in their employers than anyone except health authorities. They also expected business to protect their workers and local communities.


Kantar Media has yet another COVID-19 Barometer, although they reserve nearly all results for paying clients. The findings they did publish echo the others: more online media consumption, low trust in government, and expectation that employers will look after their employees. Kantar says that just 8% of consumers expect brands to stop advertising but 77% want advertising to show how brands are being helpful, 75% think brands should avoid exploiting coronavirus and (only?) 40% feel brands should avoid “humorous tones”.

Survey company YouGov publishes a continuously-updated International COVID-19 Tracker with timelines on changing opinions in 26 countries.  Behaviors including avoiding public places and not going to work change quickly; others such as fear of catching coronavirus and wearing masks move more slowly.  Other attitudes have barely shifted, including avoiding raw meat and improving personal hygiene.  The timing of changes correlates with the situation in each country.

Job Listings


There’s also an intriguing little niche of companies offering job information. PR agency Global Results Communications just launched a COVID-19 Job Board to help people find work.  So far, it's not very impressive: as of April 1 it had under 100 random listings from Walmart to Metrolina Greenhouses to the South Carolina National Guard.


Tech salary negotiators Candor (did you know that was a business?) has a vastly more useful site, listing 2,500+ companies that are reported to be hiring, freezing hiring, rescinding offers, or laying people off. At the moment, half the companies on the list are hiring. The site offers a very interesting break-down by industry: transportation, retail, consulting, energy, and automotive are in the worst shape. Defense, productivity and education software, and communications are doing the best.

Balandra Orchestrates Customer Journey Without a CDP

Balandra Customer Flow Diagram

The need for a system that assembles unified, sharable customer profiles is now widely accepted. So is the label of “Customer Data Platform” to describe such systems. What people do still debate is whether a Customer Data Platform should only assemble those profiles or should also include features to “activate” them in the sense of selecting customer treatments. I personally find the discussion uninteresting since the plain reality is that some companies want activation features in their CDP and others do not.  Companies that don't want activation in their CDP may already have a separate activation system or prefer to purchase a separate one.  This means that activation is optional, and, thus, not a core CDP feature. QED.

Theory aside, it’s true that the majority of CDPs do include activation features.  This makes a stronger argument for the weaker claim that most buyers want activation features in their CDP.  But this has nothing to do with CDPs in particular: it’s just an instance of the general rule that buyers prefer integrated systems to separate components. This is known (to me, at least) as Raab's Law, stated most succinctly as "suites win".

A diehard advocate of “CDPs need activation” might question whether activation systems can truly be purchased separately. My response points to Journey Orchestration Engines (JOEs), a small but intriguing category that includes Thunderhead, Pointillist, and Kitewheel among others. These products select the best treatment for each customer in each situation and transmit their choice to delivery systems (email, Web CMS, mobile app, call center, etc.) for execution. All need customer profiles to function, but they don’t necessarily meet the RealCDP requirements for accepting data from all sources, retaining all details, storing the data internally, or sharing their profiles with others. This is because their designers’ focus is on the very different challenge of making it easy for users to define, manage, and optimize customer treatments across channels.

Meeting that challenge requires presenting customer data effectively, identifying events that might require an action, selecting the right action in the current situation, and sending that action to external systems for delivery. Some tasks, such as data presentation and delivery system integration, are also found in other types of systems. The unique challenge for Journey Orchestration Engines is finding the right action while taking into account the customer’s complete situation (not just the current interaction). This requires understanding all the factors that are relevant in the current situation and choosing the best among all possible actions.

Of course, "all" is an impossibly high standard.  A more realistic goal is to understand as many factors as possible and choose among the broadest range of available actions. It’s an important distinction because the scope of available data and actions will grow over time.  This means the key capability to look for is whether a system has the flexibility to accommodate new data and actions as these become available.

This brings us to Balandra, a Madrid-based journey orchestration engine.

Balandra is designed for complex service industries such as insurance, telecommunications, and healthcare, where companies have multiple, complex operational systems. Left to run independently, these systems will each send their own messages, creating a disconnected and often inappropriate experience for each customer. Balandra intercepts these messages and replaces them with a single stream is governed by a common set of rules.

The rules themselves draw on a structure that organizes customer experience into major processes such as onboarding a new client, setting up a new service, or filing an insurance claim. Each process is assigned a combination of data, lifecycle stages, available actions, and decision rules. When an event occurs that involves the process, Balandra executes its rules to pick an action based on the customer’s data and lifecycle stage.

This may not sound especially exciting. But it’s important to contrast Balandra’s approach with conventional customer journey flows.  These follow a specified sequence of messages and events, at best with some branching to accommodate different customer behaviors as the journey progresses. But a conventional journey flow can only include a fairly low number of steps and branches before it becomes incomprehensibly complex. The rule-based approach avoids this problem by letting users  create different rules for different factors and apply them in sequence. So, you might have one rule that checks for recent customer service issues, another that checks for customer value, and another for previous purchases. Each rule would add or exclude particular messages from consideration. After all the rules had executed, a final rule would select from the pool of messages that remain available.

The advantage of this approach is that each rule executes independently, avoiding the need for a complex decision tree that specifies different treatments for different combinations of conditions. Rules can just be added or dropped into the mix knowing that they’ll apply themselves only when relevant conditions are met. For example, a rule might check for recent customer service problems and suppress new product offers within the following two weeks if one occurred. This happens (or doesn’t happen) across all interactions without explicitly building that check into each journey flow.

To be clear, Balandra isn’t the only system to take this approach. In fact, its actual rule definition and execution is done using a standard business rules engine – IBM’s Operational Decision Manager (ODM), formerly ILOG. The system does have an interface that lets non-technical users define the data associated with each process and specify connections with delivery systems. It can ingest data in real time via APIs, through event streams such as Kafka, or through batch file updates. It can support both real time interactions and batch processing for outbound campaigns.

If you’re keeping score, Balandra doesn’t qualify as a CDP because it only uploads a fraction of the data related to a customer – the interactions between customer and company systems. While this means Balandra clients might still want a separate CDP system, it also enables Balandra to use many fewer resources than a CDP would.

Balandra launched its product in 2014. It currently has four clients in production, all in Spain, and is looking distribution partners in other regions. Pricing starts around $50,000 per year and grows based on the number of customers.

Reflections on the CDP Revolution in France (and the Rest of Europe)

The CDP Institute just published a report on the CDP Industry in Europe (download here). This was based primarily on the global Industry Update released last month. This showed especially fast growth in Europe, with a year-on-year increase of 74% in the number of European vendors and 80% in European CDP employment, compared with growth outside of Europe of 38% in vendors and 59% in employment.

We spent quite a bit of time in Europe last year, so I certainly have my own ideas of the reasons behind these sharp increases. But it always seems best to get information from people who live in the region. So as part of the report we collected comments from several European CDP vendors and consultants on what they saw happening in their markets. Their complete comments are included in the report. They are largely consistent with each other, so I think it’s fair to give a summary. Here’s my interpretation:

• The European market is divided into several zones, each at a different stage of the development. The UK market is closest to the U.S. and most mature. Growth there began in 2017, paused as companies worked to meet the GDPR deadline of May 2018, and then resumed. The Netherlands and Germany are next in line, with growth taking off after mid-2018. Southern Europe, Eastern Europe and the Nordics the least mature, with limited deployment to date. Maturity can be measured by understanding of CDP, adoption levels, and the speed of sales cycles.

• Each market is served by native national vendors. These are often affiliated with marketing agencies or consultancies and usually provide a combination of data assembly and campaign management. Large U.S-based vendors have a substantial presence in the UK and Netherlands/Germany regions but little activity elsewhere. These vendors are primarily focused on data assembly. Some of the European vendors also sell throughout the UK/Netherlands/Germany markets. Few non-local vendors have much presence in Southern Europe, Eastern Europe, or the Nordic.

• France is a market of its own.  Most CDP sales in France are made by French vendors, who sell little outside of France.  U.S. and non-French European vendors do continue to try to penetrate the French market, so far with limited success.  Unlike other markets, the French vendors generally started as Data Management Platforms (DMPs) although they took a broad approach that included some CDP features from the start. They have now further evolved to towards CDP although their DMP roots still show.

• GDPR was an early impetus to CDP adoption but that momentum is now largely spent. Current interest in CDP is based on the core use cases of data unification and campaign management. In the more mature markets, where CDPs are better understood, this interest is most likely to result in buying a packaged CDP system. In the less mature markets, this interest is more likely to result in buying a solution from an agency or in building a solution in-house.

These observations largely parallel my own impressions of the region. One difference is that none of the commenters mentioned the several European CDPs that compete globally as specialists in travel, telecommunications, financial services, or retail. The reason may simply be that only one the vendors who contributed to the report is in this category.  Also, none mentioned the limited funding available to European CDP vendors, an extremely sharp contrast to heavily-funded U.S.-based firms.

There’s much more in the report, both in the vendors’ comments and in the industry data.  Again you can download it here.  Enjoy.

Understanding Adobe Real Time CDP

Adobe fully released its Real Time Customer Data Platform last November. Although they had briefed me on it before then, it was only this week that I finally caught up with them to discuss the final product. Since this is a topic of great interest – and confusion – it’s worth sharing what I learned.

Part of the confusion has to do with Adobe’s habit of reusing product names. It’s easy to confuse the Adobe Experience Platform, the core system for collecting customer data, assembling profiles, applying machine learning, and sharing the results with services and applications, with Adobe Experience Manager, the Web content management system that is one of those applications. Similarly, Experience Platform contains a Real Time Customer Profile, which is different from the Real Time CDP, one of the services that consumes data from Experience Platform. Got that?

It also doesn’t help that Adobe describes Experience Platform and Real-Time CDP as managing unknown and known profiles for activation across all channels, without clarifying that it’s only referring to first-party data. It turns out that Audience Manager, their Data Management Platform (an “application” in their terminology), holds third party profiles that aren’t shared with Experience Platform.   Adobe nevertheless uses the term “audience activation” to describe movement of first party profiles from Experience Platform into other applications, including into Audience Manager itself.

Somebody get these people a thesaurus.

But that’s just words. What really takes explaining is that Adobe has split what it considers the functions of a CDP between the Experience Platform and the Real Time CDP itself. Specifically, they describe a CDP as doing three things:

• ingesting customer data from all enterprise sources
• creating persistent customer profiles used for modeling and segmentation
• “activating” the profiles by moving them into applications

The first two functions, ingestion and profile management, are provided by Experience Platform. The third is provided by Real Time CDP.  

In other words, although Adobe’s combined stack does everything you expect from a CDP, its Real Time CDP only provides one of the three core functions. I’ll pause for a moment while you wrap your head around that.

Ready to continue? Great.  There’s nothing inherently wrong with Adobe’s approach, which is ultimately another matter of labels. In fact, I’ve recently seen several situations where the CDP is primarily an access tool that connects a master data store to marketing systems. As with Real Time CDP, the role is simply to take already-assembled data and put it in a format that’s suitable for marketers (or potentially other business users). So Adobe may be on to something.

Of course, any system that just provided data access would not be a CDP. Adobe’s combined solution does meet the CDP Institute definition of a CDP: packaged software that builds a unified, persistent customer database accessible to other systems. They might be a bit weak at the edges – I haven’t explored whether they can truly include all sources and all details within the Experience Platform database.  News that that Experience Platform excludes the third party profiles in Audience Manager does raise some questions about that. But, truth be told, quite a few CDPs have some practical limits in those areas.

You’re likely aware that some analysts and CDP vendors disagree with the CDP Institute definition, arguing a CDP should include analytics and experience orchestration. The general logic is that data is worthless unless it’s exploited, so the CDP should include features to use it. This is usually described as “activation”, a word I’m avoiding since Adobe is using it one way (to describe moving data from the CDP into application systems, with some segmentation capability but no message selection), while others often use it to include message selection, personalization, and orchestration. I personally don’t much care how anyone defines “activation” so long as they’re clear about what they mean when they use it. I happen to agree with Adobe that message selection, personalization, and orchestration aren’t essential CDP functions.  But that’s a debate for another day.

That said, it’s important to know that Real Time CDP isn’t the only Adobe service that can access the customer profiles in Experience Platform. Services including analysis, journey orchestration, and offer management provide alternative connections between Experience Platform and the applications. This is wrinkle that wouldn’t be present in a CDP that provided ingestion, profile creation, and access as part of one system. It adds complexity if one application might connect with several services. You might even worry that it recreates the crazy wall of point-to-point connections that causes people to want CDPs in the first place.  But that’s an overstatement if only four services are involved.

A more pressing concern would be how much data is actually loaded into Experience Platform. Some operational data will stay within the individual Adobe applications because no other system can use it.  Beyond that, my understanding is that Experience Platform has an extensible data model which would theoretically handle any information that users wanted it to ingest. But that could be wrong. Anyone thinking about buying the system should check that it can load the sources that matter in their situation. Remember that Experience Platform grew out of Adobe’s previous approach, which largely relied on storing identifiers within the central data store and looking up everything else in its source systems as needed. Adobe has clearly moved beyond that but some traces may linger.

Salesforce Buys Evergage But Not For CDP

The CDP Institute published its semi-annual Industry Update report today, which you download here for free. Although every word and image in the report is a jewel, there’s no question that the main story in this edition is CDP industry consolidation. Events in the past six months (stretching a bit to include early January 2020) include seven new funding rounds, three acquisitions of CDP vendors, four acquisitions by CDP vendors, and four asset sales by CDP companies.  Asset sales aside, these are all ways for companies to strengthen their business more quickly than organic growth permits.

What’s particularly intriguing is the industry position of the firms in these deals.  Using our best guess at CDP employment for each vendor, only one of the twelve vendors involved funding or either side of an acquisition is among the industry’s five largest (SessionM, bought by Mastercard). The rest all ranked within the fairly narrow band from number eight to number thirty (of 101 total). That is, they were bigger than most but not the industry's largest.  I interpret this to mean that these vendors were either adding resources for a push to reach the industry top tier or have already decided they need to be part of something else.  

Notably, the firms that engaged in asset sales were much smaller: only IgnitionOne would have fallen within the top thirty and their deal might be considered more of an acquisition, since Zeta Global is apparently still selling the product.

Careful readers will have already noticed that this chart includes one other deal: acquisition of Evergage by Salesforce, announced on Monday.  Evergage fits into the size range of the other deals and the sale can certainly be seen as an escape from the crowded campaign CDP space. But the purchase is otherwise atypical because Salesforce has stressed that they are primarily interested in Everage for real time interaction management and personalization.  Of course, Salesforce is already far along in work on its own CDP, the Customer 360 Audiences component of Customer 360 Truth, which is due for general release around June. So this deal has little to do with Evergage as a CDP.  It's about closing a gap elsewhere in the Salesforce product line, not a sudden acceleration of Salesforce’s entry into the CDP space.

Explaining Martech to a Five Year Old

One piece of paper stands between me and ending the year with a clean desk: a list scribbled on hotel stationery that compares different customer data systems to different types of motor vehicles. Richard Scarry  meets CDP Institute, you might say.

So, just in case your New Year’s plan includes explaining marketing technology to a five year old, here we go:

• Data Warehouse = School Bus. The seats in a bus are lined up in nice neat rows, designed to carry one type of cargo very efficiently. Similarly, a data warehouse is highly structured environment that is very efficient at dealing with specified data types. (See how this works?)

• Data Lake = Moving Van. A moving van is a big box that can hold pretty much anything, although it might be jumbled together in no particular order. A data lake can store any type of data but doesn’t do much to organize it.

• Integration Platform = Delivery Motorcycle. A motorcycle carries small items quickly from one place to another, but doesn’t have any place to store them. An integration platform moves bits of data between systems but doesn’t have its own storage, either.

• Data Management Platform = Fire Truck. A fire truck is a highly specialized vehicle designed to move very quickly and pour out huge volumes of water. A DMP is a highly specialized system that quickly moves huge volumes of data.

• CRM System = Taxi. A taxi carries one person and their baggage directly to a specific destination. A CRM system delivers one customer and their history to a single sales person or call center agent.

• Cloud Platform = Car Carrier. A car carrier is a frame that can hold many unconnected vehicles. A Cloud Platform supports many unrelated systems in the same rack.

Believe it or not, my original list didn’t include an entry for Customer Data Platforms. So let’s give that a moment’s thought and go with…Ice Cream Truck!  They make a lot of noise and everybody loves them.  No, wait, better still...Food Truck!  A food truck collects ingredients from various sources, converts them into delicious meals, and distributes the results to many happy customers.  A CDP combines customer data into profiles that it shares with different systems.  

You’re welcome.

Happy New Year!

Acquia Buys AgilOne CDP

Acquia, which is moving past its roots in Web content management to become a multi-channel “digital experience platform” (DXP), took a big step in that direction today with a deal
to buy the AgilOne Customer Data Platform. The deal follows Acquia’s May 2019 purchase of open source marketing automation platform Mautic  and September 2019 purchase of site building tool Cohesion.  Acquia itself was purchased in September by Vista Equity Partners  for $1 billion, which obviously supports their DXP strategy.

The logic behind this deal is so clear that there’s little need for comment. While the exact meaning of DXP is a bit fuzzy, it surely involves coordinating and personalizing customer experiences across channels. This certainly requires the unified customer data that a CDP provides.  Acquia’s heritage in Web content management doesn’t provide deep customer data unification experience, and neither does Mautic.  AgilOne is a particularly good fit because it’s better than many CDPs at identity matching, including offline as well as online data. It also provides lots of connectors to source and delivery systems, as well as advanced machine learning for segmentation and predictions. Acquia and Mautic lacked those, too.

AgilOne rebuilt its core technology fairly recently, giving it a highly flexible and scalable platform that should easily extend beyond the company’s current base in mid-tier retail.  In particular, it will be able to serve Acquia’s clients, who tend to be very large companies with multiple Web multi-sites around the world. At the same time, AgilOne gives Acquia a stronger story in retail and other B2C markets where it has been less active.  AgilOne will also gain by integrating with some of Mautic’s features, notably email and SMS delivery and complex customer journey management. And the deal gives AgilOne much deeper resources to fund growth than it had as an independent company.

What, if anything, does the deal tell us about the larger CDP industry? I’d argue it mostly reinforces the trends I described in October, of independent CDPs being purchased by companies that are not primarily marketing software vendors but need to add customer data capabilities. Nearly all major CDP purchases to date meet this description: Mastercard buying SessionM, Dun & Bradstreet buying Lattice Engines, Arm buying Treasure Data, and Informatica buying Allsight. The only partial exception is Salesforce buying Datorama, but CDP wasn’t the focus of that deal. None of the other companies trying to follow Acquia’s approach of expanding from Web content management to DXP has yet purchased a CDP. But they’ll all need CDP functions so don’t be surprised to see more deals along those lines.

Put in a broader context, adding a CDP as a module inside a DXP is an example of CDP as a component within large marketing or even operational systems, something I refer to as “CDP Inside”. I expect that to be increasingly common and, thus, a potential home for independent CDP systems as the market matures, competition heats up, and the big marketing cloud vendors release their own products.  Selling or merging to become part of a larger system is one escape path for the independent CDPs. Another path is to focus on specific industries or cost-sensitive segments where the big marketing clouds are at a disadvantage.  I expect to see current CDP vendors take both approaches, even as new entrants continue to appear.  The CDP market won't get any simpler but buyers should have increasingly clear choices, so buying a CDP may become a bit less complicated.

What Happens When Everyone Has a CDP?

October 22 was a landmark day for the CDP industry.  We reported three significant announcements:

• Mastercard bought CDP and loyalty vendor SessionM which will power Mastercard services for its merchant clients
• Teradata launched its own CDP offering, Vantage CX, which will anchor Teradata’s marketing software product line
• Tealium partnered with marketing agency Wunderman Thompson, which will use Tealium AudienceStream to manage customer data for agency clients.

Each announcement follows a string of similar previous developments.

SessionM/Mastercard: most CDP acquisitions have been made by companies that are not primarily marketing software vendors. Arm (microprocessor technology) bought Treasure Data; Dun & Bradstreet (B2B data sales) bought Lattice Engines; Kabbage (small business finance) bought Radius; Anaplan (business planning) bought Mintigo; Informatica (data management) bought Allsight; Equifax (credit bureau) bought Datalicious. The exception that proves the rule is Salesforce’s purchase of Datorama, which it uses for marketing performance measurement, not as a CDP.

I believe the reason for these deals is that the buyers want to offer services that depend on unified customer data, but find it’s easier and cheaper to buy the necessary technology than to develop it internally. Note that it's truly a build/buy choice: many of the buyers already have extensive customer data management operations, so they probably could have built the systems for themselves.  They simply realized that buying was the better option.

The implications of this are substantial.  Competitors of the acquiring firms will feel pressure to offer similar services that help their clients deploy customer data.  Such services can be important tools for retaining clients, since switching customer database providers is painful at best. For CDP vendors, these deals are a promising exit path from a crowded industry which will only become more competeitive (see below).  For marketers, these deals mean their companies gain new options for access to a CDP.  This is especially true for small and mid-size businesses that might lack the resources to buy and integrate a CDP on their own.

Teradata: major marketing software vendors have chosen to build their own CDPs rather than buying one. This list includes Salesforce, Adobe, Oracle, Microsoft, IBM, SAP (sort of) and SAS (although they don’t use the term).  Their decisions to build their own CDPs are a bit perplexing, given that most have made many other acquisitions to fill gaps in their product lines.  My best guess is they like to buy companies that give them a substantial position in a new market, and even the largest of the pure-play CDP vendors are too small to catch their fancy. It might also be that building a CDP looked simple to them, so they all decided there’s not much reason to purchase a CDP for technology alone. The time it has taken them to deliver proper CDPs suggests it may have been harder than they thought.

The marketing software vendors’ delay in delivering CDPs has given other vendors opportunities that might not have existed had the marketing software companies moved more quickly. But with the marketing software vendors products now finally reaching the market, that era is ending. This will make life more difficult for the independent CDP vendors.  I still expect many of the independents will survive by developing systems tailored to particular industries, regions, or client sizes.

Wunderman Thompson: many ad agencies have decided to partner with a CDP vendor rather than purchasing one outright. The analysis gets a little confusing here because the big ad holding companies have been purchasing data-based marketing agencies: Dentsu bought Merkle, IPG bought Acxiom, Publicis bought Epsilon. But those agencies have themselves generally resold marketing technology rather than building or buying their own. This is probably a good choice: although they have considerable skill working with customer data, they have limited software development capacity. So it makes more sense for them to rent technology from others.

Assuming they continue to work with other vendors' technologies, the agencies represent a market for CDP vendors that won’t go away. If anything, it’s likely to grow as more agencies offer customer data-based services. But agencies have special needs and are often very cost-sensitive. So only a handful of CDP vendors are likely to get much benefit.

These three lines of development all point in the same direction. The path leads to a world where unified, sharable customer data is available to nearly every organization: that is, a world where every company has a CDP.  Nirvana, you say?  Yes, possibly, for CDP users.  But remember that CDP might be a stand-alone system, part of a marketing software suite, embedded in operational systems, or provided as part of an agency’s service. So it's not necessarily great news for CDP sellers.  

The broad availability of CDP functions affects users in other ways. When CDP functionality was available only from specialist vendors, the choice of a CDP was based on finding the best system (or, more precisely, the system that best fit each buyer’s particular requirements). But when CDPs are baked into larger software and service offerings, the quality of the embedded CDP is one of many considerations in selecting a vendor. In fact, the CDP itself may be invisible, as buyers base their choice on which vendor can best meet their business needs. If the potential vendor can meet those needs, its CDP must be adequate. If the potential vendor isn’t a good fit, it really doesn’t matter whether the fault lies with their CDP or some other component.

Note that there will be exceptions to this new rule. Large enterprises are likely to assemble their own collection of best-of-breed components, including a stand-alone CDP to integrate data from disparate sources. Mid-size firms who don't want to commit to one comprehensive marketing suite may prefer broad-scope CDPs that combine centralized data, analytic and personalization functions while integrating with external delivery systems.

What doesn’t change are the needs for users to define their requirements, to accurately assess which vendors will meet them, and to deploy their choice effectively. These tasks are closely related: you can’t define requirements, assess alternatives, or deploy effectively without understanding what the CDP needs to do. So education and training of CDP users will remain important regardless of how CDPs are purchased or delivered.

Another way to look at it is this: CDP has been cruising through the Gartner Hype Cycle for the past four years. Each hype cycle stage implies a common customer question*.  Here's what we’ve seen for CDP:

• at the start, when the technology is unfamiliar, the question was: What’s a CDP?
• during the peak of inflated expectations, people had some understanding of the concept, so their question became: Do I need a CDP?
• as CDP enters the final stages of disillusionment, enlightenment, and productivity, people accept that they probably need a CDP and ask the next logical question: How do I best use my CDP?

Of course, different markets and individual users are at different stages in their own CDP journey, so we still get all three questions. But it’s clear that the third question – often phrased in requests for use cases, best practices, and maturity models – is becoming the most important. I’ve no doubt that the industry will provide more answers.

____________________________________________________________________________
*This is my interpretation, not Gartner’s.

Privacy Regulations Will Lead to Advertising Innovation

Naysayers doubted that the European Union’s General Data Protection Regulation (GDPR) and ePrivacy rules, California Consumer Privacy Act (CCPA) and related privacy regulations would have any real impact on the flow of consumer data. But it’s now clear that they will, as European regulators show they will interpret the rules to require meaningful consent to data sharing, will treat cookies as personal identifiers, and will assess significant fines for data breaches. The imminent effective date of CCPA, which rather surprisingly survived without being watered down or preempted by weaker federal regulation, confirms that privacy can not longer be ignored by data collectors. Meanwhile, actions by Google, Microsoft, Apple and other major browser are putting still more barriers in the way of data business as usual.

None of this marks the end of the advertising industry or even of targeted online advertising. Advertising was a big and important business before the Internet existed and would remain one if the Internet went dark tomorrow. Pre-Internet advertising wasn’t as targeted or measurable as today’s display and social ads, but it did work. A total end to distribution of third party personal data would still leave Internet marketers able to target based on context, content, and in-session behaviors. This probably wouldn’t be quite as effective as individual-level targeting, although I don’t recall any studies that prove this.  It’s even possible that closer attention to ad targeting methods would reduce fraud enough to more than compensate for the loss of third party data.

In any case, the hunt is on for mass advertising audiences to replace audiences based on third party personal data. The recent merger of content-based advertising leaders Taboola and Outbrain was driven in part by the desire for greater scale in content ads (those click-bait teasers that lead you off-site at the bottom of many Web pages). The rebirth of out-of-home advertising (billboards, wall posters, in-store displays, and clever niche products such as elevator and gas pump ads) as a digital channel opens another mass medium. Above all, the various forms of individually targeted TV advertising promise new levels of personalization and tracking in a medium that already has near-universal reach. Other types of video, available through YouTube, Instagram, TikTok, Snapchat, Twitch, and much else, are delivering new mass.

Taken together, these emerging options promise a new Golden Age of Advertising Creativity, as marketers explore their potential and evolve the most effective approaches to each. It’s true that a vastly more fragmented media landscape will be harder for marketers to manage. But it also means that fears of a Google/Facebook duopoly controlling all access to new customers are clearly overblown. Regulatory pressures, changing consumer attitudes, and new competition should further deflate them every day.

What does all this mean for martech in general and Customer Data Platforms in particular? The most directly affected martech systems are Data Management Platforms (DMPs), whose core function is to manage the third party data-based customer profiles that are quickly becoming extinct. Marketers who had hoped the DMP would manage all their customer data had already lost much of their interest when they discovered how limited DMP capabilities really were.  Many DMPs have repositioned themselves as CDPs, although not all have the technical capabilities required to meet the RealCDP requirements.*

The implications for CDPs are more positive, although not quite as rosy as sometimes suggested. There’s a common argument, which I frequently make myself, that the loss of third party data makes first party data more important, and that’s good for CDPs because they primarily manage first party data. This is true on some level but shouldn’t be overstated. The third party data that populates DMPs is mostly used for acquisition marketing, while the first party data in CDPs is mostly for marketing to current or former customers. So switching focus from DMP to CDP would leave a significant gap in many acquisition marketing programs.

As we’ve just seen, marketers will plenty of other ways to reach new audiences even after today’s third party data-driven advertising is gone. But there is one way that CDPs can directly support acquisition programs: by making it easier for companies to share first party data with each other, creating what is usually called second party data.

Second party data can refer to any first party data that is directly shared with another company (the second party). The CDP supports this by creating an extract file of first party data that can be sent to the second party. If that’s all that happens, it’s no different from any other extract.

But often second party data is created by comparing the customer lists of both parties and finding shared customers. The trick is often that the two parties don’t want to expose information about customers they don’t share.

One way to achieve this is to send a copy of both customer lists to another company that both firms trust. That company compares the two lists, finds matches, and returns whatever information the parties have agreed to share. Alternatively, a common identifier such as email address might be run through a standard “hashing” algorithm that will yield a unique result for each input but not reveal the actual identity information. Both parties use the same algorithm so that all records with the same identifier yield the same hash code and are identified as a match. Records that don’t match will generate different hash codes which are meaningless to the other party. With this sort of processing, there’s no need for another trusted company to do the matching because customer identifiers are not actually shared. Each party keeps a record of the original identifier and the resulting hash code, so it can tie the codes back to its actual customer records.

Here’s a practical example. Suppose a hotel chain and airline agree to identify loyalty program members on each others' lists.  That is, the airline would learn which of its customers were members of the hotel loyalty program and the hotel would learn which of its customers were members of the airline program. They can do this with the type of matching just described, adding a "partner loyalty member" flag to appropriate customer profiles.  The airline could then make a special offer to hotel loyalty members and the hotel could make a special offer to  airline program members. Each party could also send acquisition promotions on behalf of its partner to its own customers who are not  members of the partner's loyalty program. So long as each party sends the promotions to its own members, the other party never learns anything about them.

The minimum role of the CDP in this sort of second party sharing is to generate a customer list with the required data. Some CDPs can also find direct matches (acting as the trusted third company) or generate and match the hashed identifiers. CDPs with these added functions are in a position to benefit as the loss of third party data makes second party data sharing more important to acquisition marketing programs.

In sum, privacy regulations won't kill targeted marketing.  They'll actually lead marketers to expand the methods they use, promoting a healthy diversity and weakening the much-disliked Google/Facebook duopoly.  Customer Data Platforms will benefit as first and second party data play larger roles in the marketing mix. 

_____________________________________________________________________________
*load all types of data, retain all original details, store the data indefinitely, build unified profiles, and expose the profiles to any external system.

It's CDP Time for Marketing Cloud Vendors

Adobe, Salesforce and Oracle all made announcements regarding Customer Data Platform products this week. None are world-changing: Salesforce described a planned extension of Customer 360; Abobe announced triggered journey campaigns that draw on its “real time CDP”, and Oracle described CDP services from systems integrators.  But the fact that all three vendors are addressing the topic raises some interesting questions.

Why does this matter to technology and marketing professionals? Customer Data Platforms have been a hot topic for the past three years. The reason is simple: they promise to solve a pressing problem that has not been solved by anything else. That problem is the need of marketers (and others) to combine data from all sources into easily accessible customer profiles. Those profiles are needed for accurate targeting and consistent, satisfying customer experiences.

The problem is unsolved because alternate solutions fall short in different areas.  Data warehouses are largely limited to structured data.  Data lakes are not unified or easily accessible to non-technical users.  Data Management Platforms are limited to summary data about mostly anonymous individuals.  CRM and marketing automation systems don’t easily combine data from external sources.

By contrast, CDPs promise to ingest all data sources, retain all details, and share the resulting profiles with any system that needs them.  But CDPs have been developed by relatively small specialist vendors, which many enterprise buyers are reluctant to consider. So having Salesforce, Adobe, and Oracle promote CDPs will prompt more enterprise buyers to give the category serious consideration.

It also doesn’t hurt that Forrester issued its first CDP wave this week, that Dun & Bradstreet just bought CDP Lattice Engines, and that Martech Advisor’s Talking Stack podcast devoted an entire session to the topic  (humble-brag disclosure: I’m a panelist on Talking Stack).

So, yeah, CDP is getting a lot of attention right now.

What’s new in these announcements and what’s not? Adobe, Salesforce and Oracle have all previously announced something that they positioned as addressing the needs met by a CDP.

• Adobe’s original approach was to map a single customer ID across all its systems and create transient customer profiles by pulling together data on demand. It changed direction in March 2018 with news that its Experience Platform would create persistent profiles.  It released this a year later and described it as including a "Real-Time CDP".  Nothing about that changed with yesterday’s announcement, which described an Adobe Campaign application using the CDP data.  All this is separate from the Open Database Initiative, a still-undelivered project announced last September to build shared database model with Microsoft and SAP.

• Oracle announced CX Unity last October. It included a persistent data store from the start but is just now starting beta deployments. The latest Oracle announcement describes collaboration with Accenture and Capgemini to provide services around CX Unity projects.  It doesn’t include anything new about the product.


• Salesforce’s Customer 360, also announced last September, was another master customer ID used to access source system data on demand. Salesforce now says they’ll release that product this November. The bigger news is they’re developing a next-generation Customer 360 that will store its own data, bringing them in line with Adobe and Oracle. There’s no release date although they hope to start pilot projects this fall.

It's taken a while but all three vendors now acknowledge that a CDP must store its own data. That will remove some confusion from the CDP market.  Cynics might say it also confirms that software vendors define user requirements based on what their systems currently do, not what users actually need. It’s hard to interpret the CDP story in any other way, since the need for a persistent data store has always been clear to anyone who tried to support core CDP use cases such as attribution, prediction, and journey analysis.

What benchmarks should these CDP products be measured against? The CDP Institute (which I head) believes that buyers expect a CDP to ingest all types of data, capture the full detail of the ingested data, retain that data as long as the user wants, assemble unified customer profiles, and make the profiles available to any external system. We codify those as requirements in our RealCDP^TM certification program. The original Adobe and Salesforce approaches certainly didn’t store the data and had other limits about the types and detail they captured. On the other hand, they did include real-time access to current source data, which is not part of RealCDP but is important for many CDP use cases. Many other CDPs also provide real-time access as well as segmentation, predictive modeling, personalized message creation, and sometimes even message distribution. Oracle, Salesforce, and Adobe have separate products for most of those functions so they are not part of their CDPs.

What should we look for next from these vendors? The main thing to watch is how quickly the announcements turn into released products. Only then will buyers be able to judge the details of how well these vendors deliver on their promises and how their offerings compare with other, more mature CDPs. In particular, pay close attention to how easily each system connects with other products. Because the marketing clouds are built from acquisitions that remain technically separate, integrations for each component are developed and delivered in whatever sequence the vendor thinks best.  Connections to other vendors’ systems are an even lower priority. Buyers will often be left to develop their own connectors using a CDP API – which should itself be examined closely to see what it does and how hard it is to use.

Once the vendors flesh out the core CDP capabilities of their offerings, the focus will shift to improvements in user experience. This includes end-user functions such as segmentation and more technical capabilities for connecting to data sources and destinations. Reducing the technical work required to set up and maintain a CDP can have a significant impact on time to value and operating cost.  More important, it can help the CDP achieve its mission of including all data and sharing it with all other systems. Also expect the vendors to create industry-specific packages with prebuilt data models, connectors, predictive models, workflows, and reporting. These will further ease deployment and help the vendors to compete with industry-specialist CDPs.

In most of the CDP market, vendors are extending their systems by adding more activation functions such as marketing campaigns, personalized message selection, and message delivery. Many users are eager to buy one system that combines as many functions as possible. But because the marketing clouds offer these functions as separate modules, they are unlikely to expand their CDPs in that direction.  This will probably reduce their competitiveness in the mid-market.

Who else might toss their hat into the CDP ring? There have been four significant CDP acquisitions to date: Datalicious/Veda by Equifax (2016); Datorama by Salesforce (2018); Treasure Data by Arm (2018), and Lattice Engines by Dun & Bradstreet (2019). Salesforce is the only marketing cloud vendor on this list and they've positioned Datorama as a campaign analytics product, not a CDP.  Since Oracle and Adobe are far along in their CDP development, neither is likely to purchase an independent CDP vendor – although that might happen if they feel pressure to deliver a mature CDP more quickly. That Equifax and Dun & Bradstreet have bought CDPs suggests other data compilers might do the same, with the goal of adding more value than just their data.

Other potential acquirers include ad agency holding groups and consultancies who want to bulk up their data management capabilities. Moving from the other direction, companies that offer message delivery and interactions, such as email delivery, Web site personalization, mobile app development, call center, and customer support systems, might add a CDP to expand their footprint, make their products harder to replace, and ultimately let them add delivery in other channels. Case in point: unified marketing, sales and support vendor Freshworks recently purchased Natero, a customer success system with CDP capabilities.

But the big clouds looming over the entire customer data ecosystem are, literally, the big clouds: Amazon Web Services, Google Cloud, and Microsoft Azure. So far, none has shown much interest in selling customer data management tools, although they already host plenty of CDP data.  There’s a reasonable chance these vendors will eventually enter the CDP market.  But that would be a few steps up the value chain from their current offerings, which still focus primarily on data storage. The cloud services vendors are starting that climb, adding cloud connectors, data transformations, in-database analytics, machine learning, and reporting tools. Google Cloud’s purchase of Looker is a recent step in this direction. Data quality services such as address standardization and master data management could be next.  There's a way to go before these vendors are ready to add the specialized features needed for a CDP. They may also find themselves constrained by anti-trust and privacy issues. But don’t rule it out.

What do these announcements mean for current CDP vendors? At a fundamental level, these announcements confirm that there’s a real need for CDPs and that the CDP model of building a separate, persistent database is correct. That’s no small thing, given the fear, uncertainty and doubt that the marketing cloud vendors have previously spread about both. The result is to expand the market, since more potential buyers will now accept CDP as a valid option.

This does not mean the cloud vendors have given up their efforts to shape the conversation. They are now trying to redefine CDP as part of a larger integrated package – that is, of systems like their own. This isn’t likely to be successful, given how few enterprises actually limit themselves to one vendor’s products. So the ability of the current CDP vendors to work equally well with systems from any provider is likely to be even more appealing.

Still, it would be unrealistic to ignore the fact that many buyers would rather buy from the marketing cloud vendors. The long lead times between preannouncement of the marketing cloud CDPs and their actual delivery will lead some buyers to delay their purchases, which is exactly the cloud vendors’ intent. But the interval will also give potential buyers more time to think carefully about what they need in a CDP, making them smarter consumers when they do start an acquisition project. A rigorous, requirements-based acquisition process will often favor the existing CDP vendors, whose products are proven and mature.

How do these announcements relate to other developments in the CDP market? The CDP market is evolving rapidly. From its initial base in retail and media, it has spread to new industries including travel, financial services, B2B, telecommunications, healthcare, and education. Industry-specialist vendors are appearing with prebuilt data models, integrations with industry systems (such as point of sale in retail or reservations for hospitality), and staff expertise. Other specialists now focus on mid-size or small businesses and in particular geographic regions.

There’s also an increasingly clear split between CDP vendors who focus on data collection, unification, and access functions and those with marketing functions for analytics and personalization. One result is that some clients deploy one CDP for data unification and another for marketing applications. Some CDPs have extended their marketing features to include delivery systems such as email engines, Web site messaging, and even DMPs.  These were previously beyond the scope of CDP products. Vendors with delivery capabilities still allow clients to use other delivery systems – otherwise they would not meet the definition of a CDP. You can see where this leads: CDPs with a full set of marketing applications become direct competitors of the marketing clouds. It also means that CDP becomes one feature among many in these products -- a change that may lead some to deemphasize their CDP capabilities and instead partner with data unification specialists.  (See this post for a deeper exploration of that scenario.)

On the buyer side, CDPs are increasingly used beyond marketing to support sales, service, and business operations. This often means the CDP becomes a shared resource managed by corporate IT rather than marketing.  Enterprise-wide digital transformation projects and increasing concern over compliance with privacy regulations have further increased involvement of central IT and compliance teams.  In general, greater familiarity with CDPs has meant that buyers have a better understanding of what they do and what to look for, making them more sophisticated consumers and helping them to navigate the growing number of products in the market.

Entry of the big cloud vendors may accelerate the trend to industry-specific CDPs by pushing smaller vendors into niches where they can better compete with general purpose systems. The big cloud vendors may push also independent CDPs to deliver comprehensive marketing functions to the mid-market where they can be more integrated and more economical than the marketing clouds. CDPs that focus primarily on data management will probably face the greatest competition from the marketing cloud CDPs, since they compete for enterprise clients where the big cloud vendors are the strongest. But those CDP vendors also have the greatest technical lead and will find it easiest to support enterprise-wide use cases.

In short, entry of the marketing cloud CDPs will accelerate industry growth and reinforce current industry trends, not change the over-all direction. For CDP vendors and buyers alike, that is good news.