Sunday, May 19, 2019

Do Customer Data Platforms Need Real-Time Processing?

Is real-time processing a requirement for a Customer Data Platform? It’s a deceptively simple question that can’t be answered without resolving two additional questions: What do we mean by real-time processing? How do we decide what is and isn’t a requirement?

Let’s tackle the definition of real time first. There are at least four different flavors of real time processing that relate to CDPs.
  • Real-time updates. This means that new data flows into the CDP and is added to the exposed customer profiles in a few seconds. The exact speed requirement isn’t clear: although one second is widely considered to be the threshold for real time interactions (based on a study done in 1968), longer lags are acceptable for sharing customer data between systems. In practical terms, customers will not be annoyed if it takes a half-minute before a call center agent learns about her latest Web interaction.  Many CDP vendors offer what they call “near-real-time” updates, where the lag might be one to five minutes. This is obviously too slow to manage an interaction like online chat. But it can support asynchronous activities such as sending order confirmation emails.
Real-time updates are quite difficult. The first step, adding a new piece of data into the CDP data store, isn’t too hard for most CDPs, which simply append each item without touching existing data. But there’s usually additional processing to update customer profile elements, such as lifetime purchase value, predictive model scores, or last purchase date. This requires finding the right profile, reading the existing data, running whatever calculation is needed, and storing the new result. That takes time. And there’s often an additional step to copy data from the profile into a separate data store that’s optimized for real-time access, such as a flat file or in-memory database. This takes time too.
  • Real-time identity resolution. This is a special case of real-time updates, where the system needs to do some processing to associate a piece of data with the right customer profile. It's needed when the new data isn’t already associated with a known identifier such as a customer ID.  It may require substantial processing to run matching algorithms that test various combinations of data against existing records. Usually this process is limited to adding the data to an existing profile, but it might also re-examine all profiles to see if the new data uncovers a connection that would allow them to be merged or split. That requires still more processing. This sort of comprehensive reexamination is usually done in a batch process that might run anything from nightly to monthly.
  • Real-time access. This means an external system can query the CDP in real time for a copy of an existing customer profile. This might be done with an API call or by accessing a separate data store. Note that the profile itself isn’t necessarily being updated in real time: in fact, companies commonly run a nightly batch process that loads profiles and next-best-actions into an online data store. The data remains unchanged until the next nightly update.
  • Real-time interactions. This means an external system can send data to the CDP and receive back a profile that incorporates that data – for example, with an updated model score, product recommendation, or marketing messages. It’s another flavor of real-time updates but is limited to activities within a single system. In practice, real-time interactions often work by moving a copy of the profile into memory at the start of an interaction, updating the in-memory copy as the interaction proceeds, and then copying the final version of the profile back into the main database after the transaction is over. This avoids the need for real-time updates of the main database. Real-time interactions are often integrated with multi-step campaign flows so the user can guide the interaction process.
It’s important to recognize these distinctions and to clarify which are included when you’re discussing a real-time CDP. All systems in the CDP Institute’s Vendor Comparison Report say they do real-time access while only half do real-time interactions. Real-time data loads are almost universally available but the report doesn’t capture which systems can make the data available in real time. Nor do we ask about real-time identity resolution.

This brings us to the second question, of how to decide whether the CDP definition should include real-time capabilities (and, if so, which ones).  This matters because the market is confused by the variety of companies claiming to be a CDP.  A clear, widely understood definition is essential to reducing that confusion.

One approach is to start with current user expectations. CDP Institute research shows that users’ top priorities are data collection and unification. Real-time access and real-time recommendations rank far down their list. We can’t tell whether users include real-time updates in their definition of data collection.  My suspicion is that most do not. But I haven’t seen any reliable research.

A second approach is to look at what’s included in existing CDP systems. As we’ve already seen, all systems in our Vendor Comparison Report offer real-time access and about half offer real-time interactions. While we don’t ask directly about real-time updates, we do know that quite a few offer near-real-time updates, which presumably means they don’t do full-real-time updates. You may notice there’s some circular logic here, since we have to decide in advance which systems to examine.

A third approach is to look at common use cases. The data are again ambiguous because top applications like predictive modeling and personalization can benefit from real-time updates but don’t require it. Some standard CDP use cases do require real-time updates, such as providing call center agents with information about recent Web behaviors. But plenty of others do not, ranging from customer journey analysis to email audience selection.

Since none of these options offers much guidance, how do we decide?  I'll argue that we ultimately want to make the decision that best serves CDP users.  Many users need real-time updates and interactions but many others do not. If we want the CDP category to include systems that meet the needs of both groups, we should include non-real-time systems in the category.

But that's not enough.  We also need to help users understand which CDPs provide real-time capabilities and which specific capabilities are included.  CDP Institute is addressing this need with its RealCDP program, which will gather and present information about real-time capabilities.  We’ll also add real-time updates to the Vendor Comparison report. Stay tuned for more information.

Saturday, May 11, 2019

Jamie's Excellent Privacy Adventure

Jamie knew something was wrong when his alarm clock didn’t greet him by name. Weird, he thought.

Things quickly got weirder.

The water in the shower was too hot.  The traffic report mentioned an accident nowhere near his route to work. When the radio played a song he had specifically banned from his play list, he knew something serious was wrong.

“Clock, please run a system check,” Jamie muttered, annoyed.


He said it again, louder, this time staring directly at the device on his night table.


System must be down, he thought. I’ll deal with it later.

But then he looked out the window. The cars all had human drivers and pedestrians were not staring into their phones. He knew he had a much bigger problem.

A multiverse flip.

He'd read about it on the news.  Result of global warming or wireless signal overload or overlapping artificial realities. No one really knew.  What they did know was that people woke up in a different world from the one they’d fallen asleep in.  It seemed to be happening more often. 

Jamie picked up his phone. It didn’t recognize him but on-screen instructions let him unlock it with a thumbprint. He dialed LOST – League of Strayed Travelers – a service that spanned multiverses with enough cross-traffic to cooperate. He was lucky; the call was answered. By a human, which felt odd. But apparently that was the world he was in.

A quick conversation established that Jamie was indeed lucky: the local LOST staff included someone from his home universe. She would be Jamie’s guide. Soon a car pulled up – this one with both a driver and passenger.  Out hopped Emily, a cheerful young woman who quickly began explaining what was different.

“This is a world where privacy has been taken very seriously. Businesses and government are banned from storing any personal information beyond what’s needed for security purposes. That’s why your smartphone recognized your thumbprint. But look more closely at the phone and you’ll see it doesn’t have a record of your past calls or contacts. In the same way, we still have search engines and social networks and ecommerce, but they don’t personalize their services. It’s annoying but you get used to it. What I miss more is voice recognition, which is entirely forbidden as inherently invasive. Alexa was my best friend!”

She glanced wistfully at Jamie’s alarm clock, which he now realized was no smarter than a rock.

“But it gets much worse,” Emily continued. “Without gathering personal data, companies can’t train AI systems for things like self-driving cars, energy conservation, or personalized medicine. So we have more accidents, more pollution, and a vastly less efficient economy. Everyone has less free time and is poorer. Crime is worse because people have to carry cash and video surveillance is strictly limited. And, ironically, social media is still filled with bullying and misinformation – it turns out those have nothing to do with privacy and everything to do with human nature.”

Emily frowned briefly. Then her face brightened.

“But there’s good news, too. We’ve been researching the multiverse flips and have experimental devices that can move you from one world to another. We can’t guarantee where you’ll end up but have enough return traffic to be reasonably sure it won’t be too terrible. At least, most of a time. So there’s a risk but we can give it a shot if you like.”

On the ride back to the LOST office, Jamie and Emily chatted more about this universe, the flipping device, and their previous lives. Turned out she was from New Jersey. And married. By the time they arrived, Jamie had decided to try the new machine.

Emily gave Jamie one final smile as she closed the lid on the flipper. It was dark and warm and filled with white noise.

Then the lid was up.  Jamie was in a different room. New faces. No Emily.

A serious-looking man reached into the flipper and pulled out a package. “May I have this? It’s how we share information across the multiverses.”

The man opened the package and scanned the top document. “I see you’re Jamie. Welcome. I’m Giovanni.”

Giovanni took a closer look at the documents. “Looks like you come from a place where anything goes, privacy-wise. It’s very different here. We believe that people own their own data. But we respect their liberty, so any use is allowed if they give consent.”

Jamie was a bit groggy as he stepped from the flipper. “How’s that working out for you?”

Pretending not to notice that Jamie looked tired, Giovanni smiled and pointed to a seat. “If you please.”  His face turned serious.

“Mixed results, to tell the truth. Most people consent to pretty much everything. Their experience is much like yours – many free services but little privacy. Data breaches are common and much of the data is inaccurate. But the majority put up with it for the convenience and an occasional discount coupon.”

“And everyone else?” asked Jamie.

“There are two groups. Some people are privacy zealots, pure and simple – they won’t give up their data on principle. They pay extra for services that others get for free, and many services aren’t available to them at all.  They’re often left out of business and social events and have harder lives as a result.  Many live off the grid doing things crafting Faraday cage handbags. At least we don’t hear their complaints.” He did not seem amused at his own joke.

“The others are people with enough power and money that they can easily afford the cost of privacy-enhanced alternatives. They have staff or bots to maintain a social media presence without exposing their own data directly. They use special devices and software that hides their identity, regularly erases their data, and maintains separate personas for different purposes. They get the best of both worlds: privacy for themselves and convenience based on the data of others.”

Jamie was puzzled. “Why is privacy protection expensive? I’d think most of the solutions are based on software.  That should be nearly free to run for everyone once it’s built.”

“I thought so too,” sighed Giovanni. “But it’s a lizard-and-the-egg sort of thing: most people won’t pay even a little extra for privacy, especially if they’re rewarded for giving it up. So the market for privacy-enhanced systems is fairly small, which means manufacturers must charge a higher price per customer, which makes the market still smaller, which drives the prices still higher. It ends up as a luxury good.” Giovanni sighed again.

“Yes, I can see how that works out,” said Jamie. He thought for a moment. “Look, it’s nice to meet you but this isn’t my home universe. Can I go back into the flipper and try again?”

“Of course,” replied Giovanni. “With your permission.”

There was time for a quick lunch while Giovanni prepared another information package. Jamie climbed back into the flipper, relaxed for a moment, and the lid was up again. Another room. More new faces. .

Now a veteran, Jamie sat up and handed the information packet to the nearest person. A badge clipped to his shirt showed his name was Tim.

“Hi Tim. I’m Jamie. Where am I this time?”

Tim opened the package and read the cover sheet. He paused a moment.

“Not where you want to be, I’m afraid. But not a bad place. How much do you want to know?”

Jamie was disappointed but Tim seemed friendly enough. This room somehow seemed more cheerful than the last one.

“Well, I’m here, so I might as well find how things work. You’re the first place where people are wearing name tags. What’s up with that?”

“Happy to explain,” said Tim. “But where are my manners. Would you like a glass of wine?”.

“I prefer beer,” replied Jamie. They walked to the front of the building, where a pleasant café fronted the sidewalk. They sat with their drinks.

“Unlike the last two places you’ve been, our universe believes that some data should be shared with everyone while other data should always be kept private. Deciding where the line falls isn’t easy and I can’t say we always get it exactly right. But we keep making adjustments over time. The good thing is people mostly know what to expect and are treated fairly without taking extraordinary measures to protect themselves.”

Jamie didn’t get it. “How does that work? Everyone is wearing name tags, so clearly that’s something you’re required to share. But the tags just show first names. How do you deal with more sensitive data?”

“Excellent question,” smiled Tim, warming to his subject. “So few people really care. Have another drink.”

Tim drained his own wine glass and ordered another. Jamie was still working on his first beer. Tim continued.

“We apply what people in your universe call ‘privacy by design’.  Our badges do more than show our first name: they contain details that are shared on an as-needed basis. For example, when we entered the cafe, a sensor queried my badge and told the server was that I’m allowed to order a drink. But that’s all he learned; it didn’t tell him my age, let alone the name, address, birthdate, and biometrics he’d get from your driver’s license. And if there were some other reason I wasn’t allowed to order a drink – say I was already drunk – it wouldn’t have said that, either. It would just have told the server not to serve me. So my privacy is protected even while drinking restrictions are enforced.”

Jamie pushed away his beer. “So that badge knows that you’ve been drinking? I don’t think I’d want anyone keeping a log of my alcohol consumption.”

Tim looked at his own glass. “Neither would I. But the badge doesn’t keep a log; it just monitors my blood alcohol level. And it doesn’t share that unless there’s reason, like determining whether I can legally order a drink.

Jamie relaxed a bit.  Time continued. 

“There’s lots more information that the badge or other devices do log. My smartphone knows my location history, the Web sites I’ve visited, search queries I’ve made and much more. It uses those to make my life easier, same as in your universe. But, unlike your universe, the data never leaves my device. That way no one can use it in ways I don’t control. When someone wants to serve an ad to people who like red wine” – he lifted his glass – “they just query devices until they find a profile that fits the description. The profiles aren't stored outside the devices and there's no record of which device an ad was served to. That makes things a little harder for advertisers, who can’t control how often one person sees the same ad or connect ad views to subsequent purchases. But it still allows most kinds of behavior- and profile-based personalization.  The economy manages to function.”

Tim stopped short. “Sometimes I repeat myself. I hope I’m not boring you.”

He was, a little. But Jamie could see he loved the topic. “Well, I do want to try to get home. But maybe there’s something here I can bring back that would be useful. What else do you think I should know?”

Tim gathered his thoughts. “To quickly flesh out the picture, the same principles apply to other types of data. So, my phone knows where I live and where I am now, which lets it connect with navigation software to tell me how to get home. But it only asks the central navigation system for a route, without telling it who’s asking. So the navigation system doesn’t know anything about my movements over time. And we do let people view ads for payment, but there are strict rules against trading away personal data.”

“Who makes these rules?” asked Jamie. “In my world we’re pretty skeptical of regulators.”

Tim gave him a sharp look. “So are we. The rules come from a mix of legislators and agency staff.  There's plenty of lobbying from all sides. As I said before, there’s lots of disagreement and they don’t always get things right. But everyone starts from the premise of putting individual interests first and business interests second. Turns out that’s a good guide for many decisions."  Tim paused for a breath.

"And, yes, social interests like public safety come into play. So I can't drive a car if my badge says I'm legally drunk, although the badge doesn't give the car a reason.  I can actually override that rule in an emergency, but then the car also notifies the authorities and turns on special tracking devices.  So, yes, it's complicated.  But just because it’s hard doesn’t mean we shouldn’t try to make it work. You’ve already seen how poorly things turn out in worlds that apply simple solutions instead.”

“Indeed I have,” said Jamie. “I certainly don’t think my universe has it right.” He paused. “But home is home and that’s where I belong. Can we try the flipper again?”

“Of course,” replied Tim. They left the café without paying. Tim winked at Jamie. “Don’t worry. They'll charge my badge. Anonymously.”

Once more into the flipper.

Jamie’s phone buzzed to life before the lid was raised. He knew he was home.

“Good morning, Jamie,” the phone said. “You’re late for work. Shall I call you a car and let the office know you’re on your way?”

Jamie shut it off and opened the lid.

Tuesday, April 30, 2019

Privacy-Protecting Systems Are The New Green

Let’s take a break from Customer Data Platforms to do some trend-spotting. I spy with my little eye…privacy systems!

Specifically, there's a crop of systems that are privacy-safe alternatives to dominant social, search, email and other common consumer technologies. One well known example is DuckDuckGo, which positions itself as “a search engine that doesn’t track you”.  But there are plenty of others.  Some that have recently caught my attention include:
  • Brave, a browser that lets users decide which ads they’ll see and blocks advertisers from seeing behavioral details
  • Anagog, which let mobile apps track behaviors and make predictions while keeping all data on the device
  • ProtonMail, an encrypted email service  (it's one of a dozen alternatives in that market)
  • Vero, an ad-free social network
  • Chatterbox, a privacy-safe smart speaker for kids
  • Aegis One “mini-computer” for anonymous Web browsing, from a company so privacy-conscious that they apparently don’t publish their contact information (which may take things a bit too far)
A thorough search would surely turn up more examples. You could also add add products whose purpose is privacy, like ad blockers or proxy servers; the gazillion contenders in the pay-people-to-watch ads industry; privacy-enhancing extensions to standard products such as Google Chrome and Firefox; and, perhaps most prominent, the privacy-centered positioning of Apple.

In other words, privacy-protecting systems are a big and growing business.  Privacy is the new green: the cool virtue signifier for consumers and businesses alike.

This is worth noting because industry conventional wisdom has long held that consumers don’t really care about privacy, despite claims to the contrary. The core evidence has been that even people who say they care about privacy are willing to give up their personal data for the tiniest of incentives, whether monetary discounts or convenience. There’s still plenty of data along those lines, such as this Mulesoft study, which found that 49% of consumers would share personal data to get personalized service.  But the same surveys show a substantial minority don’t want their data tracked at all and many have stopped using big social media platforms due to privacy concerns. (See also this Harris Poll report for another set of similar statistics.)

It’s hard to find consistent data over time but it’s a safe bet that this GlobalWebIndex report is correct that consumers' privacy concerns have grown sharply in recent years.

The implications of this are intriguing. There’s a reasonable possibility that we’ll soon gain access to an alternative universe of online systems that protect rather than destroy consumer privacy. If government regulators finally step up their protection of consumers – as is already happening with laws like GDPR and the California Consumer Privacy Act – these systems will have a significant head start over existing products, not to mention vastly more credibility. The result could be a tipping point when network effects kick in and privacy-centric systems suddenly pull a mass audience away from the current, data-fueled incumbents.

That’s still a long shot, if only because the incumbent firms have huge revenues that give them the resources to fight back. But a fundamental change in consumer attitudes could make their brands so toxic that no amount of investment would save them once consumers recognize there are viable, privacy-safe alternatives. 

How will you know if this is actually happening? Keep your eyes out for three things:

  • funding announcements from venture capital firms that specifically cite the privacy-preserving features of their investments
  • evidence that consumers are paying real attention to privacy, based not just on surveys about attitudes but on actual behaviors (such as the decline in Facebook use, which is already happening)
  • a Scott Brinker/Terry Kawaja style logoscape of privacy-enhancing versions of standard consumer technologies

If you’re looking for a much deeper analysis of Internet privacy and other trends, take a look at the Mozilla Foundation’s recent Internet Health Report.

Sunday, March 03, 2019

Informatica Buys AllSight and What It Means for the CDP Industry

Informatica announced last week that has purchased AllSight, a Toronto-based Customer Data Platorm vendor. This is interesting on several levels:
  • A CDP vendor is being bought by an established enterprise data management firm. This is a normal event in the development of a software category: products are developed as single-purpose "point solutions" and then, after the category is validated and requirements are standardized, the capability is assimilated into larger suites by acquisition (usually) or internal development. The most surprising thing about the CDP industry so far has been that this hasn’t happened very often.  There have been some deals but the two biggest each came with an asterisk: Salesforce bought Datorama but positions it primary for campaign measurement and Arm Ltd., which purchased Treasure Data, is not a typical enterprise software vendor. By contrast, Informatica is very much a leader in enterprise data management, with a major presence in integration, Master Data Management (MDM), governance, and security. The purchase confirms what we already knew: business managers recognize they need unified customer data and enterprise software vendors will offer them solutions.
  • It illustrates the difference between MDM and CDP. This is one of those questions we still get fairly often. Informatica lays out the contrast quite nicely: they characterize MDM as limited to highly governed, structured data that delivers the “best version of the truth” about master objects (customers, products, supplier, etc.), while AllSight stores transaction and interactions in addition to the master objects, supports unstructured as well as structured inputs, and allows multiple data views and matching options. AllSight is an exceptionally powerful CDP, with features including machine learning-based identity matching, natural language-based information extraction from unstructured data, a graph data store for complex relationships among objects, ability to create derived variables, and high scalability. These are not found in all CDPs, which is probably one reason Informatica selected AllSight in particular. But all CDPs share the core CDP capability of storing all types of customer information without losing any of the details, which is beyond the scope of MDM.
  • It confirms the importance of a persistent data store. This, like storing all information, is part of the core CDP definition. Some enterprise software vendors still argue it’s enough to connect personal identifiers in different source systems and then to pull data from those systems as needed. That is how Informatica has been doing things, apparently. But they’ve now described a progression from “distributed” customer views (disconnected systems) to “federated” views (data remains in transactional systems) to “consolidated” views (data is stored centrally and synchronized with source systems). Reasons they cite for doing this include scalability and real time performance; I’ll add greater consistency, easier control, visibility, and ensured access to historical data.
  • It shows use of CDP beyond marketing. Informatica sells primarily to IT staff, which in turn supports all corporate departments, not just marketing. In their briefing on the topic, Informatica made clear that a major reason for buying AllSight is to access the funds controlled by marketers and analytics teams. But they also argued that AllSight will give them more to sell to IT staff, which will use it to better manage large volumes of unstructured data, prepare analytical data sets, add insights based on artificial intelligence and machine learning, and support specific applications such as risk and compliance. Growth to support new use cases and departments is another natural business development for any category.
  • It gives a literal example of “CDP Inside”. This is my shorthand for recognizing that CDP functions are increasingly available within larger systems, in addition to being stand-alone products. This is arguably the most important trend in today’s CDP industry.  Let's explore it in more depth.  
“CDP Inside” matters because it’s inevitable as the market matures. So far, the main reaction among vendors and analysts has been to debate which products are properly labeled as CDP systems. Not surprisingly, each CDP vendor wants a definition that matches its own configuration. But those configurations range from products that only build a unified, sharable customer database to products that build the database, analyze its contents, and use it to create personalized marketing messages. The resulting definitions are thus incompatible.  That's a bit of a problem.

The bigger issue is that even the broadest definition excludes other products that include CDP capabilities. Informatica is good example: AllSight didn’t stop being a CDP just because Informatica bought it, but it’s hard to argue that Informatica should now call itself a CDP system as a result. The same question will appear when Salesforce, Oracle, and Adobe finally deliver proper CDP capabilities – which will doubtless happen, sooner or later. Rather than calling every system with CDP capabilities a CDP, I think it's more useful to define core CDP capabilities and then to judge every system’s CDP features against that list. This is another normal market development: companies choose all the time between buying “best of breed” point solutions and buying larger suites that include similar functions. Companies with the most demanding requirements and most sophisticated users often find themselves purchasing “best of breed” products while those with simpler needs often find the product embedded in a suite is good enough.

CDPs are slightly different from other suite components because many suites are connected internally but isolated from external systems.  CDPs are by definition required to connect with external systems.  This means a suite with CDP functions wouldn’t qualify as having a “CDP inside” if those functions couldn’t also connect with external data sources, analytical, decision, and delivery systems. But there’s no technical reason that suite vendors can’t provide systems capable of such connections and their importance is probably well enough understood that most suites will provide them.  Conversely, suite vendors are under increasing pressure to open their systems to integration with external products.  This may make it easier for “best of breed” CDPs to survive than it has been for other “best of breed” products in the past.

Of course, even accepting the notion of “CDP inside” doesn’t resolve the debate over what scope of functions should be included on the CDP requirements list. I lean to a narrower definition, limited largely to building the unified, sharable database, based on the fact that analytics, decisions, and distribution systems can be purchased separately. To me, this indicates that each of those is a separate software category.  In concrete terms, it means that buyers face a separate “suite vs best of breed” decision for each of those components. The counter argument is that some analytics, decision, and distribution capabilities are substantially more effective when they are directly integrated with the unified customer database, so they should be considered part of the core CDP requirements. I suppose the easiest solution is to include the broader requirements on any list and let each buyer decide for herself which ones matter.

The more inclusive approach is in fact what we’re currently taking at the CDP Institute, where I’ve been distinguishing between data, analytical, and personalization CDPs. This seems to reduce confusion, so I’m reluctant to abandon it. But I've drawn the line at systems with delivery capabilities, such as sending email or managing a Web site.  My theory is that those are demanding operational features which are, or at least should be, the focus of the developers' attention.  So calling such systems a CDP is misleading, even though they might have a perfectly adequate "CDP inside".

Our current projects at the Institute include improved evaluation criteria for the data, analytical, and personalization functions.  These will make it easier for buyers to determine which CDP vendors fall into which category or, more precisely, how well each vendor meets the requirements for each category. The good news is that the same evaluation criteria can be applied to the CDP components inside of broader products. So the work will help marketers to make sound choices across all their options regardless of how we define a CDP.  And making good choices, not defining software categories, is ultimately what matters.

Saturday, February 16, 2019

Stride Offers a Next-Generation Customer Data Platform

It seems a bit soon to wax nostalgic about the good old days of the Customer Data Platform industry. But the industry has become so bewilderingly diverse in recent years that it is tempting to recall simpler times when all a new product needed to promise was making it easier to unify their customer data.  That was way back in 2016.

Stride is clearly a next-generation CDP, on the market for under two years and including the analytic and orchestration tools common to new CDP systems. But its primary focus on data aggregation and access makes it feel something like a throwback.  How did that happen?

The tale begins with RelateIQ, an AI startup purchased by Salesforce in 2014. Stride’s co-founders came along with the deal and became Salesforce employees. They soon discovered that existing Salesforce tools couldn’t collect and present all the customer data they wanted – Web site browsing history in particular.  They left Salesforce in early 2016 to build a product to fill this gap. Stride, the result of their efforts, reached the market in mid-2017.

The design goal for Stride was a system that could ingest and make available nearly any type of data with little technical effort and then activate that data by sharing customer lists with delivery systems. That's what they delivered.  

It all starts with data. Stride can ingest nearly any type of data with minimal preparation, loading batch files, SQL transactions, streaming data, Web tags feeds, and other sources into SQL tables or S3 buckets. Set-up requires little more than connections to source systems: the ingested data is flattened and then stored in pretty much its original form in what the company describes as a “semi-structured, flexible schema” that can accommodate any source and set of objects. Initial deployment can be completed in a couple of weeks after the source data is made available.

Users can enrich the inputs by adding custom objects, attributes, and events. These are built in structured interfaces designed for non-technical users. Stride doesn’t have its own identity matching processes, although users can map relationships between personal identifiers in different systems and the system will store links between identifiers when it finds them. Stride can perform simple deterministic matching, such as connecting emails to Web cookies on the devices used to open them.

Results are placed in the Snowflake relational database, an increasingly popular tool for cloud-based data warehouse applications. Users can decide which items be exposed for analysis and segmentation. Stride provides extensive tools to explore the loaded data, including a detailed view of all data for an individual customer.

External systems can query the Stride data through APIs.  In addition, Stride provides two sets of integrated capabilties: Audiences and Programs.

Audiences lets users build customer segments using the same drop-down interface that creates custom attributes and events. An audience can be defined as a single segment or as a waterfall sequence of segments, where each is customer assigned to the first segment they match. Segment membership is updated within minutes as new data enters the system.

Audience reports can show movement of customers between segments, can compare different segments against each other, and show segment member statistics such as average order value and number of messages received. Users can analyze or extract subsets within a segment, such as new entrants. Segments can be shared with external systems, including Facebook, Google, and Twitter advertising products. Shared segments are updated automatically as members are added or removed.

Programs assign actions to execute in external systems. Each program has eligibility criteria, behaviors that define entry conditions, actions to take when a behavior occurs, and behaviors that remove customers from the program.  Eligibility criteria can look across all events to do things such as limit contact frequency within a specified time period. The system can’t yet prioritize programs to ensure the most important messages are sent first, although Stride is working on it.

Each program can include a sequence of actions which occur over time.  Actions can also have their own qualification conditions and be chosen in a waterfall sequence.  Actions send instructions and data to external delivery systems. Instructions can trigger a single message or assign a customer to a campaign or journey. The system has standard connectors for major email platforms and marketing clouds as well as advertising systems. Behaviors can include events that trigger actions in near-real-time.

Program reports describe program-generated activities and dropout rates with reasons for each step in a multi-step program. Users can also create split tests within a program as well as global control groups to provide a baseline for measuring program impact.

One feature that marks Strike as a next-generation CDP is pricing.  Most early CDPs were targeted at enterprise buyers and rarely sold for less than $250,000 per year.  Stride pricing is based on number of customer profiles and can be as low as $50,000 per year for a company with under 100,000 contacts. The largest client is over 20 million. Clients are spread across multiple industries including retail, travel, media, financial services, and healthcare.

Thursday, February 07, 2019

European CDP Market Is Still Behind the U.S.

I returned earlier this week from a sequence of workshops, speeches, and meetings in Europe, all focused on Customer Data Platforms. Here are some observations:

- The European CDP market is indeed behind the U.S. My own conversations are with people who already care about CDPs, so they're a very skewed sample.  But vendors, consultants, agencies, and marketers I spoke with mostly agreed that the larger community is just beginning to hear about the concept. Many are seeking to position themselves as early adopters or experts, sensing a big business opportunity.

- Separate martech staff is rare. Nearly every large and mid-size company that I see in the U.S today has someone in charge of marketing technology, and often an entire team of marketing technologists reporting to the CMO. I was told this is much less common in Europe and personally didn’t meet anyone with a martech title. Nor did I hear about powerful IT departments taking charge. Rather, it seems that marketers still mostly act on their own, which is how it worked in the U.S. until a few years ago. I did have the impression that European marketers rely more heavily on specialist consultants to help them out, but that might be biased by the fact that many of my meetings were with consultants.

- DMP means something different in Europe. We consistently heard that marketers throughout Europe, and especially in France, were oversold several years ago on Data Management Platforms as a complete solution to handle all customer data needs. This contrasts quite sharply with the U.S., where DMPs have in most cases been understood as limited to serving up digital ad audiences. European DMPs are now recognized as having failed to deliver on the broader promise, which is beyond their technical capabilities. The resulting backlash greatly damaged the image of DMP products and has left marketers looking for a new solution that is truly capable of meeting their needs. Many recognize that CDP could be this solution and are intrigued.  But they're also skeptical and worried that they’ll be fooled again. This makes it harder for CDP vendors to sell their products. On the bright side, it also means the problem CDPs address is already well understood.

- CRM also means something different.  Back when Bill Clinton was president, CRM was described as a trinity of sales, service, and marketing systems, with marketing much weaker than the other two.  It commonly referred to B2C as well as B2B. Later, in the U.S., the term came to be more associated with B2B sales and customer service in general and the Sales cloud in particular. In Europe, CRM is used very broadly to mean any and all customer data, extending far beyond sales, service, and marketing, and including both B2B and B2C.  On reflection, I may have recently been hearing people in the U.S. apply the term more broadly as well.

- Use cases are everything. We’ve seen a huge demand to present CDP use cases in the U.S.  But it seemed even more pressing in Europe, perhaps because understanding of the CDP concept is weaker. One difference seemed to be that Europeans are willing to interact with vendors as a way of learning: while many U.S. buyers actively avoid vendors during the early stages of the purchase process, we heard quite a few requests in Europe to see detailed demonstrations of how individual vendors accomplish specific tasks. Maybe the European salespeople do a better job of being consultative, or maybe European buyers are less determined to find things out on their own. Or maybe it’s just my imagination.

- Immediate ROI is required. We also found a greater focus in Europe on use cases that tie directly to marketing programs, as opposed to the analytical use cases that are most common starting CDP applications in the U.S. The reason seems to be that European buyers are more insistent on finding a specific financial justification for their investment. Many U.S. buyers will accept a broader strategic justification and start with analytical use cases. This may be why European CDP vendors are more likely to offer a full scope of data, analytical, and campaign capabilities, since buying them in a single package makes it easier to tie new marketing programs directly to the CDP investment.

- National markets are distinct. Some of the big U.S. vendors are present throughout Europe, but many local vendors are largely limited to individual markets. We had some sense of this beforehand but the isolation was greater than expected. The French market in particular has its own ecosystem of CDPs and other types of software that have a major domestic position but little presence elsewhere. The Netherlands, German, Nordic and UK markets show more cross-over, probably because English is widely spoken in all of them. The greater interest in CDP-based marketing programs may also encourage this, since marketing programs are closely tied to specific local markets.

- GDPR hasn’t caused much change. We had some discussions about using CDP for GDPR compliance but privacy constraints in general rarely come up. The common attitude was that privacy rules were already tight in the countries we visited (Belgium, Netherlands, Germany and France), so GDPR hadn’t required significant adjustments. There was also some discussion about waiting to see how the rules are actually enforced, which might require further adjustments if the regulators are strict.


While these differences are interesting, they’re also fairly minor. Over all, the European marketers were feeling the same pressures as their U.S. counterparts to create unified data for better customer experiences. So while each market will have its own quirks and proceed at a its own pace, it looks like they’ll follow the same general path as the U.S.

Monday, January 14, 2019

Consumers Aren't As Into Personalization As You Think, and Other Survey Results

I see a lot of surveys -- easily a dozen each week.  Mostly they go into a big file which I mine occasionally for factoids to spice up a paper or presentation.  Sometimes I take a more thorough tour to look at some bigger issues.  Today is one of those days.

Specifically, I was prepping for a presentation in Amsterdam, which meant I needed to present general industry trends and then see what is different in Europe.  This turned out to be pretty interesting.  But I assembled vastly more data than I could include in any presentation where the audience was not chained to their seats (frowned upon by EU regulators).   So I'm sharing it all here with you instead.

(I've also packaged it all in a paper for the Customer Data Platform Institute, available here.  Much more convenient than copying this blog post if you want a reference copy.)

Note that there's more information on sources at the end of this post.  For now, let's just get to the good stuff.

Consumer Attitudes: Personalization

If marketers hold any truth to be self-evident, it’s that today’s consumers want and expect personalization. The reality is a bit different and depends greatly on the definition of “personalization”. The majority of consumers believe they receive personalized service, but many fewer expect personalized experiences. What they do expect is consistent service, shared information, and being identified as repeat customers. In other words, they expect you to know who they are and to use that data to serve them – for example, by being aware of past purchases and problems. But they don’t necessarily expect you to make personalized offers or otherwise personalize their experience.

We do see quite consistently that European consumers have lower expectations for all kinds of personalization.

Whether or not consumers expect personalization, it can still be a competitive advantage to provide it. The majority of consumers do say they’re more loyal to brands that understand them and provide good service, and more likely to stop doing business with brands with poor service. But, again, the focus seems to be more on service than proactive personalization: barely one quarter of consumers said that anticipating needs is the most important part of personalization. This may come as a shock to marketers who have put anticipating needs at the top of their list of reasons to do personalization.

These results shouldn’t be read as a reason to ignore customer needs. Companies get more revenue when they offer customers what they want, whether or not the customer expects it.

We again see that European consumers place slightly less weight than U.S. consumers on personalization, although the difference is less pronounced than with expectations. One interpretation would be that European consumers don’t expect personalized treatments and thus don’t factor it into their behavior.

Consumer Attitudes: Privacy

Marketers know they need to balance personalization against privacy. We’ve just seen that consumer interest in personalization may not be quite as high we thought. By contrast, consumers show great interest in privacy, both in general and specifically in relation to marketing. More than three-quarters don’t want companies to market to them based on personal data. Fewer than half would trade their data for personalized service, even though that’s the reason most companies give for collecting it. Although European consumers show slightly less concern about privacy in general, they are more opposed than U.S. consumers to letting companies use their data for marketing. This is consistent with the personalization results: if European consumers place less value on personalization, it makes sense that they’d be less willing to share their personal data to enable personalized treatments.

Looking beyond personalization to the broader question of trust, we again see that Europeans place less trust in business than U.S. consumers. An astonishing 68% believe brands sell their data. This may reflect the attention drawn to data sharing by the European Union’s General Data Protection Regulation (GDPR). Europeans' lack of trust in most business may also explain why they are more likely to support brands that do show high purpose.

Marketing Technology

Now let’s turn to marketers. Most European marketers will tell you that their region is behind the U.S. in adoption of advanced marketing technology. European consumer perceptions of less personalization support this. The data here do show that European marketers use fewer data sources and channels for most purposes, although the figure for inputs to attribution is higher. The differences are relatively small with the significant exception that Europeans report using personalization in 20% fewer channels (4.1 vs 5.1) than U.S. marketers.

The gap is larger when we focus specifically on data integration. European marketers are much more likely to cite challenges with linking multiple data sources, more likely to see linking data as the reason to deploy a Data Management Platform, and more likely to avoid a DMP because the technology is too complex. While integration is a substantial problem for many U.S. based marketers, it’s clear the pain is greater in Europe – despite having slightly fewer data sources to integrate.

The same pattern holds for marketing technology in general. European marketers spend a slightly smaller share of their marketing budget on martech and a slightly smaller share of their martech budget on data and analytics. But while those differences are fairly small, U.S. marketers expect much higher growth in their 2019 martech budgets. This is a significant indicator of attitudes regardless of what actually happens. Similarly, European marketers show consistently but slightly lower adoption of advanced marketing systems such as DMP, cross-channel engagement, and flexible attribution models.

Marketing Maturity

Looking beyond technology, we see that U.S. and European marketers share a high level of belief in personalization. But European marketers rank lower on other measures that indicate maturity. It’s particularly intriguing that European marketers are less likely than U.S. marketers to be prioritizing first party data, even though GDPR is generally assumed to make first party data more important.

In sum, the belief that European marketers are using less advanced technology than U.S. marketers appears to be correct.

Leaders vs Mainstream

What separates the most successful marketers from the rest? This data, all from the same survey, found that high performing marketing departments were twice as likely to be responsible for technical activities related to customer data: operations, governance, security, and schemas. This suggests that marketers do in fact get better results when they have more control over their customer data. By contrast, leading and mainstream departments had similar responsibility levels for traditional marketing activities such as automation rules, data acquisition, and analytics.

It’s important to qualify this message. Even among leading marketing departments, the majority do not have technical responsibilities. So clearly success is possible under other arrangements. It’s also important to recognize that marketing and IT will almost always share some responsibilities. And they should.

Other leader vs mainstream comparisons provide more insight into the challenges faced at different maturity levels. Mainstream marketers are more likely than leaders to cite disparate technology as their biggest martech challenge: this suggests that is the first hurdle to cross. Leaders, having started to knit together their systems, are likely to run into organizational barriers next. Once they resolve organizational problems, they can deliver results such as a single customer view and quantifying the benefits of personalization and real time marketing. Few mainstream marketers, still fighting technical and organizational battles, are able to accomplish these.

Some markers show much less correlation with leadership. Mainstream marketers are nearly as likely as leaders to lead customer experience initiatives and to run real time interactions in at least one channel. Note that single channel real interactions do not require unified customer data or any type of shared systems. So they are not by themselves an indication of maturity.

Customer Data Platforms

Finally, we’ll look at some information specifically related to Customer Data Platforms. The table below compares CDP selection priorities for enterprise vs mid-tier buyers. It supports the common belief that these groups have different concerns. Enterprise marketers give higher priority to data security and integrating data from many sources, including third party data. Mid-market buyers also rank security as their top concern but then look for help with internal data and for data analysis tools. These are probably problems that enterprises have already solved. One implication is that CDP vendors may find themselves specializing in one or the other type of buyer so they can optimize their systems for the different needs.

I also have several surveys that asked about CDP deployment. Answers vary greatly although the general result suggests that CDP adoption is getting close to DMP adoption. The very low figure from Heinz Marketing reflects the nature of its survey, which asked B2B marketers about tools for marketing analytics and pipeline management. The audiences for the other surveys were more representative but the figures still seem much higher than likely. The CDP Institute’s own estimate is that market penetration for CDPs at the end of 2018 was around 15%.

Note on Sources

This paper draws from surveys with different audiences, survey methods, and sample sizes. The origin of each item is indicated by a number that relates to the list of surveys below.  This list provides some information about each survey, as presented in the survey report.
Data from the original surveys has been processed in several ways:

• Questions have been paraphrased for brevity and clarity.
• European results are averages of country results, which have been weighted in different cases by national population, sample size, or not at all. Different surveys included different countries.
• Some U.S. results include data from all of North America.

Readers should be able to track down the original survey reports on the Internet. I haven't published links because links change too often to be useful.

1 Acquia, Closing the CX Gap: Customer Experience Trends Report 2019. More than 5,000 consumers and 500 marketers.
2 AdRoll, The State of Marketing Attribution, 2017. 987 respondents recruited by email and social media. Majority at director/manager level.
3 Aspect, 2017 Aspect Consumer Experience Index. Online survey with 1,000 aggregate U.S. sample and similar in Germany, Spain, United Kingdom.
4 Econsultancy, The Customer Data Imperative, 2018. 509 online survey respondents, primarily at large B2C brands. Mix of marketing, IT, and operations.
5 Edelman, 2018 Edelman Trust Barometer. 33,000_ online survey respondents across 28 countries.
6 ExchangeWire, Adoption vs Execution: How Media Agencies Across the Globe Are Making the Most of their DMP’s Capabilities, 2017. 470 agency professionals.
7 Frost & Sullivan, The Global State of Online Digital Trust, 2018. 990 survey responses.
8 Gemalto, Data Security Confidence Index, 2018. 1,050 IT decision makers from organizations with perimeter security systems.
9 GlobalWebIndex, Trends 19, 2019. 91,913 Internet users aged 16-64.
10 Harvard Business Review Analytics Services, The Age of Personalization, 2018. 625 responders from audience of Harvard Business Review readers. Primarily executive/senior management at large enterprises.
11 Heinz Marketing, State of Revenue Marketing, 2018. 241 B2B marketing executives, primarily small to mid-size companies.
12 Infosys, Endless Possibilities with Data, 2018. 1,062 senior executives from organizations with annual revenues exceeding $1 billion.
13 Ipsos+Medallia, The Customer Experience Tipping Point, 2018. 8,002 consumers in U.S., UK, France, Germany.
14 Mulesoft, Consumer Connectivity Insights 2018. 650 IT decision makers at organization with 1,000+ employees.
15 Relevancy Group, CDP Buyers Guide 2018. 406 executive marketers.
16 Salesforce Research, Fifth Edition State of Marketing 2019. 4,101 responses from full-time marketing leaders, primarily mid-size organizations. Mix of B2B and B2C.
17 Sizmek, Marketers Survey Results 2018: An Insider’s Look at Data, Walled Gardens, and Collaboration. 522 B2C brand marketers.
18 Spiceworks, 2019 State of IT, IT Marketing. 780 business technology buyers.
19 Walker Sands, State of Marketing Technology 2018. 300 marketing professionals. Primarily small to mid-size companies.
20 WE Communications, Brands in Motion 2018. Online interview of consumer survey panel totaling 11,000+ in U.S., U.K., and Germany.
21 Winterberry Group, Know Your Audience: The Evolution of Identity in a Consumer-Centric Marketplace, 2018. Online survey of more than 400 advertisers, marketers, fundraisers, publishers, technology developers and marketing service providers.