Spoiler alert: forget it.
There is no compliant A/B testing tool.
Meaning, there’s no “certificate” from an A/B testing tool guaranteeing you’ll avoid that 20 million euro lawsuit from a European privacy authority. But what is possible is that you choose a testing tool designed with privacy in mind. One that won’t get you into trouble when using its default configuration.
This article helps you ask right questions about your A/B testing tool vendor—before the GDPR enforcement date (May 25th, 2018). I’ll deconstruct each element important in GDPR and the upcoming ePrivacy Regulation (draft 15333), and give you a checklist at the end.
In this article…
- Data Minimalization
- Personal Data includes IP-address
- Cookies & Personal Data
- User ID’s & Personal Data
- Still Planning on Storing Personal Data?
- GDPR & End-User Data Accuracy
- I’m scared to ask, but what about sub-segmentations for variations?
- And connecting third-party data? Like BlueKai?
- Do Not Track for Europe Only? Are you Sure?
- Revenue Tracking? Order Numbers as Personal Data?
- Damn, any good news?
The principle of data minimization is essentially the idea that, subject to limited exceptions, an organization should only process the (personal) data it actually needs to achieve its processing purposes.
So what’s the minimum data needed to run experiments? I mean, do you need to collect every detail of the IP, browser, and device ID, to get results for your A/B test? No. You are collecting them because the tool can collect them. With privacy in the back of your mind, focus on the elements you need to collect and do not store the rest. What is essential to run an experiment is:
- Variation seen
- Goal reached: yes/no (on revenue goals, product count, and revenue)
When you’d like to dive deeper on the different outcomes per subsegment you need, for example:
- Subsegment: yes/no
The A/B testing tool doesn’t actually need to store one row of data for each end-user. What you need to store to get to insights is:
- Total end-users that saw specific variation (total and per subsegment)
- Total end-users of a variation that triggered a goal
And, in an e-commerce setting:
- The average products per variation and
- The average order value per variation…
…that can be recalculated on each new order (without storing the individual orders).
The rest of the statistical data can be recalculated on the fly without storing individual rows of end-user data. This allows your A/B testing tool to comply with the data minimization guidelines and lowers the risk of a privacy & security breach significantly.
Think about it. What would someone do with just the totals of an experiment? If there is no end-user data stored on your A/B testing tool, not even pseudonymous/hashed data, there is no way that end-users can be re-identified.
Personal data, as defined by GDPR, if different than previous definitions of Personal Identifiable Information (PII). Now, IP addresses, cookies, and UserID are all personal data.
Regardless of how you may feel about that definition—it’s law. On October 19, 2016, the Court of Justice of the European Union (the “CJEU”) published its judgment in Case 582/14 – Patrick Breyer v Germany. It held that IP addresses are personal data in certain circumstances. The judgment is broadly in line with the Advocate General’s Opinion in this case, from May 2016.
Personal data are defined in Article 2(a) of the Directive as “any information relating to an identified or identifiable natural person (‘data subject’). An identifiable person is one “who can be identified, directly or indirectly […]” (emphasis added). Further analysis of the issue of identifiability is provided by the EU’s Article 29 Working Party, in its Opinion 4/2007.
So IP addresses are personal data. Period. You can choose to anonymize them unless you receive explicit consent from the visitor (which you may want to avoid). But even if you do, this will always be on the border of consent—and it’ll run the risk of re-identification.
We suggest you do not store IP’s in your A/B testing tool, to stay 100% on the safe side of this new law.
Now you might ask…if we can’t store IP addresses, how do you run country targeting?
The easiest solution is to use the call from the end-user to call the tracking script and identify the country. Store the combination of variation/country in one sub-segment but not connected to a user. Just add an extra view to that subsegment, so it’s mixed with the hundreds of others in that segment and basically just +1. So again, no way to deconstruct the individual end-user from the database.
An additional note on geolocation is that country data itself is too broad, but your subsegments of region, city or zip code will require consent from the end user. Combined with other elements, it may be subsegments that are really small (even just 1 person!). So again, you could store personal data, and this would require the consent of the user.
According to GDPR, cookies are personal data too. But all cookies are not created equal. It may be possible that some require user consent, whereas others not. The particulars will be determined by the new ePrivacy Regulations—once they’re approved. For now, we have draft 15333, which has an exception for web analytics software, like A/B testing tools.
(21) Exceptions to the obligation to obtain consent to make use of the processing and storage capabilities of terminal equipment or to access information stored in terminal equipment should be limited to situations that involve no, or only very limited, intrusion of privacy. For instance, consent should not be requested for authorizing the technical storage or access which is necessary and proportionate for the legitimate purpose of enabling the use of a specific service requested by the end-user. This may include the storing of cookies for the duration of a single established session on a website to keep track of the end-user’s input when filling in online forms over several pages, authentication session cookies used to verify the identity of end-users engaged in online transactions or cookies used to remember items selected by the end-user and placed in shopping basket. Cookies can also be a legitimate and useful tool, for example, in assessing the effectiveness of a delivered information society service, for example by helping to measure to the numbers of end-users visiting a website, certain pages of a website or the number of end-users of an application. This is not the case, however, regarding cookies and similar identifiers used to determine the nature of who is using the site.
A/B testing tools are free of consent requirements, as long as they are “assessing the effectiveness of a delivered information society service.” But as soon as you go into very specific targeting of end-users, or trying to identify the nature of who is using the website—then you’re moving into an area that requires consent.
You should be double-checking that the cookies used by your A/B testing tool don’t have a unique identifier. The unique identifier can make a cookie qualify as personal data.
While ePrivacy Regulation (draft 15333) is not yet a law, and it’s intention it to compliment GDPR—GDPR itself gives us some guidance as to which types of cookies will, by their nature, involve processing of personal data. There is 1 recital that is key to this:
GDPR Recital 30
Natural persons may be associated with online identifiers…such as internet protocol addresses, cookie identifiers or other identifiers….This may leave traces which, in particular when combined with unique identifiers and other information received by the servers, may be used to create profiles of the natural persons and identify them.
This tells us that cookies which contain a unique identifier for the device and/or the individual associated with using the device, should be treated as personal data. This idea is also reinforced by Recital 26, which states that personal data is also defined by data that can reasonably be used, either alone or in conjunction with other data, to identify an individual.
So your A/B testing tool should remove any unique identifiers to comply with GDPR. In addition, never store the individual record connected to one cookie on the server side. That would make it fall outside the exception in ePrivacy Regulations.
Pseudonymous identifiers (e.g. strings of numbers or letters), which is what cookies often contain, also qualify as personal data. So under the GDPR, any cookie or other identifier that is uniquely attributed to a user and therefore capable of identifying an individual, counts as processing of personal data once this identifier is stored on the server side.
Storing a User ID specific to an individual, or any other identifier that you can cross-link to individual personal data… is …you guessed it… personal data. So, these require consent and should be avoided (Recital 30 applies also here, as does Recital 50 below).
Rec.50; Art.5(1)(b) Personal data may only be collected for specified, explicit and legitimate purposes and must not be further processed in a manner that is incompatible with those purposes. (Further processing of personal data for archiving purposes in the public interest, or scientific and historical research purposes or statistical purposes, in accordance with Art.89(1), is permitted—see Chapter 17).
Convert has a feature called Universal User ID. If this, for example, is used to reconnect sessions from an e-commerce system after users buy, the consent may be incorporated into the terms agreed on with the customer relationship. But we still suggest a separate consent for a unification of devices, sessions, and browsers—since it’s best not to assume with GDPR, but to offer absolute transparency.
So you decided to store everything in your A/B testing tool? Well storing a session-ID, User ID, or any other identifier means you have to make sure it’s
- Updated on demand
- Erased when required
Let me throw you some items of the GDPR law in context of the storage of personal data and all the fun that comes with it.
GDPR Recital 39; Art.5(1)(d)
Personal data must be accurate and, where necessary, kept up to date. Every reasonable step must be taken to ensure that personal data that are inaccurate are either erased or rectified without delay.
So when you have an A/B testing tool that stores personal data, and your tests reach significance, you need a tool that freezes those results once end-users fill out the forms you need to have on your website. If the previous paragraphs didn’t already convince you to have personal data stored in your analytics tool, this part of the GDPR law will. I’ll paste the entire article that spells it out here.
GDPR Article 15, Right of access by the data subject
- The data subject shall have the right to obtain from the controller confirmation as to whether or not personal data concerning him or her are being processed, and, where that is the case, access to the personal data and the following information:
- the purposes of the processing;
- the categories of personal data concerned;
- the recipients or categories of recipient to whom the personal data have been or will be disclosed, in particular recipients in third countries or international organizations;
- where possible, the envisaged period for which the personal data will be stored, or, if not possible, the criteria used to determine that period;
- the existence of the right to request from the controller rectification or erasure of personal data or restriction of processing of personal data concerning the data subject or to object to such processing;
- the right to lodge a complaint with a supervisory authority;
- where the personal data are not collected from the data subject, any available information as to their source;
- the existence of automated decision-making, including profiling, referred to in Article 22(1) and (4) and, at least in those cases, meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing for the data subject.
- Where personal data are transferred to a third country or to an international organization, the data subject shall have the right to be informed of the appropriate safeguards pursuant to Article 46 relating to the transfer.
- The controller shall provide a copy of the personal data undergoing processing. For any further copies requested by the data subject, the controller may charge a reasonable fee based on administrative costs. Where the data subject makes the request by electronic means, and unless otherwise requested by the data subject, the information shall be provided in a commonly used electronic form.
- The right to obtain a copy referred to in paragraph 3 shall not adversely affect the rights and freedoms of others.
So… I’ll let you think about that for a minute…
Yeah I thought so. Now all together with me… say this out loud.
- I promise to not store personal data in my A/B testing tool
- I promise to not store personal data in my A/B testing tool
- I promise to not store personal data in my A/B testing tool
So double check your A/B testing provider and make sure that there is no way that they story individual records of unique visitors.
I mean, we all want to bring better insights and good news to clients and bosses. And it’s awesome to drill down into lost A/B tests and see if some subsegment won.
But really…do you want to be one of those people that hides the total visitors on your presentation screenshots and amazes people with 1460% wins on one variation on Chrome in Sweden?
Just accept the following: you setup a test with a clear hypothesis, run the experiment, and stop it when enough traffic has arrived (no peeking). Looking for winners in tiny subsegments, doesn’t make you a better marketer.
With GDPR, the more you store, the more you will know—and the higher the chance segments of traffic become real end-users. Along with that, comes a higher demand that you protect the data, request the data (using Subject Access Requests), erase the data and update the data.
Just don’t store it, make sure your A/B testing tool does not store it, and you’ve given yourself one less thing to worry about.
IF still you decide you don’t mind asking consent, and storing all that personal data in your A/B testing tool—then make sure your A/B testing tool allows an easy way for you to export raw data based on unique visitors. Make sure it’s locked up. That it can modify interest, categories and other segmentation you’re stored about users, and it can erase the data without modifying historical results of A/B tests.
By collecting personal data, you will have to respect the rights of EU citizens and people in Europe at the time of their visit to your website. This includes:
- The possibility for them to view the data you collected on them inside experiments, audiences and segments (and goals).
- The possibility to update/rectify some or all data concerning them.
- The possibility to delete their data upon their request.
- The possibility to export all the data. So if a user is requesting it, you will have to export the data linked to his IP address(es) or unique identifier. It can be easily exported as a .csv file for example.
So when you decide you will store personal data then you should offer a Subject Access Request concerning data access, rectification, and erasure.
- the right to access
- the right to restrict processing
- the right to rectification
- the right to be forgotten
- the right to data portability
Every Subject Access Request will need be processed and resolved within the timeframe of at the latest one month of receipt, as described in the text of GDPR.
You are able to collect such data using custom dimensions, custom variables… and tons more.
Now let’s not get started again. You just promised me that you would not store anything based on the end-users rights to that data in article 15 of GDPR. But even when you don’t have any identifiers to connect that information, you may still be storing personal data. This example explains why.
From the Financial Times (UK) November 30, 2017.
Richard Lloyd, a veteran consumer rights campaigner, alleges the technology company bypassed the default privacy settings on Apple phones and succeeded in tracking the online behavior of people using the Safari browser. Google then allegedly used the data in its DoubleClick advertising business, which enables advertisers to target content according to a user’s browsing habits.The lawsuit, filed in London’s High Court, alleges Google’s tactic, known as “the Safari Workaround”, breached the UK Data Protection Act by taking personal information without permission.
Technologies which enable mobiles, tables or desktop devices can be connected to their current location. Other organizations may also want to use this type of information to provide location-specific content or advertising, however, location data, especially data about the precise pattern of an individual’s movements over time, can reveal very intimate details about that person’s personal life. This is personal data.
Data Protection Commission says: “You should treat information about the location of a device which can be tracked or located electronically as “personal data”, and comply with the Data Protection Acts in relation to it, if:
- The data relates to a living person (a “data subject”); and
- It is possible to identify the person to whom it relates from the location data itself, or from the location data together with other information which you have or are likely to acquire.
Section 1 of the Data Protection Acts defines “personal data” as:
“data relating to a living individual who is or can be identified either from the data or from the data in conjunction with other information that is in, or is likely to come into, the possession of the data controller”
So for your A/B testing tool this means that you could store an individual user in multiple segments and together this individual can be identified as a person. So the advice is not to store individual records ever—just groups of segments in the database. And ask for consent when you start collecting multiple subsegments that involve location data on locations more specific than country. Your A/B testing tool can give you an advice when you are collecting too many segments but, in my opinion, storing individual records is just asking for trouble.
You need end-user consent. Or, BlueKai and you can be joint-controllers; meaning, on the moment of end-user data collection BlueKai, has asked consent in your name. That being unlikely, at the moment of the writing of this piece, you need consent to enrich you audiences in A/B testing tools with a form of consent on the data. Do ask your data provider, for example, BlueKai.
Do Not Track (or just DNT) is a privacy setting end-users can set in most browsers. You should honor DNT as a signal for how you and your end-users want us to use data. In the last years many end-users signaled that they want a new way of engaging with brands. For example, in Canada—where Canadian federal legislation brings us changes in the Privacy Act. Or in the US Congress with a new stage in the Privacy Shield. Or in the European Union where member states voice the support for a more respectful interaction with end-users. You should hear them and respect them.
DNT has three possible values:
- Do Not Track (Opt out of tracking)
- Track (Opt into tracking)
- Null – No preference
By default, web browsers use the null value (no preference), indicating that the end-user hasn’t expressed a desire of whether they want to be tracked or not. Your A/B testing tool should have a setting to not not load ANY scripts when option one, the Do Not Track (Opt out of tracking) in the browser is set.
Jonathan Mayer and Arvind Narayanan are the two principal researchers at Stanford University who have been working on the Do Not Track technology. Their model uses information in the HTTP header to universally opt out of all online tracking.
“As a technology model, Do Not Track is clearly superior to an opt-out mechanism,” said Mayer, referring to commitments by Google other browser to support the technology on their websites.
“So the technology question is now settled: it’s Do Not Track.”
Your A/B testing tool can have the following options:
- Do Not Track is important enough and is respected by the tool without the need for customers to turn anything on or off. It just works (full disclosure: at Convert, we picked this one…).
- Do Not Track can be turned on or off by users for all the platform (check it’s default for all your projects as admin).
- Do Not Track can be turned on by users of the platform per project, and region (Europe) and should be checked by admin on each project.
Then what actually happens when it’s turned on is important.
Will you place a cookie (not suggested) or does the tool just not load at all (we selected that). The downside of the first (placing the cookie) is that cookies might trigger a warning to end-users, and the website owner might get more requests for data from users. The downside of not loading the script at all is that browser extension and users really don’t get tracked in any way (and so visitors tracked for a variation or experience might start differing from the analytics tool). In my opinion, the second issue makes more sense. The GDPR will make a mess of the numbers anyway, since different tools will get or not get consent. So matching them all up to ascertain one “true” result will become nearly impossible, anyway.
GDPR Recital 85; Article 5(2)
The controller is responsible for, and must be able to demonstrate, compliance with the Data Protection Principles.
Just a refresh of processor and controller (confusing, right?).
Article 4 defines data controllers and data processors as below:
(7) ‘controller’ means the natural or legal person, public authority, agency or other body which, alone or jointly with others, determines the purposes and means of the processing of personal data; where the purposes and means of such processing are determined by Union or Member State law, the controller or the specific criteria for its nomination may be provided for by Union or Member State law;
(8) ‘processor’ means a natural or legal person, public authority, agency or other body which processes personal data on behalf of the controller;
You are the controller in the relationship between your website visitor (end-user). Your A/B testing tool is the processor.
This means you are responsible for compliance to GDPR, and you cannot point fingers to your A/B testing tool vendor. The smartest way to do that, is to make sure your A/B testing tool vendor is doing everything to comply as described in this article.
Here I have my doubts. Will Order ID’s (order numbers or transaction ID’s) be personal data?
I don’t think so, but the separation of personal data from non-identifying information (e.g. an order number from the customer’s name and address) is just good practice.
Since we don’t want customers to worry about any possible modification in ePrivacy Regulations, we, as company decided to not store this information in our revenue tracking section of the tool.
The combination of average order value, products bought, revenue and order ID is a little too much of non-identifying information to sit in our databases. But this is up to you and your A/B testing vendor to discuss on how you’d like to deal with this. Data minimization basically requests us to store as little as we can. Since we don’t need this for revenue tracking, we are dropping it from the dataset, together with individual revenue, product count and any other order data on individual (end-user) level.
If you’re sending information on what variation and experiences the end-user participated in, we suggest consent for this.
Remember those cookie walls where you had to accept the cookies to be able to read the content? Well, those are gone. You cannot put those up anymore and so it’s either showing content to all visitors, asking consent for specific trackers or moving it behind the paywall.
The GDPR states that cookies improve a user’s internet experience—for example, remembering one’s shopping cart history. Or when completing a form over several pages.
Analytics in GDPR is possible as long as any personal data collected is only processed by the first party. ePrivacy is specific regarding cookies, where GDPR is not. Read ePrivacy Regulation draft 15333, article 21;
This may include the storing of cookies for the duration of a single established session on a website to keep track of the end-user’s input when filling in online forms over several pages, authentication session cookies used to verify the identity of end-users engaged in online transactions or cookies used to remember items selected by the end-user and placed in shopping basket. Cookies can also be a legitimate and useful tool, for example, in assessing the effectiveness of a delivered information society service, for example by helping to measuring web traffic to the numbers of end-users visiting a website, certain pages of a website or the number of end-users of an application.
Ah one last thing… US vs EU based servers
Does it matter where you host your website? Or where the A/B testing tracker script is loaded from? Or where you actual dataset is located? Yes, it would not be in this article otherwise 🙂
Servers locations are mainly about data transfers – which alone would require consent if involving personal data. We have to look at Chapter 5 of the GDPR on Transfers of personal data to third countries or international organizations. This will give us insight on what to do if the data is stored outside of the European Union.
Third countries’ level of personal data protection is assessed by the European Commission through ‘adequacy findings’, which are binding in their entirety to all Member States. Once the “adequacy” of a third country has been recognized, personal data can be transferred to this country without having to take further protective measures.
A general principle for transfers (Article 44) and transfers on the basis of an adequacy decision (Article 45) explains that the discussion often revolves mainly around transferring data to countries outside the EU (and Norway, Liechtenstein, and Iceland), where you now have only options:
- Privacy Shield (for US) and adequate countries (The European Commission has so far recognised Andorra, Argentina, Canada (commercial organisations), Faroe Islands, Guernsey, Israel, Isle of Man, Jersey, New Zealand, Switzerland, Uruguay and the US (limited to the Privacy Shield framework) as providing adequate protection.
- Standard contractual clauses, also called model contractual clauses
- Binding Corporate Rules (covered in Article 47 of the GDPR).
Most likely your A/B testing vendor is a US company—and then the easiest choice seems to be the Privacy Shield, as it’s only a one-time effort. For our company this means even though all our customer data (of their website end-users) is stored in carbon neutral servers in Frankfurt–our legal base for the US company would be Privacy Shield (we just like to be extra safe and have all servers with customer datasets in Germany).
GDPR is massive, and not just restricted to companies in the European Union. Looking now, at ePrivacy Regulations draft 1533 and GDPR, it’s our opinion that you can A/B test using Convert Experiences on all visitors without consent.
We have an entire page setup on how Convert as a company will become compliant, and an entire application roadmap on the details we are working on to make it easier to comply to GDPR.
Ask your vendor how they see their tool in terms of compliance, and when they are working on massively simplifying their database structure to A/B test without requiring consent.
Because if your tool DOES require consent—it will have a massive impact on your marketing stack. The big fear is that with a consent-based A/B testing tool, you will lose more than 80% of your traffic to work with.
What fields do you store by unique visitor in your database (after roadmap completion)?
Do you store visitor IP?
Do you store Country/Region/City or other location data by visitor?
Can you describe the cookie content or link to a support article?
How long are cookies stored?
Do you store unique user ID’s (session-ID, device-ID, User ID or email), and/or allow us to store emails or other personal data? If so where is it stored?
How does your tool assist us with end-user Subject Access Requests?
Supporting by not having any personal data stored
How does your tool support Do Not Track?
Yes, by default script not loaded (see also our GDPR A/B testing app roadmap)
What fields do you store by the unique visitor in your database (after roadmap completion)?
What server location (country) is our end-user data stored?
If end-user or company information is stored outside the EEU what legal base you are using to transfer the information.
End-user information is not stored and not transferred outside the EEU, unless customer requests debugging logging (stored max. 6 days). Customer information and debugging logs could be transferred on bases of Privacy Shield to US servers.
What security systems you have in place on databases and system where end-user information is stored?
C5, DoD SRG, FedRAMP, FIPS, ISO 9001, ISO 27001, ISO 27017, ISO 27018, PCI, DSS Level 1, SEC Rule 17-a-4(f), SOC 1, SOC 2, SOC 3
How does your company prepare for GDPR compliance?
Find all about out company GDPR compliance program, here.
How does your company assist us as controller with GDPR compliance of end-user data?
The application roadmap is now public.
Disclaimer: Note that this article represents the views of the author solely, and are not intended to constitute legal advice.