HAL Id: hal-04235032
https://inria.hal.science/hal-04235032
Submitted on 10 Oct 2023
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Distributed under a Creative Commons Attribution 4.0 International License
The Eect of Design Patterns on (Present and Future)
Cookie Consent Decisions: Supplemental Materials
Nataliia Bielova, Laura Litvine, Anysia Nguyen, Mariam Chammat, Vincent
Toubiana, Estelle Hary
To cite this version:
Nataliia Bielova, Laura Litvine, Anysia Nguyen, Mariam Chammat, Vincent Toubiana, et al.. The
Eect of Design Patterns on (Present and Future) Cookie Consent Decisions: Supplemental Materials.
USENIX 2024 - 33rd USENIX Security Symposium, Aug 2024, Philadelphia, United States. pp.1-9.
�hal-04235032�
The Effect of Design Patterns on (Present and Future) Cookie Consent Decisions:
Supplemental Materials
Nataliia Bielova
∗
Inria research centre at Université Côte d’Azur
Laura Litvine
Behavioural Insights Team (BIT)
Anysia Nguyen
Behavioural Insights Team (BIT)
Mariam Chammat
Interministerial Directorate for Public Transformation (DITP)
Vincent Toubiana
†
Commission Nationale de l’Informatique et des Libertés (CNIL)
Estelle Hary
‡
RMIT University
This document provides supplemental materials directly
cited in the proceedings of the USENIX Security Symposium
2024 entitled “The Effect of Design Patterns on (Present
and Future) Cookie Consent Decisions” by Nataliia Bielova,
Laura Litvine, Anysia Nguyen, Mariam Chammat, Vincent
Toubiana and Estelle Hary.
A Consent banners design variants
Figure 1 shows designs of all banners in our experiment.
While banners used in the study were in French language,
here we provide English translation for all banners.
B Screenshots of the task
When participants land on the fictious e-commerce website,
they first see a cookie banner they’d have to interact with
before being able to continue to the website. Figure 2 presents
screenshots of examples of what the participants would see for
one of the three fictitious websites offering to buy Bluetooth
speakers.
C Identifying behavioral levers
Table 1 shows 18 potential designs to be tested, together with
the results of the voting described in Section 3.2 of the main
paper.
D User demographics
Socio-demographic data such as age, gender and professional
status was automatically collected through the Cint panel
∗
The work was primary carried out while Nataliia Bielova was a Senior
Privacy Fellow at the LINC lab of the CNIL in 2021-2022.
†
The views and opinions expressed in this paper do not necessarily reflect
the views of the CNIL or any individual Commissioner.
‡
The work was carried out while Estelle Harry was a designer at the LINC
lab of the CNIL before May 2023.
provider before participants enter the survey itself. This al-
lowed us to better qualify the population participating in the
survey and ensure that it is as similar as possible to adult users
residing in France (see Section 3 of the main paper). Table 2
provides an overview of demographics of our participants.
E Results of all survey questions
We provide all the results of the survey questions presented
in Appendix A of the main paper. Below is the mapping of
the survey questions to the resulting tables:
• Q1: Table 3
• Q2: Table 4
• Q3: Table 5, first line
• Q4: Table 5, second line
• Q5: Table 6
• Q6: Table 7
•
Q7: because of small sample size (only 66 participants
have chosen to personalise), the collected data are not
exploitable and interpretable and we therefore omit to
present the results here.
• Q8: Table 8
• Q9: Table 9
• Q10: Table 10
• Q11: Table 11
• Q12: Table 12
• Q13: Table 13
• Q14: Table 14
1
Figure 1: Designs of all six consent banners evaluated in this study, translated into English.
Accept all
Decline all
Your cookie preferences
Our site and our partners use
cookies
that are placed on your device.
Some of these cookies are said to be
essential
and are strictly necessary for the
proper technical functioning of the site. They include tracers that
enable
audience measurement
.
If you consent,
non-essential
cookies may also be placed (you
may
withdraw your consent at any time). They are used to:
•
accurately measure your interactions with the site in order to improve our
services;
•
display multimedia content from other sites;
•
display personalised advertising
based on your profil and browsing history
;
•
share content on the social networks included on our site (e.g. Facebook).
Customise my choices
(a) Control banner.
Accept all
Your cookie preferences
Our site and our partners use
cookies
that are placed on your device.
Some of these cookies are said to be
essential
and are strictly necessary for the
proper technical functioning of the site. They include tracers that
enable
audience measurement
.
If you consent,
non-essential
cookies may also be placed (you
may withdraw
your consent at any time). They allow:
•
to accurately measure your interactions with the site in order to improve our
services;
•
display multimedia content from other sites;
•
display personalised advertising
based on your profil and browsing history
;
•
share content on the social networks included on our site (e.g. Facebook).
Customise my choices
(b) No decline banner.
Accept all
Decline all
Your cookie preferences
Our site and our partners use
cookies
that are placed on your device.
Some of these cookies are said to be
essential
and are strictly necessary for the
proper technical functioning of the site. They include tracers that
enable
audience measurement
.
If you consent,
non-essential
cookies may also be placed (you
may withdraw
your consent at any time). They are used to:
•
accurately measure your interactions with the site in order to improve our
services;
•
display multimedia content from other sites;
•
display personalised advertising
based on your profil and browsing history
;
•
share content on the social networks included on our site (e.g. Facebook).
Customise my choices
(c) Highlighted accept banner.
Decline all
Your cookie preferences
Our site and our partners use
cookies
that are placed on your device.
Some of these cookies are said to be
essential
and are strictly necessary for the
proper technical functioning of the site. They include tracers that
enable
audience measurement
.
If you consent,
non-essential
cookies may also be placed (you
may withdraw
your consent at any time). They are used to:
•
accurately measure your interactions with the site in order to improve our
services;
•
display multimedia content from other sites;
•
display personalised advertising
based on your profil and browsing history
;
•
share content on the social networks included on our site (e.g. Facebook).
Accept all
Customise my choices
(d) Highlighted decline banner.
Accept to be traced
Your cookie preferences
Our site and our partners use
cookies
that are placed on your device.
Some of these cookies are said to be
essential
and are strictly necessary for the
proper technical functioning of the site. They include tracers that
enable
audience measurement
.
If you consent,
non-essential
cookies may also be placed (you
may withdraw
your consent at any time). They are used to:
•
accurately measure your interactions with the site in order to improve our
services;
•
display multimedia content from other sites;
•
display personalised advertising
based on your profil and browsing history
;
•
share content on the social networks included on our site (e.g. Facebook).
Continue without being traced
Customise my choices
(e) Consequences banner.
Decline all
Customise
my
choices
Accept all
Your cookie preferences
Our site and our partners use
cookies
that are placed on your device.
Some of these cookies are said to be
essential
and are strictly necessary for the
proper technical functioning of the site. They include tracers that
enable
audience measurement
.
If you consent,
non-essential
cookies may also be placed (you
may withdraw
your consent at any time). They are used to:
•
accurately measure your interactions with the site in order to improve our
services;
•
display multimedia content from other sites;
•
display personalised advertising
based on your profil and browsing history
;
•
share content on the social networks included on our site (e.g. Facebook).
(f) Tricolor banner.
2
Figure 2: Three steps of interaction on a fictious e-commerce website.
(a) Before interacting with the cookies banner.
(b) After interacting with the cookie banner.
(c) Example of information participants can find on the website.
3
Objective Intervention
Ranking
follow-
ing the
vote
Improve the knowledge of Web users
Simplify
Shorten, summarise, clarify text / include an FAQ 3
Use rule of thumbs to simplify decision 13
Use images, emojis, infographics 1
Use visual aids that can be used as heuristics (e.g. traffic lights) 7
Make refusing /
personalising
attractive
Renew banners regularly 17
Increase feeling of importance and urgency to interact with the banner 14
Use personalised feedback to explain to users the implications of their choice 12
Highlight consequences of their choice 2
Encourage web designers to display disadvantages and not only advantages of accepting
11
Make it social Use social norms 18
Make it timely
Prevent all interactions with the website until users make their choice 16
Introduce a pop-up that checks whether users really want to accept cookies 8
Improve access to refusal option
Simplify refusal
Display both refuse and accept option similarly 9
Make refusing more visible and attractive to choose 5
Recommend that consent be asked every 6 months, independently of their past choices
10
Promote plug-ins that automatically deal with consent banners 15
Make decline an
option by default
Pre-select "Refuse all" 4
Put "Refuse all" on the first layer, and "Accept all" on the second 6
Table 1: 18 potential designs to be tested, together with the results of the voting.
F Regression tables
We provide results for all regression variables in Tables 15 for
consent decisions on the first website, Table 16 for average
decision time on the first website, and Table 17 for consent
decisions on the third website.
G Consent decisions for desktop vs. mobile
Table 18 presents the decline/personalise rate for desktop and
mobile participants.
Table 2: Summary of participant demographics.
Characteristics Percent (%)
Gender
Women 52%
Age
18-30 24%
31-49 34%
50+ 42%
Education
>Bac + 3 33%
Annual household revenue
>40 000C 65%
Employment status
Employed 71%
Retired 15%
Student 6%
Unemployed 5%
Other 2%
Urban-Rural
Rural 35%
Sub-urban 22%
Urban 43%
4
Table 3: Participants’ level of comfort with sharing data (
N =
4, 026).
Level of comfort
Percentage
Comfortable even without knowing how it
is used
52%
Not comfortable in any given case 31%
Comfortable only if knowing how it is used
17%
Table 4: Participants’ high level of comfort (comfortable/very
comfortable) regarding sharing various types of data (
N =
4, 026).
Type of data
Cumulative percentage for "comfort-
able" and "very comfortable" level
Past product
purchases
74%
Email address 47%
Browsing his-
tory
46%
Postal address
45%
Social media 43%
Geolocation 42%
Phone number
36%
Contacts 36%
Bank data 14%
5
Table 5: Participants’ recall of choice and banners. One star indicates that this value is significantly different from the highest
value at least to the 5% level.
Overall Control Dark Pattern Behavioral Lever
No reject
Highlighted
accept
Consequences
Tricolor
Highlighted
decline
Remembers their choice 85% 86%* 87% 87%* 84%* 90% 74%*
Remembers seeing different/
similar banners
36% 47%* 27%* 20%* 41%* 59% 24%*
N 4,026 690 693 672 660 641 670
Table 6: Participants’ reasons for refusing consent on the first website. One star indicates that this value is significantly different
from the highest value at least to the 5% level.
Overall Control Dark Pattern Behavioral Lever
No reject
Highlighted
accept
Consequences
Tricolor
Highlighted
decline
Avoid being tracked 57% 62% N/A 57%* 71% 48%* 43%*
Avoid targeted advertising 46% 57% N/A 52% 44%* 41%* 44%*
Out of habit 37% 43% N/A 51% 27%* 38%* 38%*
The fastest option 7% 7%* N/A 4%* 4%* 8%* 13%
No accept option available 3% 1% N/A 0%* 3% 4% 3%
Clicked randomly 4% 1%* N/A 0%* 3%* 3%* 8%
N 978 110 N/A 114 306 230 218
Table 7: Participants’ reasons for accepting consent on the first website. One star indicates that this value is significantly different
from the highest value at least to the 5% level.
Overall Control Dark Pattern Behavioral Lever
No reject
Highlighted
accept
Consequences
Tricolor
Highlighted
decline
Out of habit 50% 50% 47% 51% 50% 53% 50%
Fear for not being able to ac-
cess the website
35% 35% 32%* 35% 40% 33%* 39%
The fastest option 33% 34% 32%* 37% 30%* 31%* 33%
Option to refuse not available
9% 5%* 20% 7%* 6%* 3%* 5%*
To receive personalised ad-
vertisement
10% 11% 9% 10% 12% 9% 11%
Clicked randomly 5% 5% 5% 5% 5% 3%* 7%
N 2,982 574 666 551 351 395 445
Table 8: Participants’ reported satisfaction with the choice they made. One star indicates that this value is significantly different
from the highest value at least to the 5% level.
Overall Control Dark Pattern Behavioral Lever
No reject
Highlighted
accept
Consequences
Tricolor
Highlighted
decline
Satisfied by 1st banner choice 56% 54%* 44%* 55%* 64% 64% 58%*
Dissatisfied 44% 46%* 56% 45%* 36%* 36%* 42%*
Equally satisfied 16% 18% 20% 17% 12%* 13%* 14%
N 4,026 690 693 672 660 641 670
6
Table 9: Reported opinion on the banner seen on the first and second website. One star indicates that this value is significantly
different from the highest value at least to the 5% level.
Overall Control Dark Pattern Behavioral Lever
No reject
Highlighted
accept
Consequences
Tricolor
Highlighted
decline
Navigating the banner is sim-
ple
85% 84%* 81%* 86% 87% 88% 87%
Information easy to under-
stand
82% 79%* 77%* 82% 84% 83% 85%
Choices presented with my
best interest in mind
68% 68%* 64%* 70% 67%* 73% 69%*
Easy control the level of data
protection
72% 71%* 64%* 74% 76% 77% 73%*
More websites should use
this banner format
80% 78%* 72%* 82% 82% 84% 81%
N 4,026 690 693 672 660 641 670
Table 10: Participants’ responses in percentage regarding data
collected by cookies (N = 4, 026).
Type of data collected Percentage
Browsing history 62%
Past product purchases 58%
Geolocation 57%
Information shared on my social network 36%
Postal address 30%
Phone number 29%
Email sent and received 28%
Contacts 23%
Bank data 13%
None 6%
All 4%
Table 11: Participants’ responses in percentage regarding
reasons for which data is collected via cookies (N = 4, 026).
Reasons
Percentage
Decide which ads to show me 61%
Resell my data to third parties 48%
Subscribe to newsletter and send me mar-
keting emails
42%
Improve my user experience on the website
36%
Adjust prices or sales to researched prod-
ucts
29%
Adapt language and display to my geoloca-
tion
24%
Allow to view media content (e.g. YouTube
video) on the website
22%
Ensure security on the website 17%
Report user data to the government 12%
Table 12: Participants’ responses in percentage on the means
with which cookies collect data (N = 4, 026).
Cookies collect data by ...
Percentage
Tracking what I do online (browsing history,
services and products purchased)
74%
Tracking information I actively share (e.g.
on social media)
44%
Tracking information that others have
shared about me (e.g. on social media)
35%
Listening to what I say through the micro-
phones on my computer and phone
15%
Table 13: Participants’ willingness to use browser extension
to aromatically accept or refuse cookies (N = 4, 026).
Willingness
Percentage
Yes, definitely 24%
Yes, probably 38%
No, probably not 6%
No, definitely not 3%
Not sure 29%
Banking 33%
7
Table 14: Types of websites on which participants are some-
what comfortable/very comfortable to accept cookies (
N =
4, 026).
Type of web-
site
Cumulative percentage for "comfort-
able" and "very comfortable" level to
accept cookies
E-commerce 55%
Information 51%
Government 51%
Health 51%
Social media 41%
Banking 33%
8
Table 15: Participants’ reported consent decisions on the consent banners of the first website (
N = 3, 947
). Three stars (***)
indicate a significant difference with the decision made when seeing the control (
p < 0.001
). Estimates are calculated using odds
ration from a logistic regression controlling for age, gender, education, and employment status of participants.
Treatment % non-acceptance (personalize or reject) 95% CI
Dark pattern
No-decline 3.90*** 2.54; 5.87
Highlighted-accept 18.00 14.19; 22.57
Behavioral Lever
Highlighted-decline 33.58*** 27.71; 39.18
Consequences 46.73*** 40.20; 52.85
Tricolor 38.38*** 32.08; 44.23
Control Control 16.81
Table 16: Participants’ average decision time (
N = 3, 947
) in their interactions with the consent banners on the first website. Three
stars (***) indicate a significant difference with the time spent when seeing the control (
p < 0.001
). Estimates are calculated
using coefficients from a linear regression controlling for age, gender, education, and employment status of participants. The
variable for "time spent" was winsorized at the top (1%) to deal with outliers, which most likely corresponds to participants
leaving their computer and coming back to the experiment later.
Treatment Average decision time (seconds) 95% CI
Dark pattern
No-decline 4.03 3.33; 4.52
Highlighted-accept 3.78 2.94; 4.13
Behavioral Lever
Highlighted-decline 4.24 3.80; 4.99
Consequences 5.70*** 4.72; 6.68
Tricolor 5.72*** 4.74; 6.70
Control Control 4.10
Table 17: Participants’ reported consent decisions (
N = 3, 947
) when facing the Control banner upon visiting the third website.
Three stars (***) indicate a significant difference (
p < 0.001
) with participants having seen the Control banner on the first and
second website (control group). Estimates are calculated using odd ratio from a logistic regression controlling for age, gender,
education, and employment status of participants.
Treatment % who personalise or reject 95% CI personalise or reject
Dark pattern
No-decline 9.66*** 7.17; 12.79
Highlighted-accept 20.09 16.31; 25.15
Behavioral Lever
Highlighted-decline 29.7*** 24.33; 34.99
Consequences 35.81*** 30.13; 41.75
Tricolor 27.31*** 22.49; 32.91
Control Control 18.84
Table 18: Non-acceptance (decline/personalise) rate depending on device used. One star indicates that this value is significantly
different from the highest value at least to the 5% level.
Control Dark Pattern Behavioral Lever
No reject
Highlighted
accept
Consequences
Tricolor
Highlighted
decline
Desktop 27% 4% 29% 54% 36% 35%
Mobile 14% 4% 15% 45% 39% 33%
N 690 691 672 658 640 669
9
