REDACTED FOR PUBLIC FILING
EXHIBIT
PSX00204
GOOG-DOJ-04372072
Message
From:
B
ene
d
ict
Gomes
Redacted@googl
e.com]
Se
nt
: 3/23/2019 3:35:16
PM
To:
Nick
Fox [email protected]]; S
ha
shidhar Thakur [email protected])
CC
:
Abhinav
Taneja
Redac
ted@google
.com]
Subject
: Ads cy
Hey - here's what I want to send to Pragh prior to the meeting. W dyt?
Not helpful? I have to admit that
I'm
feeli ng annoyed both personally and on behalf
of
the team. This may be a
useful framing too. Maybe worth turning into a doc?
ben
Definitely the main topic. I think there are many things being done, and there is misunderstanding between the
teams that we need to sort out.
- I think your team somehow feels that we don't care about usage. This is
jus
t not true. Not true. And this is
causing a lot of angst on both sides. Almost
all
the team now does (a pivot I got Amit to start a few years ago)
is growth motivated. We do care very deepl
y.
Saying that we don't feels very strange to my team.
- most headcount (non assistant) for search has gone into projects
tha
t are growth oriented.
To
the point I worry
that we are really not investing in research or speculation adequatel
y.
However, our ability to predict growth
associated with work we do
in
the short term (months) is still limited. The longer term (years) seems like there
are very clear correlations between growth and investment and vice versa (no .investment leading to declines).
- we have spun up efforts with your team to look at
- desktop (by porting over mobile UI)
- UI tweaks to improve access points (0 query suggest, query box size etc). suggest ranking etc.
- latency efforts which are a high priority already but will get more attention.
- the explore team is working with the goal
of
increasing user journey length.
- Aside from latency, we have historically not been able to move queries
in
the very short term
in
a meaningful
way. Some suggest changes do so, and some system changes do so.
- there is angst around the metrics - I think this angst is excessive since all good metrics will be correlated - but
we are moving to a metric that will hopefully keep both teams happier and more in sy
nc.
- there are issues around forecasting that have not been fully resolved.
We can improve *engagement* in the short term. I know ways we cou
ld
do this. T
hi
s is the *equivalent
of
rpm
heroics*, but that does not he.Ip you! But we don
't
have the levers (or muscle) to increase queries in the code
yellow way. To a certain extent we wi
ll
develop a bit
of
this with a tweaks type approach to entry points, even
more focus on latency etc. But imho this is
just
far more limited than increasing engagement.
We could increase queries quite easily in the short term in user negative ways (turn
off
spe
ll
correction, turn
off
ranking improvements, place refinements all over the page).
If
we, as a company, want to go there we should
discuss that. It is possible that there are trade offs here between different kinds
of
user negativity caused by
engagement hacking. But I will say that I
am
deep
ly
deeply uncomfortable with this, and
I'd
be surprised
if
the
ads team wants this. The nature
of
how you would easily increase queries is a key reason I do
n'
t like queries as
an end metri
c.
The easy ways are almost all bad.
Ha
ving queries as a metric will, in my opinio
n,
have a subtly
bad effect as a launch metric even if we 'decide not to do the bad things'.
My claim is that our best defense against query weakness is compelling user experiences that make users want
to come back. This is why I care so much about organic shopping info as a driver
oflong
term query growth - it
Redacted
REDACTED FOR PUBLIC FILING
is one
of
the largest unaddressed areas where we co
ul
d easily have a better experience.
We
have not
bu
ilt that
experience desp
it
e my pleas to
do
so for 4 years
or
more. Short term revenue has always taken precedence. I
think we are finally on the right track.
I think
we
are also not thinking consistently about the page evolution. The
se
ar
ch team strongly feels that
we
can
ge
t longer journeys by faster, smaller, in page expansion
s.
Ads is geared towards
who
le page experiences
today.
In
-page expansions (think of the related questions u
ni
t) requir
es
new, smaller, ad experiences
we
are not
yet building.
We
hope that feed
wi
ll
be another such opportunity and are pivoting the GSAs towards
that. Again, this is a case where queries as a metric does the wrong thing for the product/user.
I think the teams are somehow thinking about
the
problem
in
fundamentally different ways and this is causing
unhappiness on both sides.
We
need to fix
at
least the communication and probably the reality to get to a better
place.
I'm
happy to work with you on doing so in the ways you feel are necessary for both teams.
ben
On F
ri
, Mar 22, 2019 at 9:52 PM Prabhakar
Rag
ha
van
[email protected]
> wrot
e:
Let's talk about thi.s Monday please. The current revenue
CY
was addressed by heroic RPM engineering, but
the core query soft
ne
ss continues without mitigation.
At
this rate the full year plan is a bad miss and will need
dras
ti
c steps on the query sid
e.
On Fri, Mar 22, 2019 at
9:
13
PM
Be
nedict Gomes
Redacted @google.com>
wrote:
Congratulations! I suspect an additional good side effect is that some
of
the plans that this ramped up will
pl
ay
our well through the year!
ben
On Fri,
Mar
22, 2019 at 5:46
PM
Jason Spero
[email protected]>
wrote:
+ l appreciate the
pa
rtnership here. Proud
of
the x-fcn approach and
of
the outcome. Well done. J
On Fri,
Mar
22, 2019, 5:44
PM
S
hi
v Venkataraman
[email protected]>
wrote:
Thank you team for the great team effort on
the
cy. Looking forward to the chrome and search related work
st
reams that are continuing to give us a ta
il
wind in q2 and beyond.
Shiv
On Fri,
Mar
22, 2019, 5:40
PM
Jerry Dischl
er
[email protected]>
wrote:
Congrats on the exit and thank you all for
the
partnership that led to this excellent result'
Looking forward to the continued partners
hi
p with the Search, Chrome and Sales teams on the remaining
open issues.
Best,
-Jerry.
On F
ri
, M
ar
22, 2019 at 5:25
PM
Darshan Kantak [email protected]> wrote:
CONFIDENTIAL -
DO
NOT FORWARD
TL
;
DR
Redacted
GOOG-DOJ-04372073
REDACTED FOR PUBLIC FILING
Week 7 marks the end of our code yellow, we met both our exit criteria as of
today•·
Ads and Shopping teams
just hit 1.0 on
Q1
launch OKR with the launch of ·sugarshack". Previously we had already met the other exit criteria
for one week
of
revenue exceeding plan. There are several follow ups from Cersei 19 - (a) improvements to
ongoing collaboration between Search and Ads leadership and teams (e.g., jointly review of Ads•heads-up
process) (b) planned experiments to understand the impact
of
two major Chrome M69 releases and assess if there
are opportunities to claw back some of the query declines (c) work closely with Search product team to drive
suggest tweaks to improve query volume and (
d)
Sales will continue
to
push forward with BFM changes to include
RSA
in
Creative Excellence, and drive keyword coverage.
Our sincere thanks to everyone involved
in
helping us exit this Code Yellow
in
7 weeks
and
facilitating future
collaboration to limit such events.
current revenue performance
Google.com
at confidential to
plan for this
week confidential plan
expected for EoQ) - softer performance compared to
prior weeks driven primarily by EMEA weakness due to recent Brexit news cycle and Dutch shooting leading to shift
in
query commerciality / "user distraction'. Global desktop query weakness persists.
Weekly
Weekly
Performance
Performance EoQ
EoQ Performance
Performance
EoQ
EoQ Outlook WoW
Outlook
WoW
Growth
Growth
7
7 day YoY Growth
day YoY Growth QTD
QTD YoY Growth
YoY Growth
Search
+ Mobile
+ Computer/ Tablet
+AFS
Confidential
Workstream 1: Search/Chrome Query Weakness
Chrome Product: @
Redac
te
d
@redacted
@Redacted
• M72 omnibox metrics confirmed the pre-launch experimental metrics from
M69
-· confirmed the expected query
loss (1.09% for all three ablations vs 1.08% from original M69 exp; non-nav
is
0.38-0.71 %) and Chrome bene
fi
ts
(0.2%-0.5% improvements
in
suggestions usage/ recall/ precision and typing time/characters). Rollback not under
consideration
as
changes were Chrome positive
and
aligned with long term product strategy.
• M73+ forward looking omnibox experiments
are
rolling out, early data available next week (timeline). Launches
subject to Chrome's process.
• New effort to measure impact for New Tab Page shortcuts feature from
M69
in
process: two new search box UI
variant launch candidates, ablation
of
the shortcuts feature for new users are targeting 73/74 (timeline).
• Chrome wi
ll
notify Ads Heads
Up
of
any future launches with query loss. Chrome and Vasco will continue to
collaborate
on
experiments
in
Chrome.
Search
Pr
oduct: @Redacted @
Redac
te
d
• Identified top tweaks to pursue (in yellow at go/cersei-search-prioritization) based
on
headroom analysis, priors
from Explore, Suggest, Growth, and SRP teams, and guidance from redacted@
• Ads
UI
team received Search
UX
approval this week to start Suggest UI tweak experiments
• Search PgM leads @redacted
Redac
t
ed@)
driving program framework and work plan
Workstream 2: Search Ads Launches. Leads:
@Redacted
,
@Redacted
,
@Redacted; @Redacted
•
AQ
team is ramping up Sugarshack
launch (previously
"stretch") to full launch today. This is fine grained format
pricing slack tuning launch that will deliver
Confidential RPM
• Shopping's
Q1
who
le-page RPM impact to date has
been confidential
vs.
our confidential Q1
OKR. Overperformance
is
due in part to accelerating and prioritizing launches with positive whole•page
RPM
for the Cersei effort.
• With this, the team has now launched all projects necessary to push the team over the quarterly estimated confidential
long term incremental RPM target.
R
edac
t
ed
GOOG-DOJ-0437207 4
REDACTED FOR PUBLIC FILING
Wor
kstream 3: Advertiser/Se
ctor
Weakne
ss
- Leads: @
Redac
ted
Global: Updated
BF
M
me
t
rics
for TopsfTails
to
incl
ude
RSA
in
Creative Excellence,
and
d
rive
keyword coverage
via
augmentations
to
optiscore (coming
in
3 months).
Close-The-Gap Plans:
• EMEA:
Progress
on
key Search close-the gap initiatives (dashboard):
o Accelerating OptiScore Usage (+4.3 QTD,
-0.6
wow) SE leadi
ng
on usage at 28% and accelerating
fastest (+0.15pt wow).
Pr
ogress for OptiScore +0.1 QTD, progress from the Ads & Extensions and
Keywords
& Targeting sections are being cancelled out by negative growth
in
Bids & Budgets section.
o Boosti
ng
Travel through peak (95% EOQ,
+0
.3ppts WoW
):
Continued positive trajectory
dr
iven by
UKI (
+0
.5ppts) and EEM (+1.2ppts). EMEA coverage score stable wow (60, +7pts qtd). New Hub Travel
MG
structure and cade
nc
e to be introduced in April for faster learning and scaling.
• Americas:
"Close the
Ga
p" actions
are
Search-focused
to
optimize for quick impact. The Q1 Close the Gap
contest is driving results in Automated Bidding
(2.5pp improvement since contest launch). Tops and Tails KPls
are
also
trending positive with 3 out of 4 KPls seeing w/w improvements (Auto-Bidding: 0.9pp, Optimization Score:
0 .6 and Creative Excellence:
0.
4p
p)
APAC: Close-the-gap plans drove
Confidential
Confidential
.
GCS: Continues to deliver over confidential estimated EOQ performance to qu
ota
on
Search
(and
overall). Operating
business
as
usual: teams are reviewing headroom opportunities across budgets and product priorities where
applicable.
Top
Ad
vertiser Performance (based
on
largest Search quota gaps):
Confidential
•
us
Finance
Confidential to quota confidential Y/Y confidential
EOQ
perf this
wee
k)
• us
Auto confidential to quota confidential Y/Y confidential EOQ
pert this week)
Please let us know if y
ou
have
any questions
Darshan
You received this message because you are subscribed to the Google Groups "cersei-2019" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
Red
ac
te
d
Redacted @google.com.
To post to this group, send email tol
Redacted
@google.com.
To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/cersei-
2019/CALZKkYvcB8OFNZ8VZSCDjErRkiOMDsPUOsvSMmxDtY-O%3D lRbjw%40mail.gmail.com.
You received this message because you are subscribed
to
the Google Groups "cersei-2019" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
Redacted
Redacted
@google.com.
To post to this group, send email tol
Redac
t
ed
@google.com.
To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/cersei-
2019/CALysAStaF drVf9ne0M-xOY%3D6sK58Y74UOVOW J H4qk.KEASufbA %40mail .gmail. com.
Sent from Gmail Mobile
p.r.
Redacted
GOOG-DOJ-04372075
Sent from Gmail Mobile
REDACTED FOR PUBLIC FILING
Redacted
GOOG-DOJ-04372076