Intro: Google Search Fallback on Assistant
Challenge + Response

Role: Design Lead for Google Search Fallback on Assistant
XPA Teams > Pixel, Speech, NLU, Assistant, WearOS, and many more
Launched: 2022 across Google Assistant surfaces

Our goal was to help users find the right content where Assistant fails to answer on Smart Displays. Across Google Assistant and Search we’ve evaluated ~17M punt queries in month over month on Smart Display devices. PUNTs create user friction in user’s assistive journeys and don’t provide reliable results when users are searching using their voice. My role spanned vastly to drive the UX direction, designs and partner with XPA teams.

I helped to define some core principles our teams agreed on to set the framework for when and how a PUNT should be handled.

  1. Help users with their journeys.
    Not all PUNTs are a good fit for Search experience. Build mechanisms to fulfill queries that Search is helpful to user’s journeys.

  2. Don’t hurt Assistant branding.
    We should not hurt Assistant branding by suggesting search experience for irrelevant,  sensitive / inappropriate use cases. 

  3. Scale easily.
    Our solutions should scale to multiple surfaces, languages and use cases easily. 

  4. Answer fast.
    We should not add any latency for overall Assistant experience on DG. 

  5. Don’t hurt the web ecosystem.
    We should not have any negative impact on the ecosystem (e.g. do not hurt clicks, monetization, impressions etc).   

Part I: The investigation begins…

Our MVP goal was to only trigger queries that are information seeking and could be fulfilled by the Search. Some punts are necessary to educate users on accomplishing certain tasks like device action, account linking etc. We will not fulfill educational punts or punts that are due speech recognition errors, wrong TTS handling, seeking personal info, device actions, etc. Part of this MVP is to learn.

We will never have a perfect triggering and there will be cases we will over trigger in vertical punts. But we will take the following measures to identify new punt patterns and tune the triggering accordingly. 

  • Do continuous evals and evaluate the losses (within the team).  

  • SxS evals to maintain W/L bar for launch and identify loss categories. 

  • During LE, run an arm to ask users if they would like to see Search results and collect the feedback to further tune in signals.

Part II: Design

Through multiple iterations and discussions there were clear objectives I had to figure out to make fallback work on smart display devices.

  • Enable a VUI hand-off to guide the user (i.e. “Here are some results from search”) And also testing the VUI handoff in a multiple-arm LE to track the success rate of the PUNT.

  • Glanceability

    • Increase font / content size overall (~25% minimum or force browser zoom 1.5x / 2x)

    • Content should fill the screen; min. 24 pt

  • Rendering SERP 

    • Disable ads 

    • Hide Search bar 

  • Content should be accessible for users including tap target

    • Links from the SERP will open natively in SD including with existing apps (e.g YT links will open with YT app on SD).

Part III: Ship It and Repeat!

We have launched Search Fallback on Smart Display devices. But there are several cases where users don’t want to see Search results and just want it to return, thus giving negative feedback in the feature survey. In order to give users the option to choose to see or avoid the Search results, we will bring the dialog flow to the Search Fallback feature and let users decide. Below you’ll see an example conversation flow that asks permission before surfacing search results.

In addition to gathering user data and enhancing the e2e flow our team also improved how we handle discovery to make it that much easier for users to discover and search on their own. Partnering with mobile leads to define entry points on the discover screen and input plate. For more information on how Search Fallback expands to other surfaces including Pixel Watch. Check out this page.

Here’s a quick look at some of the difficulty, leadership and impact metrics for this project…

Difficulty & Leadership

  • Guided UX vision and secured buy-in from Search and Assistant leads by defining MVP feature requirements solution for Search Fallback while driving an iterative design process. This resulted in strong impact and punt reduction as seen in the ‘Impact’ section.

  • Proposed and designed new UX patterns for rendering search result pages within iFrame. Adjusting UI to hide the search bar and increase element sizes (e.g. font, image, etc.) to fit the surface indented.

  • Established new TTS and Voice inputs for hand-off to indicate to users that Search results are returned. 

Impact

  • 67% Punt reduction on Smart Displays

  • +1.88% Assistant DAU and +2.02 Assistant Queries

  • +2.72% Multi-day slice return rate

  • 5.73 SxS Win/Loss ratio (significant increase over previous UX)