Search curation: Making content more findable and relevant
COMPANY
Microsoft
MY ROLE
Senior Content PM
CONTRIBUTIONS
Content Strategy
Project Management
Content Training Programs
Workshop Development
When Office customers need help with a product, they typically start with a search. They may seek out answers on Office.com, our support site, or inside a product, using a Help pane. Often, they start on Google.
An analysis of the top 100 queries for each Office product revealed that while our writing teams published lots of content, users struggled to find it. If the content did appear in top positions on the search results page, it didn't always answer the right question.
This discoverability issue was so significant that we needed an all-hands-on-deck effort to fix it. With partners on the Office Search team, I led a three-month, org-wide change management initiative to:
.
-
Raise relevance scores for top 100 queries across 6 Office products.
-
Train 75+ writers to identify, prioritize, and better address the key search intents driving volume for top queries.
-
Determine how to scale analysis and optimization efforts.
Understanding search intent
I worked closely with our Search team to identify the goals, outcomes, and success measures for a search curation project, partnering with one of their Content Project Managers to run the project. While fuzzy ownership can be confusing, in our case, sharing the authority and accountability made sense and worked well.
Initially, I led a pilot project for Excel to work out a reasonable, repeatable process for improving search results for specific queries. With the methodology defined, we were ready to train the entire organization to review search data for their top 100 queries, to identify gaps and opportunities.
Meanwhile, complementary projects were happening on the engineering side to improve our internal search engine and address technical SEO issues impacting our ranking in Google. I was involved in each of these efforts, too, as part of a larger initiative to tackle search and discovery challenges.
Training teams to identify intent
The Office content organization consists of a large team of writers who produce how-to articles, videos, technical guides, and other types of web content for Office 365 users, IT admins, and developers.
After the initial project kickoff, my partner and I developed and delivered a two-day workshop for these teams, where we taught them how to assess the quality of search results for their top queries.
High volume queries or keywords are often single words (archive) or short phrases (archive email). Assuming we know what someone is expecting to find when they enter such a query is dangerous. Instead of guessing, we taught writers a quick way to uncover the top 3-5 search intents for each search query, though a lightweight analysis exercise. This knowledge helped writers determine if relevant content was showing up in search.
The goal of the curation project was to fix existing issues with content findability. But it was also about getting teams to adopt user-focused, data-driven planning practices. Starting with user needs set us up to deliver more effective content, right from the start.
Overcoming resistance to change
This was my first time leading a large change initiative, let alone one that required a large team of content professionals to temporarily (and immediately) shift focus from their own goals and priorities.
We had top-down support from our executive team. That was critical. Yet, this project ruffled feathers. Some felt that we were questioning their expertise. What helped here was gently but relentlessly restating the case for change, using examples and customer data to reinforce the need.
Interestingly, our teams had recently taken a personality test, sharing the results with each other in a workshop setting. Knowing the work style and preference for each writer was an unexpected gift. It helped me understand why certain people reacted negatively or suspiciously. I was able to adjust my communication style, depending on who I was talking to.
Over time, we overcame most of the initial resistance by getting teams to understand and empathize with the friction in our current search experience. Then, it was a matter of identifying champs, amplifying success stories, learning from losses, and measurably improving experiences for our customers.
Outcomes
A few highlights of doing this work:
-
More relevant, useful content: The curation push resulted in 55 new videos, 92 blog posts, and 88 new or revised articles and training courses, to plug gaps and fix poor search experiences.
-
Improved search results: Among other quantifable results, our NDCG ("search goodness") scores for the top 100 queries for our 6 most used products were 14 points higher than scores for a randomly sampled set of queries.
-
Scaled delivery: To scale beyond our top products, I hired and trained a team of vendors to do ongoing intent analysis and curation for an expanded set of Office products and languages.
-
Content leadership: Ultimately, the curation initiative proved the value of doing deep analysis to understand what customers are searching for and why. It also paved the way for a specialized content analysis role at Microsoft.