Harrison 
In late July, we announced OverflowAI, which involves some exciting new features—one of those features being OverflowAI Search. This post will take you through research, activities, and milestone decisions that led us to the alpha we recently launched.
Background and context
The emergence of new generative AI tools and technologies has changed the way users search for information. It’s now easier to find answers in an instant and frictionless way, providing a natural conversational experience to narrow down your queries. Access to this new way of finding solutions has led users to change and adapt their expectations.
Since these AI tools are still relatively new, there are some known issues. For example, LLMs are known to hallucinate to fill knowledge gaps, raising concerns about accuracy. Stack Overflow, on the other hand, continues to be a more reliable resource than LLM. This is because users still trust his Stack Overflow content quality and the large number of human technical subject matter experts on the site.
These changes in user expectations and behavior have created interesting problems to solve. How can you reduce the friction in finding answers to questions based on trusted content?
Strategy
Our expectations for finding answers quickly and effectively are higher than ever, and our patience for time-consuming methods is decreasing. We had the opportunity to adapt to meet these expectations and improve the way users search for answers on Stack Overflow.
To reduce frustration when resolving issues, we’ve put together a strategy that focuses on what your product can do to help users more effectively. The goal of this strategy was to drive user retention and engagement in a way that is useful and meaningful to users. To achieve this, we launched a design sprint.
Design sprint
Design Sprints allow cross-functional teams of product managers, designers, engineers, researchers, and community managers to quickly solve challenges, ideate solutions, build prototypes, and deploy them to users in five days. and can be tested.
Our goal during the sprint was to “reduce the friction in finding answers to questions.” Sprinters used the following metrics to show that their goals were being met.
- Reduce time to get or find answers
The group looked at the current user journey, mapped the “What should I do” questions to that journey, and identified the best areas to focus on. These areas are likely the two loops that cause the most friction for the average user.
- Review each piece of content to determine if it is helpful/relevant and consider any related questions.
- Testing the solution > Refine questions/queries
We knew from past research that these identified loops and friction points were real and that the scale of the problem determined the amount of time users spent in the loops. We also know that the ability to articulate problems is a skill that engineers develop through practice.So the group asked themselves
- How can you help users more quickly identify the content they really want?
- How can you refine your questions and make the journey to answers smoother?
Problems with search today
Now that we know we’re tackling the problem of finding content faster, we dug deeper to better understand the current state of search functionality and its biggest limitations.
- complexity and confusion: Users often struggle with Stack Overflow’s search interface and may even need guidance on how to use it effectively. The results can be inaccurate and checking the results can be tedious.
- duplicate question: Duplicate questions are asked on Stack Overflow when users can’t find an existing answer due to low relevancy between the survey results and the search. Duplicate data is then closed, resulting in a poor user experience.
- Dependency on external tools: The current search experience on Stack Overflow often fails to meet users’ expectations for search accuracy and relevance, forcing users to rely on external search engines.
- Changing user expectations for content discovery : The rise of AI tools like ChatGPT is changing the way users get information. More users are relying on his AI for quick answers, and their patience for sifting through search results may be wearing thin.
Ideating on problems, goals, and solutions
If you’re familiar with design sprints, you know that there were many ideas that our group considered. But after much brainstorming, iteration, and refinement, we narrowed it down to the following set of problems, solutions, and goals. For more information, see the Overflow AI Search announcement post.
- problem: Users have difficulty finding relevant answers on Stack Overflow.
goal: Provide answers that match the user’s intent and make search results more relevant.solution
: Improved search results with a hybrid solution of Elasticsearch and semantic search. - problem: Users find it time consuming to browse through different questions and answers.
goal: Reduce time to find relevant answers while leveraging community expertise.solution
: AI-powered search overview for the most relevant answers - problem: Users may have difficulty articulating or identifying the problem.
goal: Unique answers speed up time to answer and reduce the number of duplicate questions.solution: Conversational search refinement
Testing our assumptions
The last day of the sprint was dedicated to research, presenting preliminary designs and proposals to users, and capturing the next key takeaway.
- Speed and immediacy of response are important. This confirmed previous research regarding the friction currently experienced by users. Users want to get answers to their problems as quickly as possible. They prefer asking a question to a colleague or to an AI rather than asking on the site, as it takes time to create a question and users have to wait for an answer.
- Proper search refinements and signals can help speed up the process. Many users know what elements of the question they want to narrow down to (such as tags, version, recency, etc.), but you can learn more here. Users found it valuable to be able to directly copy code snippets. It was also important for users to see votes and other helpful metrics. While it’s helpful and desirable to have AI select important information and accurate summaries, you still want to see search results side-by-side so you can act quickly.
These discoveries help confirm that our problem is legitimate, and the solution we were seeking is a combination of finding a solution to a problem that cannot be articulated and one that has already been asked. We’ve identified it as an interesting way to bridge the gap between finding solutions to problems. Additionally, users were excited that Stack Overflow, the world’s largest source of developer knowledge, was trying to help users find answers faster.
Continuous improvement with design and research
Following the design sprint, we moved into a weekly design and research sprint where we presented mockups and prototypes of our brainstormed solutions to a mix of long-tenured and new Stack Overflow users. This allows us to directly measure user reactions, assess the perceived value of these solutions, and understand user expectations in a more concrete way.
The feedback gathered from these sessions was directly reflected in the development of the search solution. We iterated on these designs weekly and used these learnings to refine and adjust features to better meet the needs of our users. From our conversations with users, we’ve learned that the following principles and insights are critical to the success of this feature:
- AI as a flexible and seamless option:While there was a general sense of excitement from survey participants about Stack Overflow’s early exploration into AI, we still wanted adoption of an AI platform to be a seamless and flexible experience. . This allowed us to enhance the existing search experience by adding an AI summary of the most relevant questions and answers alongside your search results. Users can always choose to see improved search results instead of digging deeper into the overview. It also extends the experience by allowing users to join the conversation if they need additional help narrowing down their questions. Alternatively, if the user wanted to start the conversational search experience immediately, they could do so.
- Highlight sources and recognize our community: Although many research participants are using AI tools, there is still some degree of skepticism about the capabilities of AI. They still consider Stack Overflow to be an essential source of information, especially for complex problems that require human expertise. With this in mind, we wanted our solution to be able to highlight trusted and verified content from the community by prominently displaying the sources used. Participants liked being able to see citations that show the origin of AI content and being able to dig deeper into the sources used. They expressed concerns about voting and reputation regarding sources. We simplified the design by placing voting arrows next to the sources, letting users know that they can vote on individual sources just like they would vote on answers.
- Measuring confidence: Early on, we considered the idea of displaying a confidence indicator of the quality of a user’s answers. We found that users value answer quality metrics based on human feedback, such as the number of upvotes for a source and the reputation of those who respond to the source. This highlighted how important it is to focus on the human interactions that exist within communities. As a result, we display these metrics alongside the source to help users better understand the quality of the answers.
- Challenges in giving credit: Participants offered a variety of opinions on how to properly recognize the sources of information that inform AI responses. Some advocated giving votes and reputation to all sources, while others believed that sources should be given credit according to their actual contribution to the summary. In particular, concerns arose about giving credit to sources that were of low quality or did not contribute enough to the answer. This insight also fed into the decision to break down votes to the individual source level. However, this is still an unresolved issue. In the alpha version, we are allowing users to vote on sources, but not awarding reputations as an interim step to learning how to best balance.
- Expectations for accuracy, usefulness, and relevance: Our research shows that users value Stack Overflow for consistently providing reliable, high-quality information and setting high standards for accuracy. Implementing hybrid Elasticsearch and semantic search can help you get search results that better match your questions. The main focus of the alpha version is to measure and improve the quality of insights provided by AI responses.
- Importance of user feedback: User feedback was key to improving the AI. In an early concept for user testing, we tried an approach that allowed users to upvote the entire AI brief in exchange for providing feedback. However, there was confusion as to whether they were supporting the AI, all sources used by the AI, or just providing feedback to the AI. This allowed us to clearly separate AI feedback and upvotes for sources.
Final thoughts
We’re excited to start making improvements to Search Alpha. We would like to thank those who have already contributed to this process and recognize the hard work that our users have already done to share their input in weekly sprint sessions. That feedback is already influencing and shaping this feature, and we’ve learned a lot over the past few months.
At Alpha, our goal is to continue this process of learning and building with our community. Search and search improvements are not set in stone and are still in development. As you roll out your alpha version to a wider audience, keep in mind that feedback during this period may impact the final product.
Ultimately, we hope to achieve our mutual goal of helping users find answers to their questions faster and more efficiently, and reducing a lot of the friction they currently experience.
If you are interested in the technical details of implementing semantic search, check out this detail.
