Jamie Ferguson

Process

Phase 1: Discovery

This project spun out of a broader project where we were looking at what a more personalised experience could mean for our users.

Our first concept that led us down this path was an augmented search experience which we tested with some young people. While we got positive feedback around the idea, the actual implementation was not found to be ideal — young people were familiar with search working in a certain way and a generated answer was not seen as fitting with the established mental model at the time (this was before Google introduced their 'AI Overview'!).

This experiment also coincided with us working with a data scientist who was able to create a working proof-of-concept of a question and answer based tool using a RAG (Retrieval Augmented Generation) based approach. Impressed by the answers that were generated, this gave us the confidence that the idea was viable and enabled us to create an alpha version that we could put into production.

Phase 2: Experimenting in production

While the organisation was generally excited by the idea it was clear that there was understandable concern around how something like hallucination could damage the brand. Even though the use of the RAG concept meant that hallucination was extremely unlikely we decided to avoid generated answers altogether for the alpha release and instead use an LLM to 'match' the user query to a pregenerated question and answer pair.

As the main purpose of the first release was to try and understand how young people would interact with a tool such as this, I set up a dashboard in Heap to monitor how it was being used, and we also kept a log of the ~10,000 queries asked by young people over the course of the alpha stage.

Phase 3: The shift to generative

This wealth of data enabled us to validate the need and to start to understand the main use cases young people had for a tool such as this.

With this increased confidence in both the concept and capabilities with the new tooling we set about figuring out how we could move from the pre-generated responses to generative. This was a truly cross-functional endeavour, with members from clinical governance, content, tech, product and myself all embedded in the core project team.

Along the way the organisation also felt that conversational AI would become a core component of a broader 'first step' strategy that could bridge the gap between our traditionally content driven core approach and the 1:1 human support of PeerChat.

Outcome

With the organisation fully behind this new strategy we launched an MVP version on the ReachOut homepage, as well as a widget on every page of the site.

Myself and members of the core team proceeded to review user queries in Langsmith and continued to iterate on the core prompts and safety guardrails.

We are continuing to learn and iterate, with the next main focus being how this tool can better prepare young people to connect with the right human supports at the right time.

Process

Phase 1: Discover and define

After doing a full review of the current platform, I conducted some interviews with SMEs in the organisation to try to understand the pain points and requirements.

Rather than build another bespoke solution, I suggested we consider buying and integrating a 3rd party platform. Having previously worked at a SAAS company I knew how much effort it takes to build and maintain a platform and it didn't make sense with the kind of team we have at ReachOut.

Once the business had agreed to this approach, the Product Manager and I began researching potential platforms, eventually decided on two different options - each with its own set of compromises. This also coincided with the hiring of the Coaching Team Leader who was able to work with us to find solutions to those compromises.

I ran some user research activities to better understand the needs of parents trying to support their teens through tough times, as well as testing the performance of the current landing page.

This highlighted the challenge that the concept of 'coaching' was not well understood in this context, so we looked for ways to improve how we communicated the service.

Phase 2: Ideation and testing

While narrowing down platforms I worked with a Content Designer to design a prototype onboarding flow.

We then tested that flow with some parents to get a sense of any remaining friction, along with some revised landing page concepts. We refined the prototype as we went and ended up with a flow that was significantly improved from the current state.

Phase 3: Design and develop

With the platform decided on and a clearer idea of how best to communicate the programme to parents, I was able to finalise the design of the landing page as well as start to build out the final onboarding flow in the new platform.

As we had opted to go with a 3rd party platform, the actual UI design requirement of the coaching experience was limited to the landing page itself, however I was heavily involved in the experience design within the platform itself as well as helping to support the coaches in transitioning to this new platform.

Outcome

This change did not come without its challenges. As well as the inevitable difficulties that came with the coaches needing to become familiar with an entirely new system, there were also some functional issues with the platform we chose that, despite it being the best option available, will never be as fit-for-purpose as a bespoke solution.

However, we now have a much-improved booking experience for parents, and more importantly one that can be updated as needed with zero engineering effort. We also ended up with a landing page that better communicates what is a difficult-to-understand proposition, and that better reflects the ReachOut brand.

The end result is that the coaching team have been able to easily iterate on the service, allowing them to consistently meet or exceed their session targets.