Five lessons learnt from launching an AI chatbot with Abbie Chatfield
Late last year, Tourism NT introduced a new offering, ChatNT, an AI travel expert empowered by media personality Abbie Chatfield. Here, Tourism NT's director of digital marketing and data services, Kristie Beattie, reveals the five lessons they learnt from the launch.
In July this year, a discussion began at Tourism NT about the prospect of launching an AI chatbot imbued with Abbie Chatfield’s signature tone and personality.
Our Domestic Marketing team and their project agency, Edelman, had earmarked Abbie as an ambassador for Tourism NT’s 2023 summer campaign, but they wanted to bolster the PR potential of the campaign by leveraging the pop culture buzz around ChatGPT – to get interstate audiences talking about summer in the NT in a different way, inspiring them to travel here, just like Abbie did.
The idea? Launch ChatNT: the first Australian branded AI chatbot to generate celebrity-like conversations and interactions with audiences.
Abbie Chatfield
ChatNT was envisaged to be a campaign-based tongue-in-cheek travel inspiration tool powered by Abbie and OpenAI, that would get mass audiences thinking (and talking!) about summer holidaying in the Northern Territory.
Keeping it tied to a campaign has the added benefit of managing user-expectations, therein helping us safeguard the Tourism NT brand as we made the first leap into working with generative AI. The performance and learnings from ChatNT could then be applied to any future AI-based travel planner tools for Tourism NT.
And so, the question that ultimately landed with me was: can we do it?
As someone who has worked in digital marketing for more than a decade, I was aware that AI is by no means a new technology in PR, media and marketing. In reality, it has been the backbone and reason for the success of many media platforms that we use on a daily basis. But ChatGPT has brought AI to the mainstream, and more and more brands are looking for creative and innovative ways to generate value in this space.
So my answer to whether we should embark on this project was an easy ‘yes’ – however, the details around how we’d get to the end result remained a bit murky.
To help, we made the decision to partner with Microsoft and built ChatNT using OpenAI’s GPT-3.5 model via Microsoft’s Azure OpenAI Service, which gave us access to the same technology behind ChatGPT. To keep ChatNT on-topic with summer in the NT, we utilised Azure Cognitive Search, a semantic search index tool to ingest and index Tourism NT data. Azure Cognitive Search was, at a basic level, how we told the model: “if somebody asks you a question, here is where you’ll find the answers”.
ChatNT would go on to become an Australia-first when it successfully launched in November 2023. Since then, ChatNT has sparked thousands of conversations, increased time spent on Northernterritory.com by over 70%, contributed to a social reach of over 1.54 million, and generated more than 100 pieces of top tier domestic media coverage.
And while the development process went fairly smoothly, like with any project, there are always key takeaways.
Here are five of the biggest lessons we learnt through launching ChatNT:
1. Create a data framework
Before you start collating the relevant data for your AI chat tool, consider your project’s objectives, your brand and/or content pillars, and your customer journey, to create a framework optimised for quality responses. Like a carefully curated wine list ordered by country, region and grape variety, identify the hierarchical structure of your data (or in our case, our content) to help the AI know where to go for the most important and relevant information.
For us, this looked a bit like this:
What is summer in the NT? > What are the regions of the NT? > What can I do in the NT? > What are the tourism products in the NT? > What is the travel advice for the NT? > What are some miscellaneous topics related to summer in the NT?
Once you have a structure in place, you can start gathering your content. In the instance of ChatNT, we discovered that instead of several lengthy documents covering a wide range of topics, the Azure Cognitive Search service was better at digesting a larger volume of shorter documents.
Without this structure and approach, there was a risk that ChatNT would provide answers that were less relevant or not helpful to the user.
2. Identify content gaps up-front
Once you’ve created a data framework, begin identifying your content gaps so your chat tool has a wide range of knowledge to meet user-expectations of quality in conversation and response accuracy and diversity. Continuing with the wine list analogy, a sommelier would quickly recognise if their wine list was too heavy in Barossa Shiraz and should include more Grenache for that region. Identifying content gaps also relies on having an innate understanding of both your brand and your users.
In our case, we identified the need to have more content on cafes, breweries, canoeing and New Year celebrations.
It will not likely be possible to identify every single question related to your products and experiences, so in lieu of this, it’s important to use your System Prompt to teach your AI what to say when it does not know the answer to a question.
3. Contextualise everything
Contextualising is about adding as much explanatory information into your Source Data for places, products or terms that may seem commonplace, but can in fact be very obscure for your AI tool and its users.
For ChatNT, this included providing context around region names such as the Top End and Red Centre, and helping the AI understand that ‘salties’ were saltwater crocodiles and ‘freshies’ were freshwater crocodiles – and that ‘crocs’ could be either ‘salties’ or ‘freshies’ (confusing right?).
The lesson here is that you should assume your AI has a base level or zero knowledge about nuances pertaining to your brand or product. You need to give your Source Data as much context as possible. This is key to the AI being able to generate robust and relevant answers to user queries, and the development of a conversational, free-flowing ‘human-like’ tone we have all become so used to when engaging with ChatGPT.
4. Refine your System Prompt
A System Prompt is what gives your AI its personality and purpose. It is made of up instructional messages to influence and govern the behaviour of the language model, which in our case was OpenAI’s GPT-3.5 model.
The System Prompt is where you tell the model who it is, what tone of voice it should use, how long its answers should be, how it should greet and say goodbye to users, and any other relevant parameters that safeguard your brand, product and even the build of the chatbot itself.
For ChatNT, we worked with Abbie Chatfield to get her input on her favourite phrases and typical greetings and goodbyes, which we then transposed into the System Prompt. However, testing showed us that we needed to constantly review and refine the System Prompt to get ChatNT to behave the way we wanted it to. We were up against time on this one, so the lesson learnt is to factor in extra time and resources to test and optimise your System Prompt.
5. Testing is king
Just like with the build of a new website or app, the principles of user-acceptance testing need to be applied to the build of an AI chatbot. However, what differs slightly is who makes up your testing cohort and when ‘good enough’ has been achieved.
Typically, you would design the bulk of your testing efforts around your audience segments and ask them to test against specific use-cases or experiences.
However, for an AI chatbot, you really need to expect the unexpected by going broader and almost allowing for a free-flow testing approach to uncover unknown insights and reveal any conversational or knowledge gaps.
Kristie Beattie
For ChatNT, we initially designed a fairly robust testing program against specific themes and topics but later discovered the best results came when we simply shared the test link to our friends and family. Who would have thought we needed to cater to sky diving in the NT or get super detailed on the particulars around kite surfing?
The good news is that optimising Source Data following testing is fairly straightforward. However, with the infinite number of possible queries and the very finite amount of time and resources, it is also important to draw a line in the sand and press go when you’re confident your chatbot is delivering on your original objective in a way that is safe for your brand.
No AI chatbots are full-proof or perfect yet, so perfection should not be your benchmark. If you’ve optimised your Source Data, tested robustly, placed visible disclaimers on your chatbot, and enacted a system to promptly address user feedback, you’ll have ticked the key boxes.
For ChatNT, our main objective was always to create an above-the-line PR activation that would spark mainstream conversations about summer in the NT. And with Abbie Chatfield on board, we were sure we’d get people talking!
