Business Analysis Homebase

Terms and introductions

Chatbots are a conversational platform for individuals to ask questions and receive answers in an automated fashion. They strive to emulate human conversation, without a human getting involved.

Different products use different words, but in order to make a chatbot, you’ll need content in a particular format, using the below general terms.

• Category - ​The category that your response falls under
• Key Question - ​A core fundemental answerable question that will trigger the response, what all other utterances will link off of.
• Response - ​The reply from the bot that is provided when matched
• Chips - ​Suggestion buttons/prompts that will appear underneath the response to allow users to click through, or control the conversation
• Utterances - ​All of the other forms/variants of a key question that will trigger the same key quesiton response. Used in training the AI if it uses machine learning, or used for fuzzy matching if using rudementary NLP

Natural Language Processing is the big buzzword behind chatbots. There are 2 major types at the moment, Machine Learning and Fuzzy Matching.

1. Machine learning. The AI will develop and understand the context, usually with hidden tables provided by the vendor that translates some information so you do not need to manually populate it. The model becomes more sophisticated with the more training data that is provided. This is more akin to your Generative Pre-trained Transformer 3 (GPT-3) and OpenAI.
2. Fuzzy matching. In premise, the bot takes your question, and compares the characters to the list of utterances. Which ever utterance has the highest confidence (by matching the % of characters presented in the question in the same order to your utterance) will trigger the response. There is very little “intelligence” behind this methodology, and is the equivalent of an excel table looking up a response with a fancy front end. No vendor will admit to using this, and all will attempt to convince you they are using #1. You must be very careful about this!

Early advice

The most common agreeable comment from every vendor engaged was that the content of the bot is the most important aspect, and that the technical platform itself is replaceable in comparison. Having well written and planned out content is the most valuable part to the development of a chatbot or virtual assistant. Content of high quality must include the original intent, the answer, the flow, contextual implications, and diversity of utterances/alternative phrases. 

ℹQuick content tips:

• Keep responses under 350 characters for readability.
• Never start a response with “Yes” or “No” in case a false positive triggers, and then you give people the wrong response.
• It’s better to send 1 response, than 3 responses one after another.
• Use chips/button prompts to guide the conversation.
• A large volume of diverse utterances/alternative phrases will make for a better user experience.

The main feature that separates lower maturity and high maturity chatbots/virtual assistants is contextual capability; the ability to request further information (location/time/occupation/device/preference/status) to provide a correct answer or highlight/utilise important information already provided in a lengthy statement to further personalise the response. Our investigation found most vendors offering chatbot “AI” platforms are incapable or have high difficulty handling context (e.g. the location portion of the sentence “Do I need to wear a mask at a café in XYZ suburb”) which we learned too late in the process, and may be a terminal blocker to your work as one author has personal experience with.

In general, a degree of human intervention is required; Humans are required to write/rewrite content into a chatbot-suitable format, develop the flows/scripts/chains of actions, provide suitable alternative utterances/phrases, and validate the correctness of the answers provided. The technology available commercially at the time of drafting is not at the point of providing automated continual improvement, nor the level of intelligence required to automatically crawl existing web content and simplify it into a suitable form. With the development of new language processing platforms such as Generative Pre-trained Transformer 3 (GPT-3) and OpenAI, in future years this could become a point approaching complete automation. 

Your content should be static if possible. Rapidly changing content, topics, and utterances will be hard to track, hard to update, and put additional burden on both the content writers as well as the AI itself. Chatbots work best when information remains the same so both users and the bot can learn what to provide to lead to the right answer and ultimately outcome. Hypothetically, writing the answer once and updating/adding additional training utterances daily would be the best practice, but realistically some answers may need to change over the life-time of the bot.

In regards to capability of each vendor, it can be painfully difficult to track and assess. There are so many vendors and all have evolving products. Typically vendors don’t provide detailed roadmaps given the sensitivity of the intellectual property behind them, which makes it even more difficult to gauge capability. Every team/department/organisation has different chatbot requirements, so it would be useful to understand what use-cases you’d like to address through this program as that could change your preferred vendor.

You can engage directly with vendors for sales pitches (with some such as Microsoft offering free developer time and Proof of Concepts) but it’s recommended to have considerable caution regarding over-promises of critical features such as contextual understanding (which is often a series of hard coded utterances creating the illusion of context, which is prohibitively expensive to maintain ongoing). If you do intend to approach vendors, it is highly recommended to provide a sample set of test data and scripts that include large variable context in order to evaluate/demonstrate the capabilities end-to-end. Again, your content is the most valuable resource so do not provide a fulsome copy of all intended content. A live test of contextual information helps immensely, and these samples are good ones to use for testing:

Key Question 1: “How do I install an application on my Windows PC?”

Answer: Double Click the .exe file

Key Question 2: “How do I install an application on my Mac PC?”

 Answer: double-click the disk image or package file (looks like an open box)

When the user asks “How do I install an application on my PC” what is the follow up question? Does it ask a follow up question? Is that follow up question hard coded by the vendor? Is the platform a genuine AI capable of detecting that there is missing context required to fulfil this request?

Key Question 1: “Can I see the Sydney Harbour Bridge in Victoria?”
Answer: “No, you need to travel to New South Wales”

Key Question 2: “Can I see the Sydney Harbour Bridge in New South Wales?”

Answer: “Yes, although you would need to be in the CBD”

When the user asks “Can I see the Sydney Harbour Bridge” what is the follow up question? Does it ask a follow up question? Is that follow up question hard coded by the vendor? Is the platform a genuine AI capable of detecting that there is missing context required to fulfil this request?

⚠ Reminder; Never start a response with “Yes” or “No” in case a false positive triggers, and then you give people the wrong response. We just did it for a demo purpose.

Do not build in live handover capability if you don’t have the resources to make use of it. On the other side, it is unlikely all conversations can be handled by the chatbot, so you may require a call centre to intervene when a conversation is failing. Users would rather a chatbot fail to answer the question 3 times and then be directed to a human operator (via the same interface) than a chatbot fail to answer the question 3 times with no options to how to proceed further with their enquiry.

So you still need to go ahead with it after all these warnings?

Here are some sample requirements to get you started:

Additional reading:

Gartner did have some good insights which was published after our development, worth a read

Stefan Carton

Updated: 2/08/2022