The possibilities of artificial intelligence are almost limitless – creating videos, writing texts, coding websites. We also became curious and tried our hand at a custom GPT: the exhaustion bot. Our experiences in the practical report.
You can read exactly what the exhaustion bot can do here. In short: it knows the data from our exhaustion studies and can answer questions, develop hypotheses and interpret data on this basis.
You can try out the beta version here.
Why do we want a bot that tells us what we already know? To test how it works. What data can the AI process and how, what does it manage, and where does it start to hallucinate because its limits have been reached? We wanted to learn from a practical example.
The challenges:
– Data preparation
– Definitions
– Testing
– Error
Custom GPT – what is that?
A custom GPT is a way of customising an artificial intelligence in ChatGPT. In a mask and a chat function, you can feed the AI with information and instructions on what and how it should be. For example, if I need an AI as an author for my LinkedIn texts, I create a custom GPT, send it a document with several texts I have written and instruct it with ‘You are my LinkedIn author and write texts on my suggested topics that are similar in style to my sample texts.’ I can also fine-tune details such as emoji quantity or links to the website. So easy, so practical. And as soon as something bothers me, I write to the AI in the settings to tell it what to change.
Data preparation
Back to the exhaustion bot. The task prompt was written quickly, but it gets tricky when it comes to data preparation. How do I give the AI the data so that it can categorise it in the right context and compare several different survey fields with each other?
The study brochure as a PDF is not suitable, as we wanted to give the bot all the survey results – even those that didn’t make it into the brochure. We also wanted to withhold our interpretations and conclusions from the bot, as we are interested in what it can achieve at this point.
The bot combines the answers to various questions to form hypotheses.
From Civey, the market and opinion research institute with which we conducted the studies, we have data sets for all questions with a breakdown of the answers according to various socio-demographic factors. We transferred these into tables and gave them to our custom GPT and immediately realised that this didn’t work either.
The unprocessed data tables are too confusing for the GPT: too many design elements, empty rows, redundant row and column names.
In a second attempt, we normalised the tables, i.e. deleted all duplications, created many small sub-tables and provided them with keys.
The table links the exhaustion intensity, i.e. the exhaustion value on a scale of 1-10, with the Enr (1=not exhausted, 2=exhausted, 3=neutral) to form three groups.
The table links the intensity of exhaustion with the frequency generated by the respondents.
This worked out much better, the bot was able to allocate some things. However, in this case the AI also needs the explanations of what exactly the tables mean, stored in the settings. This also worked smoothly for one survey period, but the bot was unable to compare the results of two years of studies.
The preparation is much more intuitive, but the bot does not run without errors because it does not understand the hierarchy of the questions.
Back to tables, which are more intuitive at first glance. The version of the bot has proven itself longer than its siblings, but after the standard questions it becomes apparent: the study design with its changing sample sizes (total, employed, people who answered question 2 with x, etc.) is not correctly categorised. The bot compares answers to different questions that are not comparable.
The bot lumps the answers to two questions together, even though the samples differ and the answers are therefore not comparable.
The solution: We have coded the questions. Each table row is clearly labelled with a numerical code, the codes show a clear hierarchy and affiliation to each other and to the questions.
So if question 1 in 2023 has the code 01_00_23, the answer options of an evaluation option are 01_01_23 to 01_08_23. At the same time, question 1 of the following year 2024 is 01_00_24 with 01_01_24 to 01_08_24. It seems complicated, but once written down it follows a simple logic that both humans and machines can process. We have written the explanation of the code in the bot’s settings so that everything is clearly defined.
Definitions
It is possible to prohibit the custom GPT from searching the internet. In our case, we want to ensure that all numbers come only from us, which is why we have forbidden it to do just that. That’s why we’ve also given it the most important definitions in the settings – otherwise, without Google, it will simply start making things up.
In our data, there is an analysis by religion, whereby only ‘Catholic’ and ‘Protestant’ are explicitly queried. The other options are ‘Other’ and ‘Non-denominational’. The settings therefore say:
Definitions to help the bot differentiate.
We also gave the bot the question hierarchy in writing. Likewise stylistic wishes, for example the rounding rules (7.46% or directly 7.5%?), the presentation of graphics or the tone. Incidentally, our first rule is:
“The GPT only displays the percentage values contained in the file. If a question is not covered by the data, the GPT politely points this out and explains that it may not make any statements outside the file.”
Better safe than sorry.
Testing
If you want a bot to work, you have to try it out again and again. And preferably by as many people as possible. Different ideas bring different answers and are the quickest way to discover common error patterns. The problem: the results of an AI are not repeatable. If colleague A gets the wrong answer to question 1, colleague B may answer the same question correctly. It is not so easy to distinguish the real errors from the random hallucinations of the AI.
Mistakes
Artificial intelligence makes mistakes, and so does our bot. The learning process will always continue, and the current data set will most likely not be the last processing we try out to reduce errors. But: every error teaches us something about how black box AI works. This is precisely why this project is so important to us.
We are happy to pass on the knowledge we have learned, here you can find our training courses in the field of artificial intelligence. If you have any questions or feedback, you can reach us at connect@auctority.net.
