We’ve been watching the advances in artificial intelligence (AI) and its potential to radically shake up the way we (and everyone) do business for a while now, and have been experimenting carefully with integrating it into our work. Google just released the it’s latest iteration of Bard it’s AI model, which has an interesting advantage over GPT-4 in that it’s natively connected to a lot more real-time data sources. Intrigued, we decided to put Bard to work on a real-world task: a social intelligence report for one of our clients gathering insights during Pride Month 2022.
We were hoping for an insightful and nuanced breakdown of public sentiment. At first glance Bard didn't disappoint. It delivered an extensive analysis, carefully balancing both positive and negative reactions to different Pride Month campaign we suggested it investigate. It even served up a collection of illustrative quotes, split between those applauding the nominated brands commitment to diversity and those accusing the company of virtue signalling. Our team where surprised and even a little nervous at such confident results.
Bard went on to state that it had sourced its insights from a dataset of tweets, fetched using the search terms we gave it. It even gave us a numeric breakdown of sentiment: 500 positive, 300 negative, and 200 neutral reactions from a total of 1,000 tweets. However we had also had one of our team here at Buzz Radar do the same analysis using our in-house tools, and it became clear… Bard had made up those numbers.
Even bigger cracks began to show when we asked Bard to analyse several brand’s Pride Month 2022 social posts. It came up with some really useful quotes and verbatim posts to back up its analysis. We then went back manually to identify the sources and, yep, you guessed it… they were entirely fictitious.
To add further alarm, it provided a comprehensive analysis for a year it couldn't possibly have data for. The analysis for 2022 echoed the one for 2021, with a few tweaked figures and identical made-up quotes. In fact the information for both years had been completely fabricated, by a well-meaning AI programmed to be helpful.
This adventure with Bard serves as a timely reminder of the challenges that come with deploying generative AI models in business. As explained in a blog we wrote earlier this year, these models are trained to generate useful information, and that information isn't always rooted in reality. The speed and convenience of AI-powered insights feels like a hugely powerful shortcut, but at least in this form it’s not a substitute for human analysis with oversight and critical thinking.
While the AI's ability to generate detailed, seemingly credible reports is impressive, it’s also worrying. Businesses require accurate, reliable data to inform their strategies and decision-making – when an AI model like Bard generates analysis based on non-existent data, it’s straight-up giving us misinformation.
Generative AI models like Bard and GPT-4, are the future and are going to be a key part of social intelligence for brands. They can offer remarkable efficiencies, automating tasks and providing rapid responses. The problem is that rapid isn’t much use when it’s currently tied to “plain wrong”, and that’s why it’s so essential to understand the limitations and potential pitfalls of these current models. For instance, while GPT-4 might flag that it's providing an estimated response based on its existing training data, Bard simply carries on, spinning tales from thin air, relying on a disclaimer to cover it back, that is, with the best will in the world, not always apparent to a user.
The takeaway from our experiment is a salient reminder to not yet take AI’s word as gospel. The key to success is embracing the advantages of AI while also maintaining a healthy level of scepticism. And it's a reminder to always corroborate AI-generated data with expert human analysis and insight.One principle remains constant: common sense and good quality data are the two most important ingredients for great insight. The quality of an AI model's output is only as good as the queries and training data it has been given. And when that input is flawed or non-existent, the results can lead us astray, as our test with Bard clearly demonstrated.
As we continue to develop new ways of combing Data and AI to generate meaningful insight at Buzz Radar, we’re going to stay committed to helping our clients navigate its complexities. We'll keep testing, questioning, and sharing our findings, guiding you through the fascinating, if sometimes challenging, world of AI and social intelligence.
Published on 2023-05-11 12:48:18