WE TOOK ON A.I. AND WE LOST... TECHNICALLY.
By Donagh Humphreys
Many years ago, we gave birth to ‘F*ck It Friday’. FIF is an experiment in creative freedom with no defined deliverable output. Once a month, a cross-section of talent from the agency gets stuck in and throws caution to the wind in a bid to do new, bold, free-form experimental work. We’ve been doing this since forever. In fact, our establishment of a reflection-agency in Second Life back in 2007 was one of the first big experiments. Since then we’ve experimented with media mixes, new film and photography ideas, paper cutting… and once someone wanted to pour paint over things in front of mirrors, (It created a beautiful aesthetic in fairness) and one of our other creatives decided to even create custom filming equipment. In the last few years, we’ve f*cked around with 3D modelling to building games in the Metaverse. At the peak of the hype cycle of AI we felt it was high time we stopped messing about with the various AI tools and instead, got competitive.
For our newest Fuck It Friday… We took on the machines.
OVERVIEW
The task was simple. An experiment where one set of Thinkers (the informal demonym for someone who works at Thinkhouse) had to ideate and create social media assets that would be used as the basis for a recruitment campaign on our social media channels using only AI tools. The opposing team would be allowed to use traditional methods and creative tools only.
To make the task fair and balanced, the two teams were given an identical timeframe in which to complete the task – a single Friday. Initially the brief was for a single piece of short form video content (used to drive awareness), supported by stills assets that would be to drive clicks to the website via carousels. However, due to technical issues that was reduced to just carousel images to be used for ‘click to website’ assets.
THE TRAD APPROACH
Team A - dubbed the "traditional team" - comprised members from client service, copywriting, social and digital innovation, PR and the creative technology teams. The process was quick and simple. A forty-five-minute brainstorm a couple of days in advance where the idea was conceived and thought starters for accompanying copy discussed.
The idea was somewhat of a spark of genius from one team member who suggested the old claw arcade game as the creative concept. Layered on top of this were beautiful and subtle nods to life at Thinkhouse (kind of like an in-joke) with the inclusion of the “lady on the rock statue” which we have in the windows of our office. Also accompanying this was the short and sweet copy line that had a double meaning – “Have you got the skills?”
THE AI APPROACH
The AI team, again comprised of the same disciplines, had a different process. In order to embrace as much AI technologies as possible the team were not limited as to how many AI tools they could use, but were mandated to only use AI tools to ideate, strategize and ultimately create their content. Using ChatGPT for ideas and scripting purposes the team then used Midjourney and Stable Diffusion to create their content.
The team began by building out their plan through the generative text tool ChatGPT. Resulting in an hour-long conversation between the team and ChatGPT to build a strategy, with each segment of the video laid out clearly for the image or video based AI machines to follow.
From there to maintain Thinkhouse look and feel, we fed the tools with as much base Thinkhouse content as we could give it. And used prompting to ensure the TH colourways were included, and negative prompting (including concepts to exclude) to tweak the AI whenever it was continuously generating its own concepts rather than following the specific brief created. This was the biggest challenge in the process, as prompting more general ideas gave the machine more room to emulate Thinkhouse’s look and feel, but as the ChatGPT brief was so specific it struggled to nail both parts of the assignment.
What emerged was original, intriguing but overall, a rather peculiar creative piece:
PLAN B
Our experiment took a turn upon the reveal of both sets of creative. Pleased with the traditional content we felt that we had a new, creatively arresting, and original piece of content that was unlike anything we had done before but still very true to our look and feel and ultimately felt Thinkhouse-y. By contrast the creative that the AI team produced was interesting but left many colleagues feeling a sense of unease around posting it to our owned social media channels.
Thankfully a happy compromise was made whereby we utilised AI developed content originally on the cutting room floor but that we felt was more representative of the agency.
COMPARISON OF RESULTS
With a centralised budget the two final pieces of creative were set up as carousel ads and run off in an AB test against one another. The ad was given enough time to learn and optimise towards whichever creative was driving better results.
Social media ads by their nature allow for a multitude of measurements, from reach to engagement, and clicks to dwell time. The criteria we chose was cost per link click; The average cost per click on links within the ad that led to advertiser-specified destinations, on or off Meta technologies ie. The applications page on our website. For a campaign that was set up to drive recruitment – this was the natural fit.
Cocky or as disparaging as teams might have been at the beginning of the campaign the results were a turn up for the books and could not have been closer:
The winner by a margin of ¢0.7 – that is less than one cent per link click. So technically the AI creative won, however our traditional overall drove 2% more link clicks and ran at a lower CPM €2.75 vs €2.86.
Hence the rather click-baity title of this piece - we took on AI and we lost… technically.
In reality their difference in performance was close to negligible. The real takeaway is the tonnes of learnings around the benefits and drawbacks of AI creative, limitations and ethical concerns on all things AI that you can read here now.