By Cameron Davies, Head of Comms Planning, Miroma Founders Network
Around this time last year, I posted an article on LinkedIn about the tension between AI efficiency and the slow, unglamorous grunt work that teaches us the building blocks of good marketing.
My concern was that we were automating away the work that taught marketers how to think, without finding intentional and effective ways to replace the learning it once provided. It’s a cost that won’t be felt for over a decade, by which point those affected will be running our agencies with a gap in their foundational knowledge.
A year on, industry discussion has moved to the AI skill gap, agencies redesigning junior roles around AI-integrated workflows, and, thankfully, the risk of a generation highly proficient with AI but short on marketing foundations and sound critical thinking. So, as an industry, we’re aligned. But inside the consensus that we just need to learn the tools properly and build the judgement to know when they’re wrong, sit some assumptions.
The first: training on a tool can, in certain scenarios, lead to poorer performance. A Harvard field experiment with BCG and 758 of their consultants illustrates this best. Published in Organization Science this year, they ran three groups: no AI, GPT-4, and GPT-4 plus proper prompt training, on a task outside the model’s scope. The group with no AI got it right 84% of the time, while the group trained to use the tool best only managed 60%.
Competence with the tool isn’t competence at the task. Fluency just buys the confidence to accept whatever comes back.
As a seasoned marketer, with years of experience behind you, that’s manageable, and low-risk. A few months or even years in, it isn’t. Worse still, the same study found that with AI, a wrong answer has a stronger chance of going unnoticed, meaning the valuable feedback needed to learn never arrives.
Furthermore, most fixes currently on the table ask something of the human: be curious, think before you prompt, and evaluate the output critically. But in doing so, they seem to assume a homogeneous tool. Yet the design of the tool, alongside the squishy (soon to be extinct) bit pressing the buttons, is a significant variable, especially when the objective is competence at the task.
Two recent studies on education make the point better than I can. The first, published in the Proceedings of the National Academy of Sciences (PNAS), saw Bastani and colleagues give a thousand students either a standard LLM, the same model set to give hints instead of answers, or nothing. On a later exam with no access, the chatbot group scored 17% worse than those who’d never had one. The hint-only group came out level.
Meanwhile, a second study by Harvard in Scientific Reports showed that an AI tutor built to release one step at a time, and make students attempt the problem before offering anything, produced more than twice the learning of an active-learning classroom, in less time.
Same technology, opposite outcomes, and the key variable was tool configuration.
So what does this mean for us, and for the marketers we’re training?
- Set the tools up to make them go first. Questions, prompts, one step at a time, their read on the audience before the model’s, their hypothesis on the solution before it offers twelve. Almost everything we’ve rolled out is built to produce a finished thing as fast as possible, which is the one design decision the research says does the damage. Attempting the solve first is what makes the answer stick and ensures it’s right, not just sounds right.
- Build the debate in early. Use AI to challenge the thinking at key stages, not just to produce it, so the first time an idea gets tested isn’t in front of the client. It isn’t marking the work, it’s surfacing important questions before someone else does, testing the idea from every angle rather than letting the first one go unchallenged. That’s what lets a junior properly own what they present, because they’ve already had to defend it.
- Taper the kind of support, not the amount. Guidance that helps a novice starts getting in an expert’s way. Juniors need AI configured to scaffold, and seniors need it configured to produce, because only the seniors can reliably tell when what it’s produced is wrong. We’ve built it backwards, handing finished-answer tools to people least equipped to audit them, because junior tasks are the automatable ones.
Tool knowledge and task knowledge are both critical, and we’re only building one of them. The juniors we’re redesigning these roles around will be in the boardroom in a decade. I’d like them to arrive able to own and direct the output, rather than pass it along.
