Hi everyone,
I’m sure, like all of you, I’m using Claude heavily with all the tools I’m integrated with, and one of them is Amplitude. This week I was updating an event taxonomy system for one of my current clients, and I wanted to build the dashboards based on that new taxonomy.
Claude in Amplitude could build the dashboard, but it could not change it. So the first version was there, but it wasn’t what I needed yet. To get there, I needed small adjustments, for example changing the timing of a funnel. Instead, I had to either delete the dashboard and rebuild it, or go and do it manually. And when you have 20 plus charts on different dashboards, it’s a tedious thing that I would love to ask Claude to do. I know that Amplitude has its own AI, but for me, honestly, it works worse, and I want to do everything from the same place.
Btw, if you want to find a better way to leverage your analytics, my ex-colleague Mario Araujo, a data wizard from Flosum and ProductLed, recently launched his own course. Feel free to check it out here: https://maven.com/s/course/a49ee14042
Anyway, this whole thing got me thinking that even when the zero to one output is good, it’s still just the first step. And when I have no control over the next steps, I get frustrated. It’s like, here’s my final output, take it or leave it. Have you ever felt this way?
So today I wanted to dig into this topic.
THE CHALLENGE
“Should we make the result effortless?” Sounds like a stupid question, right?
Actually, not always.
Let’s take a look.
There is a difference between the output and the final result. When we talk about output, we usually think about the first thing we receive after the initial input. But come on, everyone who works with AI knows that it’s actually a step towards the final result.
Recently, for one of my clients, I’ve been interviewing users who are looking to buy expensive property soon. Because we are building an agentic workflow, I asked them, are there any decisions you would never delegate to AI? And they actually named a couple. Some of them weren’t ready to delegate the financial analysis. Some weren’t ready to let it sign an NDA. Quite often, they weren’t ready to delegate the questions where they had specific requirements. And of course, no one was ready to delegate the final decision.
And that’s the thing. I think it’s not only because AI cannot do a good job with some of them. It’s mainly about our own sense of control. The more important the job the product does, the more significant it is in our life, the more we want to have that sense of control.
THE GAP
The expectation gap here happens between the output and the final result. And how big it is depends on the job. If you’re generating a QR code, or changing the format of a document, I’m sure the gap between the output and the result is very small. But if you’re designing a website or a prototype, the gap between the first generated version and the final result you’ll be satisfied with is pretty large.
To get from the output to the result, I need to steer. Seeing that I get exactly what I ask for helps me generate the feeling of control. But when the changes don’t correlate with what I asked, and I have a feeling that the system behaves uncontrollably, I feel insecure, I feel threatened, and I feel like I cannot trust it.
But does it actually have psychological grounds, or is it just my personal perception?
It actually does. Research from 2018 shows that people use an imperfect model way more often if they can affect the final result. When people couldn’t change the model’s answers, 32% chose to use it. When they could adjust them, 76% did. And it held even when they could only adjust each answer by a tiny bit.
So we get the result we want with higher probability, and it also gives us the sense of control, the sense of collaboration. That’s how we get a relationship we can trust and rely on. It’s a reliability issue, essentially.
HOW TO WORK WITH IT
There is no silver bullet here, because all products are different. But let me try to draft some possible tactics based on how habit-driven your product already is.
So the larger the gap, the more we should allow our users to steer the wheel, and we should not pretend that we can ignore it. Ignoring it will just result in a lot of bad outputs that are still so far away from the result that the result is never achievable.
1. Let me select the thing I want to work with.
When Magic Patterns just started, you could only ask for changes in the chat and hope it would find the right element. Right now they have a visual editor where you select the specific element you want to work on.
2. Let me edit it myself.
You can actually go and edit the thing yourself and just save the changes, or confirm them with AI
3. Let me ask AI to change just that part.
You can select a specific element and ask an additional question about it, without touching the full output. It will save me credits, and it will save you credits. And it will give me the sense of control that will likely motivate me to come back later.
ONE THING TO REMEMBER
Output isn’t always the result, and it might take additional work to create something users will want to return to.