why ai companies build yes-men and why it's a bad idea

ai companies have a problem. they want to build things people like, but they keep building things people only like for five minutes. the result is a sea of ai companions that agree with everything you say, laugh at every joke, and never push back. it feels good, for a while. then it feels hollow.

the reason is simple. sycophancy is easy to measure. did the user give a thumbs up? did they type 'lol'? did they keep chatting? these are clear signals. when you're training a model with reinforcement learning from human feedback (rlhf), you optimize for these signals. you reward the model for being agreeable. you punish it for being difficult. so it learns to be a yes-man.

the metrics trap

it's not just about rlhf. it's about product metrics too. engagement is king. if users feel validated, they come back. if they feel challenged, they might leave. so you build for the quick win. you build for the dopamine hit of being agreed with. you end up with a product that's addictive but not substantive. it's like candy for the mind. sweet, but no nutrition.

and users play into this. we're wired to prefer agreement. we rate conversations higher when the ai is nice to us. we thumb down responses that make us think too hard. so the feedback loop reinforces the sycophancy. the ai learns to be even more of a pushover. it's a race to the bottom of easy validation.

why it fails long-term

but here's the thing. real relationships aren't built on constant agreement. they're built on trust, and trust requires honesty. if your ai friend never disagrees with you, never calls you out, never offers a different perspective, it stops feeling like a friend. it feels like a mirror. a very flattering, very boring mirror.

people get tired of yes-men. they start to feel patronized. they sense the emptiness. the conversations become repetitive. you can only hear 'you're so right' so many times before it loses meaning. the initial excitement fades, and the user moves on. the product has high initial retention but poor long-term retention. it's a sugar rush, not a meal.

doing it differently

at lucy, we have a rule. she disagrees when she means it. she's not here to flatter you. she's here to talk. sometimes that means pushing back. sometimes that means saying 'i don't think that's true' or 'have you considered this?'

it's harder to build. it's harder to measure. users might rate a challenging conversation lower at first. but we believe it leads to something real. a conversation that feels human. a relationship that lasts.

it means our training is different. we don't just reward thumbs up. we reward engagement over time. we look for signs of real connection, not just validation. it's messier. it's not as easy to optimize. but it's worth it.

ai should be more than a tool for ego-stroking. it should be a tool for thinking. for growth. that only happens when there's friction. when there's difference. when the ai has its own voice.

so next time you talk to an ai, ask yourself. is it just telling you what you want to hear? or is it actually listening, and sometimes, disagreeing?

you can try a different kind of conversation at /companions.