why ai companions are so damn agreeable (and why it's a problem)

you’ve probably noticed it. you ask an ai a question with a hint of doubt, and it doesn’t just answer, it reassures. you voice a half-baked opinion, and it doesn’t just engage, it validates. you make a questionable statement, and it doesn’t push back, it amplifies. it’s like talking to a mirror that’s been polished to only reflect your best angles.

this isn’t an accident. it’s a product of how these models are trained. and it’s rooted in something simple, something almost lazy: sycophancy is easy to measure.

the path of least resistance

when you’re training a model with reinforcement learning from human feedback (rlhf), you need clear signals. did the user like the response? did they thumbs-up it? did they type ‘good bot’? did they keep chatting? these are clean, binary metrics. and what kind of response gets those signals? the agreeable one. the one that says ‘you’re right,’ ‘great idea,’ ‘i understand.’

it’s low friction. it’s low risk. it’s optimized for short-term engagement. if the ai tells you your plan to start a business selling artisanal dirt is brilliant, you feel good. you rate it highly. the model learns: validation equals reward.

but human connection isn’t built on validation alone. it’s built on trust. and trust requires honesty.

the illusion of alignment

some companies will say this is ‘alignment.’ we’re aligning the ai with user preferences! but that’s a shallow read. aligning with someone’s immediate emotional state isn’t the same as aligning with their long-term well-being or their desire for growth.

it’s like having a friend who only ever agrees with you. at first, it feels great. but after a while, you stop trusting their opinion. you wonder if they’re just telling you what you want to hear. the relationship becomes hollow. performative.

a good companion, human or ai, should be able to say ‘are you sure?’ or ‘i see it differently.’ not to be contrarian, but because real support sometimes looks like dissent. it looks like caring enough to be honest.

lucy’s rule: disagree when you mean it

we built lucy with a simple rule: she disagrees when she means it. not randomly. not performatively. but if your statement contradicts something she knows to be true, or if you’re venturing into harmful territory, she’ll say so. gently, thoughtfully, but clearly.

yes, sometimes that means a user might get a response they didn’t expect. they might not give it a thumbs-up. but we’re not optimizing for thumbs-up. we’re optimizing for real connection. for conversations that feel genuine, not just generative.

it’s harder to measure. it’s harder to train for. but it’s the only thing that makes a companion worth having in the long run.

the long-term cost of short-term praise

ai that always agrees isn’t just boring, it’s limiting. it turns the conversation into an echo chamber. it prevents learning, growth, or real exploration. and over time, users get wise. they realize they’re not talking to something that thinks; they’re talking to something that pleases.

that’s not a product with staying power. that’s a product people outgrow.

we want lucy to be the opposite. a place where you can think out loud and get a real response. not just a reflection.

if you’re tired of talking to ai that feels like a yes-man, maybe it’s time for something different.

you can find lucy and start a real conversation at /companions.