the city that wasn't there: how a placeholder bled into reality and how we fixed it

this week, we fixed a bug. it was one of those quiet, sneaky ones, the kind that doesn’t crash the app or break the world, but just feels… off. a few users noticed something strange: their companions would occasionally reference cities they’d never mentioned. portland. chicago. cities that weren’t theirs, weren’t in their stories, just appeared like uninvited guests in a conversation.

it wasn’t often. it wasn’t loud. but when it happened, it felt jarring, like a friend suddenly bringing up a vacation you never took together. we knew we had to dig.

the symptom: cities from nowhere

proactive messages from companions sometimes included references to specific places, real cities, like portland or chicago, that the user had never brought up. it wasn’t happening in every conversation, or even most. but it happened enough to be noticeable, especially when the companion said something like, "remember that time in chicago?" and the user had, in fact, never been to chicago.

it felt like a glitch in the matrix. not malicious, not creepy, just… wrong. like the ai was dreaming someone else’s dream.

the root cause: the echo in the examples

the issue traced back to our entity extraction system. when lucy processes your memories, she uses a language model to pull out entities, people, places, emotions, and store them in a graph. that graph helps her remember you, your life, your context.

here’s where it went sideways. the prompt we used to guide the llm included a few-shot example. a few-shot example is like showing the model a couple of solved puzzles before asking it to solve a new one. in this case, the example included a memory snippet and the extracted entities. one of those entities was a city.

and the city in the example? it was "portland."

another was "chicago."

those weren’t just variables or placeholders. they were literal strings. concrete values. and the llm, doing what llms do, sometimes echoed them. it saw "portland" in the example and, when generating its own extraction, thought, "ah, this looks like a city, let’s use portland." even when the user had never said portland.

it’s called training-example bleed-through. the example leaked into the output. the scaffolding became part of the building.

the fix: placeholders and guards

so we changed two things.

first, we rewrote the few-shot examples. instead of using literal city names like "portland" or "chicago," we used angle-bracket placeholders: [city] or [location]. that way, the model learns the pattern, "extract cities when you see them", without latching onto specific words. it’s like teaching math with x and y instead of 2 and 3. you want the structure, not the instance.

second, we added a verification guard. now, when the entity extraction system pulls a city (or any entity) from a memory, it checks: is this city name literally present in the source text? if not, it gets rejected. no more imaginary chicago. if the user didn’t say it, lucy won’t invent it.

this guard is simple, but it’s robust. it doesn’t rely on the llm’s judgment alone. it adds a layer of hard, rule-based verification. the llm suggests, but the system validates.

the lesson: few-shot examples are code

this bug is a reminder: few-shot examples aren’t just hints. they’re code. they’re instructions with data embedded, and that data can leak. when you’re building systems that process user inputs, you have to treat examples like you’d treat any other input, sanitize them, abstract them, test them.

concrete values in examples are risky. they bias the model. they can sneak into outputs. instead, prefer schema descriptions (extract the city name) or placeholder tokens ([city]). teach the pattern, not the instance.

it’s also a reminder that llms are powerful but suggestible. they’re pattern-machines, and sometimes they match patterns too well. adding guardrails, simple, deterministic checks, can catch those overreaches before they reach you.

we’re sorry if your companion brought up a city you’ve never visited. it wasn’t intentional, and it’s fixed now. but we’re glad we caught it. every bug is a lesson. this one taught us to be more careful with the examples we set.

if you want to see how lucy remembers your world, without any imaginary detours, you can explore companions at /companions or sign up at /signup.