Search Images with Words

The Spark

You look at a dog and you just know it's a dog. You don't think about it. Light hits your retina, something fires in your brain, and the word is just there. That's insane if you actually stop and think about it. Pixels became a word. How? Does the brain think in pictures first and then assign language to them? Or does language shape what we see? There's no clean answer, but the question keeps pulling me in.

And then there's the machine side. When a VLM processes an image and a sentence together, what's the shared representation that both modalities are actually living in? A vision encoder produces one kind of feature. A language model produces another. Somewhere these have to meet in a space that's useful for both. How does that shared space form? Is it learned end-to-end or forced through a projection? And once you have it, can you fine-tune it for a specific domain and actually trust the output? What breaks, and how do people deal with that in production?

I'm comfortable on the vision side. Detection, segmentation, tracking, that stuff makes sense to me. But I've never properly understood the moment where vision meets language. And now I'm stuck on a more practical question too: if I actually want to solve a vision-language problem, what do I reach for? Do I need a VLM? Do I need a segmentation model like SAM? A foundation model like DINO? Some combination? How do I even decide?

I opened my phone, typed "birthday cake" into Photos, and it found every birthday cake I've ever photographed. I never tagged any of them. My phone did this offline. Google does the same thing but needs a data centre. Same problem, completely different engineering. I want to understand all of it. The neuroscience, the models, the engineering, the tradeoffs. And I want to build something with it.