Your Photos app is a gateway into your life.
Things you care about, people you spend time with, foods you enjoy, changes in your life over the years, places you've been to.
The apps do an alright job of conveying its value by sharing memories every now and then, but I always wondered how much more value lay in your photos and videos.
To my delight - earlier this year, Google released Gemini Embedding 2, a model that lets you embed text, photos, audio, and videos into one shared space.
I spent the next couple of days gathering over 30,000 of my photos, videos, notes, and music over the last decade.
Embedded them, built a Google over it. I called it Trove.
I could enter keywords like you do in Google, and it would return the closest matches across text, photos, and videos.
For example - if I typed New York, it would return my writings about my time in NYC, photos of Manhattan Bridge and Statue of Liberty, videos in Central Park, and Jay-Z's “Empire State of Mind” song among other results.
If I searched for Indian food, it would show everything most relevant to Indian food. But that's not it - I could open any photo, crop just a part of it, like the naan, and search with that. It would return the photos and videos closest to that crop.
It was very impressive, but it still required that I ask questions and look at results to see the story it tells.
So I wanted to take it one step further - if I made it queryable for AIs, what would they ask, what story would my data tell?
So I gave Codex a goal to make it queryable for AIs like Codex.
Then I gave Codex another simple goal, asking it to learn about me.
This unlocked something profound.
It thought for a while, decided what questions were worth asking, then turned them into queries, learned from the results, and did a few rounds of this.
It was able to tell
- How my friend groups changed over the years
- What kinds of foods I enjoy
- My interests (through screenshots)
- Places I've lived in, visited
And a lot more!
