I made a web app for exploring my entire @readwise library in semantic space (with the help of Claude)
First we embed every paper from the library into high dimensional space and find the top principal components within it. We label those axes by sending sampled points from the poles to an LLM to figure out what they mean.
Then we feed each paper into an LLM again, asking for short summaries of its position on each of those labeled axes, and why. Creating standardized “cards” of the same length and content for each one.
Then we embed those cards, and project those high dimensional embeddings into a low dimensional space, So we can make a scatter plot. Now we can also reorganize by any of the features we’ve found to make different 2 x2s, and find citations between all papers from arXiv, and even pull the frontier of “ghost papers” that are cited by my reads but not in my library yet.
It’s all reusable and costs pennies. I can package it up if people are interested, but thought you might like to at least see. Note that it’s not as easy as just asking Opus to do this, it kept wanting to hardwire an ontology based on a few data points it read. I had to steer a lot and know what I wanted, and a good deal about the domain to keep it from doing some ridiculous hacky shit. But it’s so much better than me at dataviz in JS that we make a good team