Andrej Karpathy on X: "LLM Knowledge Bases Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest. In this way, a large fraction of my recent token throughput is going less into manipulating code, and more into manipulating" /

So: Data ingest: I index source documents (articles, papers, repos, datasets, images, etc. ) into a raw/ directory, then I use an LLM to incrementally "compile" a wiki, which is just a collection of. md files in a directory structure. The wiki includes summaries of all the data in raw/, backlinks, and then it categorizes data into concepts, writes articles for them, and links them all. To convert web articles into.

md files I like to use the Obsidian Web Clipper extension, and then I also use a hotkey to download all the related images to local so that my LLM can easily reference them. Important to note that the LLM writes and maintains all of the data of the wiki, I rarely touch it directly. TLDR: raw data from a given number of sources is collected, then compiled by an LLM into a. md wiki