search: map the index from a file instead of building it at each start

Every start read the whole corpus, 352 MB on Tim's Fastmail, and indexed it again in memory: about 3 s, 248 MB resident, and a peak of 749 MB while the file and the index were both held. Write the index as a segment file beside the corpus, laid out as memory holds it: the messages, sorted thread and message ids, sorted words with their postings, and the strings they name. A start maps it, so the operating system keeps only the pages searches touch, and indexes in memory just the corpus records past the segment's end, its later messages and the messages gone since. Words sorted make a prefix a binary search instead of a scan of every word. Once 5,000 messages are indexed in memory, or with no segment, the load writes it again from the whole corpus, read a buffer at a time rather than whole. A corpus started over removes its segment. On 92,000 synthetic messages of 500 words, the segment is 173 MB, written in about 1 s and mapped in 1 ms; searches took under 10 ms.


Author rockorager.dev
Date Commit Parent 602536a6
+707 -62
5 changed files
+707 -62
5 changed files