Implementing Incremental Index Updates in Sourceengine with IndexWriter and SearcherManager

Desired Outcome

Perform incremental additions, updates, and deletions to a Sourceengine index while keeping the search service available and seeing the latest data without rebuilding the entire index.

Prerequisites

• Java 8 or later JDK installed.
• Sourceengine (Lucene core) library version 8.x or higher available in the classpath.
• A writable filesystem directory to hold the index (e.g., /var/index/myapp).
• Basic understanding of Lucene concepts: Document, Field, Analyzer.

Procedure

1. Set up the index directory and analyzer

   Path indexDir = Paths.get("/var/index/myapp");
   Analyzer analyzer = new StandardAnalyzer();
   

2. Create or open an IndexWriter (thread‑safe, use a single instance for writes)

   IndexWriterConfig config = new IndexWriterConfig(analyzer);
   config.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
   try (IndexWriter writer = new IndexWriter(FSDirectory.open(indexDir), config)) {
       // writer is ready for changes
   }
   

   Note: The try‑with‑resources block ensures the writer is closed after the batch; for long‑running services keep the writer open and close it on shutdown.
3. Add a document

   Document doc = new Document();
   doc.add(new StringField("id", "doc1", Field.Store.YES));
   doc.add(new TextField("content", "Sourceengine incremental indexing example", Field.Store.YES));
   writer.addDocument(doc);
   

4. Commit the changes to make them durable and visible to readers

   writer.commit();
   

5. Initialize SearcherManager to serve fresh Searcher instances

   SearcherManager searcherManager = new SearcherManager(FSDirectory.open(indexDir), analyzer);
   

6. Obtain an up‑to‑date Searcher and run a query

   try (SearcherManager.SearcherAndReference ref = searcherManager.acquire()) {
       IndexSearcher searcher = ref.searcher;
       TermQuery query = new TermQuery(new Term("id", "doc1"));
       TopDocs hits = searcher.search(query, 10);
       System.out.println("Found " + hits.totalHits + " document(s)");
   }
   

7. Update or delete a document (example: delete by id)

   writer.deleteDocuments(new Term("id", "doc1"));
   writer.commit();
   

8. Refresh readers – SearcherManager automatically detects the commit; calling acquire again yields a refreshed Searcher.
9. Rollback uncommitted changes if an error occurs before commit

   writer.rollback(); // discards adds/deletes since last commit
   

10. Close resources on application shutdown

    searcherManager.close();
    writer.close(); // if still open
    

Expected Checks

• After each commit(), a new Searcher obtained via searcherManager.acquire() should reflect the added/deleted documents (verify hit count).
• If rollback() is called, a subsequent search should return the state as of the last successful commit.
• No file lock exceptions should appear when multiple threads use the same IndexWriter (it is thread‑safe).
• Monitor heap usage; large batches may cause OutOfMemoryError – mitigate by committing after a configurable number of documents (e.g., every 1000).

Recovery Options

• If a commit fails due to I/O error, the index remains at the last committed state; retry the commit after fixing the underlying issue.
• If SearcherManager fails to refresh (e.g., due to a corrupted index), recreate it from the directory; the existing index files are unaffected by SearcherManager state.
• In case of accidental data loss, restore the index directory from a backup and restart the writer.