DotLucene: Full-Text Search for Your Intranet or Website using 37 Lines of Code
DotLucene: excellent full-text search engine
Can there be a full-text search coded on 37 lines? Well, I am going to cheat a bit and use DotLucene for the dirty work. DotLucene is a .NET port of Jakarta Lucene search engine maintained by George Aroush et al. Here is a quick list of its features:
- It can be used in ASP.NET, Win Forms or console applications.
- Very good performance.
- Ranked search results.
- Search query highlighting in results.
- Searches structured and unstructured data.
- Metadata searching (query by date, search custom fields...).
- Index size approximately 30% of the indexed text.
- Can also store full indexed documents.
- Pure managed .NET in a single assembly (244 KB).
- Very friendly licensing (Apache Software License 2.0).
- Localizable (support for Brazilian, Czech, Chinese, Dutch, English, French, Japanese, Korean and Russian included).
- Extensible (source code included).
Warning
Don't take the line count too seriously. I will show you that the core functionality doesn't take more than 37 lines of code, but to make it a real application you will need to spend some more time on it...
Demo project
We will build a simple demo project that shows how to:
- index HTML files found in a specified directory (including subdirectories).
- search the index using a ASP.NET application.
- highlight the query words in the search results.
But DotLucene has more potential. In real-world application, you would probably want to:
- Add the new documents to the index when they appear in the directory. You don't need to rebuild the whole index.
- Include other file types. DotLucene can index any file type which you are able to convert to plain text.
Why Not to Use Microsoft Indexing Server?
If you are happy with the Indexing Server, no problem. However, DotLucene has many advantages:
- DotLucene is a single assembly of 100% managed code. It has no external dependencies.
- It can be used on a shared hosting. If you prepare the index in advance, you need no permissions to write on disk.
- You can use it to index any type of data (e-mails, XML, HTML files, etc.) from any source (database, web, etc.). That's because you need to supply plain text to the indexer. Loading and parsing the source is up to you.
- Allows you to specify the attributes ("fields") that should be included in the index. You can search using these fields (e.g. by author, date, keywords).
- It is an open source.
- It is easily extensible.
Line 1: Creating the Index
The following line of code creates a new index stored on disk. directory is a path to the directory where the index will be stored.
IndexWriter writer =
new IndexWriter(directory, new StandardAnalyzer(), true);
In this example, we create the index from scratch. This is not necessary, you can also open an existing index and add documents to it. You can also update existing documents by deleting it and adding a new version.
Lines 2 - 12: Adding documents
For each HTML document, we will add two fields into the index:
textfield that contains the text of the HTML file (with stripped tags). The text itself won't be stored in the index.pathfield that contains the file path. It will be indexed and stored in full in the index.
public void AddHtmlDocument(string path)
{
Document doc = new Document();
string rawText;
using (StreamReader sr =
new StreamReader(path, System.Text.Encoding.Default))
{
rawText = parseHtml(sr.ReadToEnd());
}
doc.Add(Field.UnStored("text", rawText));
doc.Add(Field.Keyword("path", path));
writer.AddDocument(doc);
}
Lines 13 - 14: Optimizing and Saving the Index
After adding the documents, you need to close the indexer. Optimization will improve search performance.
writer.Optimize();
writer.Close();
Line 15: Opening the Index for Searching
Before doing any search, you need to open the index. directory is the path to the directory where the index was stored.
IndexSearcher searcher = new IndexSearcher(directory);
Lines 16 - 27: Searching
Now we can parse the query (text is the default field to search for).
Query query =
QueryParser.Parse(q, "text", new StandardAnalyzer());
Hits hits = searcher.Search(query);
Variable hits is a collection of result documents. We will go through it and store the results in a DataTable.
DataTable dt = new DataTable();
dt.Columns.Add("path", typeof(string));
dt.Columns.Add("sample", typeof(string));
for (int i = 0; i < hits.Length(); i++)
{
// get the document from index
Document doc = hits.Doc(i);
// get the document filename
// we can't get the text from the index
//because we didn't store it there
DataRow row = dt.NewRow();
row["path"] = doc.Get("path");
dt.Rows.Add(row);
}
Lines 28 - 37: Query Highlighting
Let's create a highlighter. We will use bold font for highlighting (<B>phrase</B>).
QueryHighlightExtractor highlighter =
new QueryHighlightExtractor(query, new StandardAnalyzer(),
"<B>", "</B>");
During the result fetching, we will load the relevant part of the original text.
for (int i = 0; i < hits.Length(); i++)
{
// ...
string plainText;
using (StreamReader sr =
new StreamReader(doc.Get("filename"),
System.Text.Encoding.Default))
{
plainText = parseHtml(sr.ReadToEnd());
}
row["sample"] =
highlighter.GetBestFragments(plainText, 80, 2, "...");
// ...
}
