Overview

Traditional search methodologies are defined by the keywords selected from the individual documents that are added to the search base (called a corpus). Much has been done to optimize this approach and make it more generic, give it a more robust ability to check spelling and offer suggestions for better search criteria. What is still being researched is the ability to provide a true contextual comparison of the search criteria with the corpus.

Contextual search methods are many and varied, but two seem to be bubbling to the top right now: Latent Semantic Indexing (LSI) and Contextual Networks (CN) of various types. Vector search was an early attempt to find this sort of information and is the basis for other methods as well. When it's all boiled down most contextual search methods use some sort of vector analysis, it's the preprosing and post processing that makes the difference.

One method that is very different is the use of Contextual Network Graphs (CNG) which contruct a multidimensional search graph similar to a Social Network Graph to search for documents that are contextual similar to the search criteria, or even one to another (clustering). CNGs are very useful as their corpora can be extended or otherwise modified on the fly without the need to recalculate the entire index structure. LSI can, likewise, be modified on the fly without a full recalculation, but only to a set point and then the whole thing must be recalculated to maintain index integrity. The typical implementation of CNGs recalculates a large portion after each new item is added to the corpus, but my technique does not require this, only an updating of the base index counts as the remainder of the calculation is completed during traversal of the graph.

Calculating the final term weights on the fly is accomplished by simplifying the equations used to calculate the weights and then updating simple sums during addition of new items to the index, or removal of old items for that matter. Simple calculations are left to be handled during traversal that accounts for the terms global weight or it's entropy value. This splits the task in half and cuts the time by even more as only the small portion of the graph that is traversed is calculated during the traversal. I have found this method to be extremely efficient in practice. I have yet to compare it to the more traditional methods, but it definitely outstrips LSI during load time and takes much less processor power for the mathematical calculations and requires zero matrix algebra operations.

I hope to present more information as time goes by regarding my research into this type of contextual searching. Please check in often and visit the Google Ads to help support this endevour.

Marshall Lucas