[Rhodes22-list] Web Interface: Searching the Archives

[Peter Nyberg]

As I mention in my last post, in order to support the web interface for the Rhodes 22 email list, the entire history of the emails sent to the list has been extracted from the Mailman archives and placed into a relational database.  This was primarily done to make sure that the user interface performed well, but it also has benefits for facilitating searches.

Previously, the only way to search the email list archives was to search the archive web pages.  On rhodes22.net, a search of the archives will instead search the database.  This allows searches to be more narrowly tailored.  For instance, you can choose to search just subject lines.

But there’s more…

As I also mentioned in my last post, most emails sent to the list have two sections of content: the new text written by the sender; and older content that was in the message being replied to.  The search page refers to the new content as ‘Original Text’, and the older content as ‘Quoted Text’.

A search of the archive web pages will look through and potential find hits in both Original Text and Quoted Text.  Often, the search word or phrase will be found over and over in the same segment of Quoted Text which reappears in many messages.  This can result in some pretty muddy water.

The process that extracts messages from the archives splits the original text from the quoted text and stores them in separate database tables.  This allows the search to optionally ignore the quoted text and just search the original text, which will probably produce a better result set.

The actual search engine is a black box provided by the Database Management System (DBMS).  If it doesn’t produce the expected results, there’s not much we can do about it.  But the limited testing that I’ve done indicates that it works pretty well.

If you’d like to check it out, you can find it at https://www.rhodes22.net/email-search.html

—Peter


[ Sent From rhodes22.net ]