[Rhodes22-list] Web Interface: How it works

[Peter Nyberg]

This post is about some of the inner workings of the web interface for the Rhodes 22 email list on rhodes22.net.

The Rhodes 22 email list is run by a program called 'Mailman'. Most of the data displayed by the email list web interface comes from Mailman archives.  But, to make the data easier and more efficient to retrieve and display, it is taken from the archives and placed into a relational database.

Initially, a bulk form of this process was run to convert many years of historical data.  After that, the process was run periodically to get recent updates.  Currently, the process is set to run every 5 minutes.

When the process finds a new message, it attempts to break it down into its component parts (e.g. sender, date, subject, main body, and quoted text from prior messages). The parsing of the message can be tricky because message formats are not entirely consistent.  

Every message viewed in the web interface will have a line at the bottom that reads ‘See the original archive post’.  Clicking on this should open up a new browser tab that will display the message in the Mailman archives.  This was useful to me during development as a way to check to see if a message had been garbled by the conversion process.  I’ve decided to leave it in the interface for now.

 It seems to universal behavior of mail client programs that when you reply to an email, a new window is opened that contains the content of the message that’s being replied to.  Usually, this is set off, or indented in some way to enhance readability. The message parsing process attempts to differentiate new text vs old text, and if it succeeds (which it generally does), it will store them separately in the database.  This is done mostly so that searches can be better targeted (more on that in a later post), but it also affects how messages are displayed in the web interface.

When a message is initially displayed, only new text will be shown.  If the full message also contains text from earlier messages, a ‘show more >>>’ link will appear at the bottom.  Clicking on this will, naturally, display the older text.

One final bit of behavior of the web interface that might seem odd has to do with sending new messages to the list.  When a message is sent, a dialog box will appear stating that the message should appear in the web interface in 5 to 10 minutes.  The delay is because even though in theory the web interface could put the message directly into the database and display it immediately, it doesn’t do that.  Instead, it waits for the Mailman archive retrieval process to find it and add it to the database.

So again, the new web interface can be found at https://www.rhodes22.net.  Don't be shy about letting me know if you find problems, issues or bugs.  

—Peter


[ Sent From rhodes22.net ]