addedValues Plugin

...Powerful Free! Database Expansion for Manila

(1 or more words)

Get tropes here!
Click to see internals
Report bug


Viewable with Any Browser

Join Now

Metadata Architecture Rethink


The Metadata plugin is probably the most widely used Manila plugin. Recently a new version added more sophisticated indexing, metadata for gems and shortcuts and more which will probably further increase its popularity. However,in my opinion, the architecture has reached it limits, it has become very hard to maintain, the memory usage has increased markedly, search speed has decreased for some common search kinds. The user interface could also do with a revamp, it becomes onerous to go to the plugin home-page if there are many variables defined. I believe it's time for a rethink. This document is a draft of my thoughts to date.

Goals of a rewrite

  1. Speed up index handling. Make the distinction between data and indexes more clear.
  2. Improve the UI - proper navigation menu rather than links at bottom of home page, get rid of huge table on home page.
  3. shift processing from filters to new/changed message callbacks to support Versioning.
  4. Increase maintainability of code. Get rid of gridbox logic. Get rid of pseudo message numbers.
  5. Make rpchandlers and scripting API more central - Make logic more independent of page table. Prefer adrSite over pta as much as possible

Index Handling

The metadata 4 index was a kludge. ASE was introduced as an alternative, but ASE resembles an object oriented implementation of serious database indexing. Most people do not use metadata to index data sets large enough to warrant this level of overhead. And overhead there is, all my metadata 4 queries became slower on metadata version 5.

The way I propose to do this is to use the strengths of Frontier and Usertalkspecifically assignable script objects and the ability to create and compile scripts from within scripts. The intent is to move most logic in performing the really basic tasks such as "get a metadata value" and "put a metadata value" into tailored scripts that are generated at definition time. There should be no comparison logic at run time. Indexing is handled that way too. The tables are designed to optimise searches and the algorithms are encapsulated into tailored scripts within a methods table that belongs to the meta variable.

The choice of msgnum for key brings few advntages. When a search is done the object of interest is the list of messages, gems or shortcuts addresses, not message numbers or worse pseudo message numbers. So the indexes are organized to use address - actually string.popFilefromAddress values. msgnums are available as a property of the hits in the same way other builtins are.

Regarding builtins, they should be

Note that if goal 4 is achieved, metadata API forms a nice basis for writing sensible control scripts, perhaps as second order plugins.

This is easily the most important part of my proposed architecture. The utility of the plugin is based on its ability to perform queries quickly and then second to retrieve values from DG messages and other indexed entities. if it cannot do that efficiently the rewrite is a waste of time. So this is where most of the thought has gone. Details on the page Metadata rewrite - Index Routines.

UI changes

If I want to check an index in metadata and I have a lot of variables defined, its an long wait to get to the link I need at the bottom of the page. We need a state of the art UI that uses menus, techniques to keep screen redraw down, and a better organization to reflect real world workflows.

I have noticed that naive users have no idea that there is a cost to indexing, so any new UI should make the cost clearer and influence the choices by choice of defaults and "do you really want to do that?" dialogs.

I have played a bit with this and come up with a design that addresses some of these issues and allows selecting multiple variables for comparison and parallel editing (click to view full size images)

addedValues Schema 1: addedValues Schema detail:

In the detailed table, I have tried to divide the settings into categories that present questions to a naive user as they come to define a variable, there is work to be done here, it's not meant to be final.

Use of callbacks

Versioning is an incredibly powerful technique, it allows backing out of changes made in error. The simple version I have implemented to date is already useful in practice, but it audits messages before images and metadata updates too late.


Most contentious of my goals I suspect. But the proof is how hard it was to get metadata 5 debugged. I suspect I will have to prove this by doing. Anything else is just hot air.

RPC Handlers and Scripting API

This may look like an odd thing to put up there with the other goals, which no matter how contentious, are clearly good things. But in fact I have come to believe that it is central. I could have used metadata many times from other contexts, as a central way to handle adding values to messages, if only I didn't have to pass or dummy up a page table and fight the architecture. I don't want to use RPC because that involves all the overhead of going thru builtins.betty and providing a username etc.

I see a core layer of working scripts with 3 groups of interfaces - local script API, RPChandlers and a web interface.