addedValues Plugin

Importing Data: Performance

A note on performance.

Every record that is entered is going to be screened to ensure it complies with the variable you have defined, it will then be registered, then indexed variables will be indexed and sorted. These are not trivial processor operations. There is also the matter of Frontier/Manila memory management.

Thus, the following guidelines will save you pulling out some hair:

• import in chunks of 1000 records of 20 variables - 10 of which are indexed - maximum
• this should take between 3 to 7 seconds per record, or 30 to 45 minutes to process, depending on the machine and many other extraneous processes
• your system administrator should save a back-up copy of the root in which these are being loaded after each process. Restarting Frontier will also release memory and give your next load a speed boost.
• ideally, you want to conduct the import an a computer that is not being sollicited for other services; those web pages will be slow in getting served & things can get bogged down...

Need we repeat ourselves... Test your imports. Your source data may be very reliable. Or it may not. addedValues is practical in the sense that it screens each record; if that value does not fit, the record is not loaded. So you can go back, edit your original file and re-import only those records that require cleaning up.