This post was extensively updated and incorporated into my new book, The Data Conversion Cycle: A guide to migrating transactions and other records for system implementation teams. Now available on Amazon.com in both Kindle format for $4.49 and paperback for $6.99.
I just watched an interview with Kate Parson, a Senior Director at EMC. She was talking about Project Propel, which replaced their two decade-old ERP solution with a nice, shiny new SAP solution. The part that really caught my attention was her statement that, early in 2012, they had to start deleting tables in their database, because they ran into “an integer limit.” They had accumulated so many records that even Oracle couldn’t handle them. Yes, you read that correctly: EMC, provider of massive storage solutions, couldn’t handle the sheer volume of data records they had accumulated.
I make my living moving customers off of old HR, payroll, and benefits administration solutions and onto a nice, shiny new one in the Cloud. Naturally, a big chunk of every project is moving records from the old solution to the new one. We always recommend that customers only move “current” records, rather than attempt to load history. While you need to retain history records for some period of time, they don’t need to be kept commingled with current records, in the system of record. They can be stored in off-line databases, with restricted access, or as reports, on paper or in PDF format, or any number of other approaches. But whatever approach you use, at some point those records will need to be destroyed, in accordance with your organization’s record retention policy for that sort of data. You guys have a record retention policy, right?
We need to adopt a “Green” mentality for data records. We need to make proper disposal of old records that have come to the end of their useful lives as much a part of system design and maintenance as disaster recovery. Ensure that you have a plan to move records from on-line to secure off-line storage at some well-defined point. Ensure you have the ability to later purge them from off-line storage. Ensure these activities are scheduled as part of the annual operations calendar.
Of course, some records may need to be retained past their normal lives because of a court order, as part of a legal dispute. And some types of records may be subject to summary analysis as a class or group, rather than a simple look-up (think Lilly Ledbetter or other sorts of class action lawsuits). This is why your legal counsel should review your record retention policy, to ensure you keep records as long as required, and no longer. So the selection of the proper storage tool set for history records has to take into account the potential need for these contingencies. Be sure you understand all possible uses (and customers) for history records before you settle on a storage medium.
The bottom line here is that proper stewardship of the organization’s data records requires a life cycle mentality. Just as you have a plan to destroy old hard drives (you do, right?), you should have a plan to manage destruction of old data records. At some point, all of that data quits being an asset, and becomes a liability; legal, technical, or administrative. Recognize the risks, and treat them as such.