The Data Conversion Cycle

Many IT projects involve moving operations and users from one system, already in production, to a new system.  Many of these are transaction-processing systems, such as financial management systems, customer relationship management systems, or human capital management systems.  Typically, project teams will need to migrate data records from the system currently in production, which we’ll refer to as the legacy system, to the new system.  This activity is typically referred to as data conversion.  During the course of such an implementation project, there will typically be several conversions performed in support of prototyping and testing.  Consequently, it is useful to think of data conversion as a cycle, repeated as needed.

The diagram below depicts a common model of the data conversion cycle.  There are others, with more or less detail, but over the next few weeks, I plan to write about the various tasks in the model below.  Note that my perspective on the subject is driven by my experience implementing enterprise resource systems, mostly human capital management, payroll, and benefits administration systems of the last twenty years or so.  These days, I manage projects that involve migration to cloud-based, Software as a Service solutions, so I may make some generalizations that don’t fit all situations.  So if you have any questions or objections, please note them in the comments.  I’d like this to be a public collaboration.

Data Conversion CycleNext week, I’ll address defining the scope of the conversion.

Thus I Refute #NoEstimates

In a recent blog post, Woody Zuill – fellow blogger, fellow middle-aged software guy, and fellow banjo picker – gave his personal definition of the #NoEstimates Twitter hash tag that has been the venue for a lot of online discussion lately.

#NoEstimates is a hashtag for the topic of exploring alternatives to estimates for making decisions in software development.  That is, ways to make decisions with “No Estimates”.

Woody and several others in the Twitterverse and Blogosphere have been advocating for making decisions (more accurately, applying constraints) that don’t require estimates, in order to eliminate the need to make other decisions that would require estimates.  Neil Killick recently posted an explanation of how to do Scrum without estimates, including this example:

Instead of depending on an accurate estimate for predictability we can take away the unknowns of cost and delivery date by making them… well, known. The Product Owner can fix the delivery date based on a concrete budgetary and/or time constraint (e.g. 3 days before the Australian Open starts for the Australian Open app is a concrete time constraint, and “we have to build something for $30,000″ is a concrete budgetary constraint).

Implicit in the example is that once time and schedule are fixed, only the scope is flexible.  While this would certainly make things easier for the software developers, it adds complexity for everyone else.  To explore Neil’s example, let’s assume that the Australian Open app is a public-facing, self-service web site where the general public can buy tickets, see the daily results, read and comment on the news coverage of the event, and possibly follow links to the personal web sites of the players.  But what else is involved in serving the public audience interested in the Open?  I ask because, in the real world, not all members of the public are going to use that web site.

Presumably, the Open will have a call center to handle inquiries and purchases from those who don’t find or can’t access the web site.  They may also have an interactive voice response (IVR) system to support certain kinds of queries and possibly even ticket purchases.  So the Decision Makers will want to make the Australian Open app fit into this operational framework, providing capabilities and services for the public that supplement or complement each other.  To do this, they need to have a firm list of capabilities (or scope) defined for each of the three projects – Call Center, IVR system, and web app.  That way, the can staff the Call Center appropriately, work with the marketers to provide instructions to people in directing them to the IVR or app, and otherwise direct user traffic.  But if the Call Center folks and IVR configuration team don’t know until three days before the Open starts what functionality the web app will have, that doesn’t leave them much time to react, do they?

Of course, it might be wise for the Decision Makers to conduct a “buy or build” analysis, since large events are quite common and have spawned a number of service offerings, including Software as a Service.  But in order to conduct that analysis, they would need a firm feature / capabilities list and cost estimate for the internal development effort, to compare it with the vendor offerings.  And if all the internal developers can commit to is cost and schedule, it makes comparisons impossible.

A group of frogs who live at the bottom of a well are enjoying their nightly chorus of croaking, when one of them announces that he is studying astronomy.  When he gets a quizzical look from one of his comrades, he explains that it is the study of the sky.  “You know: that little round thing at the top of the well.”

Complexity will persist in the face of all attempts to reduce it through simplifying assumptions.  Few modern business problems are simple enough to reduce to a small number of decisions, which then drive all other activity.  Apples falling out of trees are fine for explaining gravity to school children, but celestial mechanics has to contemplate multiple bodies.  Similarly, modern business problems are not simply about software.  And that is why we should devote our energies to improving the quality of our software development estimating methodologies, rather than advocating adoption of simplifying assumptions that don’t reflect a holistic view of the operating environment.

Just Observing the Trends

It’s useful to monitor trends.  For example: software-as-a-service (SaaS) provider Salesforce.com is projected to pass SAP as the worldwide leader in customer relationship marketing (CRM) software, with well over US$2B in annual revenues.  Fellow SaaS vendor Workday competes directly with SAP and Oracle, replacing their premises-based ERP human capital management (HCM) and payroll solutions for most new customers.  Revenues for the quarter ending in January doubled year over year.  And Zuora, a SaaS vendor providing a complete solution for selling subscriptions (think ZipCar) has literally created their own niche.  Zuora CEO Tien Tzuo, former chief strategy officer and employee number 11 at Salesforce.com, recently noted, “Something happened 18 months ago. … [now] the CIO has to justify why it isn’t SaaS or cloud.”  Of course, most SaaS solutions have to share information with premises- based solutions, so corporate IT is still a part of the implementation and on-going operation.  But as organizations transition to more SaaS solutions and outsourced services, IT is watching a lot more cloud-to-cloud integrations, from the bleachers.

Another trend: according to W3Techs, WordPress is the content management system in use on 18.2% of all internet sites (including this one).  more importantly, it’s used by nearly 16.7% of Alexa Internet’s “top 1 million” websites, up from 14% two years ago.  It’s popular for the same reason that microwave ovens are popular – you don’t need to know anything to use it, and you get fast results.  Of course, it also helps that it’s free.  I pay a few bucks a year to DreamHost for them to host my domain, but the truth is, if I hosted it on WordPress.com, like nearly 66 million other bloggers, I wouldn’t have to pay anything at all.  And while there are literally thousands of WordPress consultants and a variety of premium services available from the Mothership, most of us don’t use them.  We just pick out the plug-ins, widgets, themes, and other bits we want to use, configure them, and start producing content.  No programming required, just DIY.

Sill another trend: users are getting accustomed to scanning the online stores for software from their phone or iPad, downloading several apps, and after playing with each of them, deleting the ones they don’t want to use.  Both Apple and Google claim more than 800,000 third party applications are available for their respective platforms, and Apple says there are more than 300,000 apps optimized for the iPad.  In many cases, the software uses a “freemium” model, so that users who decide to opt for a more full-featured or ad-free version can simply upgrade for a few dollars, after trying the free version.  The obvious challenge for corporate IT is how to provide secure access for employee-owned devices that will be used to access and manipulate corporate data.  But the more difficult question is, how do we manage their expectations of corporate-developed applications?

I mention these trends because I keep seeing “purists” argue that Agile development methods are producing better software.  And I absolutely agree – WordPress, Salesforce, Workday, and Zuora all use Agile methods, mostly Scrum.  They are developing truly amazing applications, with consumer-grade user experiences, and grabbing market share from the long-time leaders.  They are succeeding at a level that should gratify the folks who composed the Agile Manifesto at Snowbird, Utah in 2001.  But they are gradually putting corporate IT department software developers out of work.

As practicing IT project managers, we need to look ahead a few years, and prepare ourselves and our organizations for the near future, based on the trends we see developing.  And for many of us, the near future will be about transitioning to consumer-grade, user-DIY, SaaS, and mass market mobile solutions that will take BYOD to a whole new problem set.  It’s already proving to be an interesting decade – we just have to avoid getting bogged down with the last decade’s debates.