Data migration sits at the heart of almost every significant Australian government IT project, yet it's treated as a technical afterthought far more often than a first-class workload. When agencies move from legacy systems to modern platforms, the data has to come with them. How well agencies plan, execute, and validate that transfer determines whether a go-live date holds. In practice, data migration is where timelines slip, budgets expand, and post-implementation reviews uncover the worst surprises.
Why data migration keeps catching agencies off guard
The core problem is sequencing. Most program plans attach data migration to the final weeks before go-live, treating it as a step that happens after everything else is built. By that point, there's no schedule buffer left. When the data doesn't load cleanly, the project team faces a binary choice: delay the go-live or accept dirty data in production. Neither is good. Delayed go-lives attract ministerial attention. Dirty data in a production system serving citizens creates downstream integrity problems that can take years to untangle.
The honest answer is that data migration should start at discovery. Before a single line of new platform configuration is written, agencies need a clear picture of what data exists, where it lives, what format it's in, and whether it's actually worth migrating. That last question is the one most teams skip. Government systems accumulate records for decades. Not all of them need to follow the agency into a new system.
The source data problem
Legacy government systems frequently hold data that was entered across multiple decades, by multiple operators, under multiple data standards that weren't enforced consistently. Fields that look structured often aren't. Date formats vary within the same column. Mandatory fields were sometimes left blank because a workaround existed at the time. Free-text fields hold structured data that was never migrated into proper columns because nobody got around to it.
Agencies that discover these issues during data profiling, ideally six to twelve months before go-live, have time to decide what to do. Agencies that discover them during a dress rehearsal two weeks before go-live do not. Data profiling doesn't require the new system to be ready. It requires access to the source, a data analyst, and a willingness to document what's found honestly rather than optimistically.
Three things consistently emerge from profiling exercises on Australian government source systems:
- Duplicate records, often created when manual entry bypassed unique-key validation
- Referential integrity breaks, where child records point to parent records that no longer exist
- Classification mismatches, where data categorised under an old taxonomy doesn't map cleanly to the new system's field structure
None of these are unsolvable. All of them take time. The teams that surface them early have options. The teams that surface them late have a crisis.
Transformation rules and the documentation gap
Every migration involves transformation: source data has to be reshaped to fit the target schema. The rules that govern this transformation need to be documented explicitly, agreed between business owners and technical teams, and tested against real data samples before they're run at scale.
What agencies often produce instead is a rough mapping document that covers the obvious fields and assumes the edge cases will be handled during testing. Edge cases are not handled during testing. They're found during testing, panicked over, resolved inconsistently, and then discovered again in production when a citizen or officer encounters a record that was processed differently from the one next to it.
Formal transformation rules documentation forces the right conversations early. It makes business owners articulate what they actually want to happen when a source value doesn't match a target enumeration, or when a mandatory target field has no source equivalent. These are policy decisions disguised as technical ones. Getting them made before migration runs, rather than during, is the difference between a controlled migration and a reactive one.
Cutover planning and the dress rehearsal discipline
Agencies that run data migration well treat the production cutover as a performance, not a first attempt. That means running full dress rehearsals against production-scale data volumes, timing every step, documenting what breaks, fixing it, and running again. A single dress rehearsal is the minimum. Two or three is better.
The dress rehearsal surfaces the things that only appear at scale: the extract job that takes 40 minutes on a subset but 14 hours on the full dataset, the transformation that processes 10,000 records cleanly but throws errors at record 10,001 due to a character encoding edge case, the load process that works fine but locks a database table in a way that blocks the validation script from running in parallel.
Cutover timing matters too. Most agencies schedule production migration over a weekend to minimise service disruption. That window is real, but it's shorter than it feels when you're in it. Every step in the cutover plan needs a time estimate, an owner, a go/no-go checkpoint, and a rollback trigger. If the migration isn't through validation by a defined point on Saturday night, what happens? Who makes that call? If the answer is "we figure it out when we get there," the cutover plan isn't finished.
Validation and reconciliation after go-live
A migration that loads without errors is not a migration that succeeded. Data can transfer completely and still be wrong. Validation means checking that the data in the new system matches the data in the source system in a way that is meaningful to the business, not just technically present.
Record counts are a floor, not a ceiling. Matching totals on key financial fields, checking that high-value or high-risk records transferred with their relationships intact, and sampling manual spot-checks across different data segments are all part of a credible validation regime. Acceptance testing for the broader system should include data validation as an explicit workstream, with business owners signing off on data quality, not just functional behaviour.
Post-go-live, agencies need a parallel-run period or at minimum a clear reconciliation window where discrepancies can be reported, investigated, and resolved before the source system is decommissioned. The temptation to decommission the legacy system immediately after cutover is understandable but risky. If a data issue surfaces two weeks after go-live, having the source system available for comparison is enormously valuable. Without it, the agency is reconstructing intent from incomplete records.
Sensitive data and security obligations
Government data migration almost always involves personal information, often at scale. The Privacy Act reforms now place sharper obligations on how agencies handle data in transit and at rest during transformation activities. Migration pipelines are not exempt from these obligations simply because the data is moving between internal systems.
Agencies need to map the data flows involved in their migration, identify where personally identifiable information or sensitive data passes through intermediate staging environments, and ensure those environments are secured to the same standard as production. Staging databases loaded with production citizen data and left accessible during a migration project are a genuine security risk. This isn't hypothetical: it's the kind of exposure that forensic investigations find after the fact.
Encryption in transit, access controls on staging environments, audit logging of who accessed migration data and when, and a formal data destruction process for staging copies once the migration is validated are baseline controls. They should be in the migration plan from the start, not retrofitted when a security review flags the gap three weeks out from go-live.
What separates the agencies that get this right
The agencies that run clean data migrations share a few consistent characteristics. They treat data migration as a workstream with its own plan, its own resourcing, and its own governance. They start profiling source data before the target system is built. They put a named data migration lead in the program structure, not just a database administrator who gets assigned the work during delivery. They fund enough time for dress rehearsals. And they make the business own data quality decisions rather than leaving them to the technical team.
None of this is exotic. It's discipline applied to a part of the project that most programs underinvest in because it doesn't produce visible deliverables until something goes wrong.