With the National Data Strategy setting out the Government’s desire to create a data-led public sector, as well as an appetite from councils to deliver data-driven services, many of them now see that migrating from legacy systems to a new provider offers the ideal opportunity to improve data quality and use this investment to its fullest potential in the service of local citizens.
Data migration can be seen as a ‘necessary evil’ when implementing new tech to improve staff and citizens’ lives, and may not be given the attention it deserves. The consequence of this is data decay - key data is excluded from transfer, or is clumsily aggregated in an attempt to retain information with little thought about its future use or accessibility.
Getting data migration right goes a long way to repressing fear and scepticism of change for stakeholders, which is vital in any digital transformation project. With careful planning and execution, data migration can be broken down into a series of manageable areas and councils can succeed in protecting valuable citizen data, making informed decisions and improving services.
Here are the most important elements for councils to consider when undertaking a data migration.
The perfect local authority data migration team
Data migrations require staff with technical and non-technical knowledge. I’d recommend that teams include an expert on the area to which the system relates, who will take the lead on the initial data mapping exercise. This is a key individual who understands current processes and legislative requirements for that data, which is often sensitive. They will also understand and consult with other stakeholders who rely on the data – for example the land charges department which might be in need of solid statistics.
This expert should also understand the legal requirements for councils when it comes to data retention, and they should be empowered to make decisions about which data should and shouldn’t be migrated. These teams should also include an individual with technical knowledge of the source system. This person will have a good understanding of the source database and work closely with the above expert to convert general rules into technical requirements and can also read into the underlying data, helping to inform mapping decisions.
Skeletons in the data closet?
Identifying anomalies in the data up front lets the team correct them before the actual migration, cleansing records at the source. They can also build rules into the migration to resolve issues during the extraction. Identifying data quality issues too late in the migration may result in unexpected delays.
Data quality issues may originate from legacy data migrations with poor mappings or minimal data, invalid or duplicated records, accidental deletions, changes to the way staff used the system in the past or the poor administration of records. It’s important to identify any issues in the data before migrating it - side-stepping the potential for precious citizen data to be lost, damaged, or inaccessible.
Pick your battles
Councils may often believe that a successful transformation relates directly to how much data was carried over. Although this is a quick and convenient metric to use, it can lead to wasted effort, with migration teams focusing too much energy on data that won’t have much value in the future.
To ensure the effort pays off, take a value-based mapping approach. Each field of data you map out should have some value to the organisation. Ask these questions - is there a legislative requirement to hold this data? Do any external departments need this data? Will it aid the council’s future decision-making and improve services?
If the data does not meet any of these criteria, you should question whether it should be migrated. By taking this approach, effort will only be spent on high value data that is essential to the provision of local services.
Stop hoarding data
It’s tempting to keep hold of every piece of data, ‘just in case’. But look at what data retention rules exist today. By excluding useless data from the migration (in line with the value based approach mentioned above), the new system will contain only compliant data and the time it takes to extract the data, and the storage requirement, will reduce.
When you reduce the storage requirement, data backups will run faster and the time taken to restore a full backup of data in the event of a disaster is reduced. It isn’t uncommon for councils to suffer data losses or even security breaches, so recovering that data should be as easy and painless as possible.
Get organised and test, test, test
Data migration is a complex process for councils due to the enormous amount of sensitive data they hold on often legacy-style systems. It is also prone to change between test cycles. Record all mapping decisions in a single table and make it easily searchable for when it is needed. Data extraction routines should also be re-runnable. All of this sets the team up for rapid migration and successful testing.
Testing is also crucial to the success of a full-scale migration. Testing should include a range of record types at different stages of the business process, as well as manufactured test records. Test records do not need to 'make sense' or follow standard processes but are designed to touch as many mapping points as possible and hit every avenue. Manually migrating test records into the target system in line with the mapping document gives a preview of what to expect when records are migrated en masse, as well as highlight any areas that don’t line up with the new processes.
Finally, make sure to validate all data and make sure it is compliant with the new system. That way, the council won’t become the latest headline because of data loss or a fatal security mistake.
Doing the above will ensure that the public’s data is kept safe and services can continue to be run as usual following the go-live. Local authorities hold a raft of sensitive data, from social housing waiting lists to benefits applications, planning applications and the like, and that’s excluding any internal data from staff working in the organisation. A digital transformation project is a huge undertaking. Getting the data part right will mean minimal disruption for staff or citizens, while valuable information is kept both safe and accessible.
Tony Armitage is data team lead at Arcus Global
