Tony Armitage 24 March 2021

Getting the data right: seamless data migration for councils

Getting the data right: seamless data migration for councils image

With the National Data Strategy setting out the Government’s desire to create a data-led public sector, as well as an appetite from councils to deliver data-driven services, many of them now see that migrating from legacy systems to a new provider offers the ideal opportunity to improve data quality and use this investment to its fullest potential in the service of local citizens.

Data migration can be seen as a ‘necessary evil’ when implementing new tech to improve staff and citizens’ lives, and may not be given the attention it deserves. The consequence of this is data decay - key data is excluded from transfer, or is clumsily aggregated in an attempt to retain information with little thought about its future use or accessibility.

Getting data migration right goes a long way to repressing fear and scepticism of change for stakeholders, which is vital in any digital transformation project. With careful planning and execution, data migration can be broken down into a series of manageable areas and councils can succeed in protecting valuable citizen data, making informed decisions and improving services.

Here are the most important elements for councils to consider when undertaking a data migration.

The perfect local authority data migration team

Data migrations require staff with technical and non-technical knowledge. I’d recommend that teams include an expert on the area to which the system relates, who will take the lead on the initial data mapping exercise. This is a key individual who understands current processes and legislative requirements for that data, which is often sensitive. They will also understand and consult with other stakeholders who rely on the data – for example the land charges department which might be in need of solid statistics.

This expert should also understand the legal requirements for councils when it comes to data retention, and they should be empowered to make decisions about which data should and shouldn’t be migrated. These teams should also include an individual with technical knowledge of the source system. This person will have a good understanding of the source database and work closely with the above expert to convert general rules into technical requirements and can also read into the underlying data, helping to inform mapping decisions.

Skeletons in the data closet?

Identifying anomalies in the data up front lets the team correct them before the actual migration, cleansing records at the source. They can also build rules into the migration to resolve issues during the extraction. Identifying data quality issues too late in the migration may result in unexpected delays.

Data quality issues may originate from legacy data migrations with poor mappings or minimal data, invalid or duplicated records, accidental deletions, changes to the way staff used the system in the past or the poor administration of records. It’s important to identify any issues in the data before migrating it - side-stepping the potential for precious citizen data to be lost, damaged, or inaccessible.

Pick your battles

Councils may often believe that a successful transformation relates directly to how much data was carried over. Although this is a quick and convenient metric to use, it can lead to wasted effort, with migration teams focusing too much energy on data that won’t have much value in the future.

To ensure the effort pays off, take a value-based mapping approach. Each field of data you map out should have some value to the organisation. Ask these questions - is there a legislative requirement to hold this data? Do any external departments need this data? Will it aid the council’s future decision-making and improve services?

If the data does not meet any of these criteria, you should question whether it should be migrated. By taking this approach, effort will only be spent on high value data that is essential to the provision of local services.

Stop hoarding data

It’s tempting to keep hold of every piece of data, ‘just in case’. But look at what data retention rules exist today. By excluding useless data from the migration (in line with the value based approach mentioned above), the new system will contain only compliant data and the time it takes to extract the data, and the storage requirement, will reduce.

When you reduce the storage requirement, data backups will run faster and the time taken to restore a full backup of data in the event of a disaster is reduced. It isn’t uncommon for councils to suffer data losses or even security breaches, so recovering that data should be as easy and painless as possible.

Get organised and test, test, test

Data migration is a complex process for councils due to the enormous amount of sensitive data they hold on often legacy-style systems. It is also prone to change between test cycles. Record all mapping decisions in a single table and make it easily searchable for when it is needed. Data extraction routines should also be re-runnable. All of this sets the team up for rapid migration and successful testing.

Testing is also crucial to the success of a full-scale migration. Testing should include a range of record types at different stages of the business process, as well as manufactured test records. Test records do not need to 'make sense' or follow standard processes but are designed to touch as many mapping points as possible and hit every avenue. Manually migrating test records into the target system in line with the mapping document gives a preview of what to expect when records are migrated en masse, as well as highlight any areas that don’t line up with the new processes.

Finally, make sure to validate all data and make sure it is compliant with the new system. That way, the council won’t become the latest headline because of data loss or a fatal security mistake.

Doing the above will ensure that the public’s data is kept safe and services can continue to be run as usual following the go-live. Local authorities hold a raft of sensitive data, from social housing waiting lists to benefits applications, planning applications and the like, and that’s excluding any internal data from staff working in the organisation. A digital transformation project is a huge undertaking. Getting the data part right will mean minimal disruption for staff or citizens, while valuable information is kept both safe and accessible.

Tony Armitage is data team lead at Arcus Global

For your free daily news bulletin
Highways jobs

Commercial Customer Manager

North Yorkshire County Council
£39,880 to £43,857
Are you someone who is innovative and has the drive to increase revenue for the Council whilst ensuring excellent service provision to our customers? Northallerton, North Yorkshire
Recuriter: North Yorkshire County Council

Social Care Lead Officer

Sandwell Metropolitan Borough Council
£35,745 - £40,876 pro rata per annum
A vacancy has become available for a part time Social Care Lead Officer, to cover in... Sandwell, West Midlands
Recuriter: Sandwell Metropolitan Borough Council

Communications & Corporate Affairs Manager

Sandwell Metropolitan Borough Council
£47,845 - £50,900 per annum
To apply for this role please upload your most recent CV that sets out your relevant experience against the job description. Sandwell, West Midlands
Recuriter: Sandwell Metropolitan Borough Council

Career Grade Planning Officer

Royal Borough of Greenwich
£27978 - £37722 per annum + n/a
Career Grade Planning Officer - West Area and East Area Team The Royal Borough of Greenwich continues to faces an unprecedented growth agenda in terms England, London
Recuriter: Royal Borough of Greenwich

Highway Engineer - Reactive Maintenance

Bracknell Forest Borough Council
£35,354 - £40,506 incl. LWA, plus £963 car allowance
We make a difference to the environment we live in. Come and help us manage our highway infrastructure. Bracknell, Berkshire
Recuriter: Bracknell Forest Borough Council

Public Property

Latest issue - Public Property News

This issue of Public Property examines how how flexible workspaces can lead the way in regeneration for local authorities, Why local authority intervention is key to successful urban regeneration schemes and if the Government’s challenge of embracing beauty is an opportunity for communities.

The March issue also takes a closer look at Blackburn with Darwen Council's first digital health hub to help people gain control over health and care services.

Register for your free digital issue