Bespoke Data Engineering: Delivering 50TB of Encrypted Data on Physical Media

When a publicly listed data company needed a partner to deliver large format, heavily encrypted data sets on physical spinning disk media, they turned to DataMove. 

What began sounding like a typical “logistics” requirement quickly transpired into a full digital engineering project. This, for us, is the true definition of a “white glove service”.

The Challenge at Hand

Each project delivery so far has involved between 15TB and 50TB (terabytes!) of data, generally composed of tens of millions of heavily encrypted proprietary files. This data is housed securely in the cloud and needs a careful drawdown process, writing to physical hardware (spinning disks) and then finally delivered in person to the customer.

There were several technical constraints to overcome and find a working solution to finesse:

• Rate limits on downloads from the cloud meant that traditional (naïve) transfer methods would take weeks, maybe even months — not the days that we needed to target.
• Write thresholds on the destination drive hardware created bottlenecks.
• The client specified a particular hardware configuration — an external device running RAID across legacy enterprise-grade spinning disks — chosen for reliability and security over raw speed.
• Delivery deadlines were commercial — this was an ongoing project which needed a reliable delivery mechanism.

Our Approach

Our in-house engineers designed a bespoke solution to tackle the rate limit and throughput bottlenecks.

We developed custom Python scripts — one of which uses a partial segmentation method — to parallelise downloads in a way that both maximises sustained throughput to the spinning disks whilst respecting the source rate limits which would always be apparent in any case. This approach effectively bypasses both choke points.

We already run triple dedicated gigabit fibre pipes on lease from Openreach. This gives us essentially unlimited network capacity (with disk rates in mind) to ingest this data at scale.

The hardware build follows the client’s spec — RAID across enterprise spinning disks, specifically configured for the reliability profile the customer required. The finished projects are delivered by DataMove staff to the customer in person.

Why Does Any of This Matter?

It would be easy to look at this as a layman and see it as a “simple data backup”. It isn’t. This is a bespoke engineering solution with a multi-faceted custom codebase developed just for this one client and their specific data drawdown requirement. No two projects at this kind of scale could ever be the same.

The difference between taking these steps or not would ultimately be failure and an inability to deliver.

This is what “white glove service” means to us — hearing of a problem, figuring out a solution, and then actioning a deliverable time and time again.

“When we urgently needed large quantities of data transferred in ways we’d never done before, we turned to DataMove. They were able to provide a capability that was previously not possible, delivered in an extremely collaborative and timely manner.”

— Customer

The Ongoing Relationship

We have delivered multiple projects for this customer to date and the relationship continues. Every project we carry out benefits from our custom scripts and our infrastructure which is already in place.

 We are continually improving this setup, which means subsequent projects run faster, with guaranteed reliability and — ultimately — predictability for the client.

Services Used

• Specialist White Glove Services
• Custom Data Engineering and Python Development
• Dedicated Network Infrastructure (Triple Dedicated Openreach Gigabit Pipes)
• Hardware Build, Configuration and Secure Delivery

Need a Bespoke Solution?

If your project doesn’t fit into a standard service category — if it needs custom development, specialist hardware or a problem-solving approach — we’re happy to talk it through.

Get in touch to discuss your requirements today.