DEV Community

Cover image for Getting started with automated data downloads
Charley Glynn
Charley Glynn

Posted on • Updated on

Getting started with automated data downloads

Data can be incredibly useful. By analysing or interpreting the information contained in data, better decisions can be made.

Ordnance Survey is one of the world’s oldest data organisations. We have gathered information about Great Britain since 1791, organising, analysing and disseminating maps and other information. As information technologies have evolved from tables and charts and pen-and-paper calculations to databases, vector graphics and sophisticated machine learning algorithms, OS has been a leader in adopting and innovating new ways to capture, store, process and share valuable location information.

The challenge — managing data

As the scale of data grows, so does its potential usefulness — and the complexity of managing it. Understanding a small table of numbers is easy, less so for a complex database with dozens of tables and millions of rows. With code though, it has never been easier for programmers to retrieve and manipulate large volumes of data and extract and understand the valuable information they contain.

image

boundaryline-base-zoomstack-newcastle-small

As every data scientist and web developer knows, well-structured datasets are difficult to find and even harder to create. Data wrangling is a major task and is often overlooked by those less familiar with the data science process. For algorithms to extract accurate and meaningful information from a dataset — especially using a powerful analysis libraries openly available like Python’s pandas — the data needs to be consistently structured and clean.

The answer — automated data downloads

Consistent with our tradition of working to make accurate and authoritative location data about Great Britain useful to our users and customers, we’ve been making a wide range of our datasets available for download over the web. Through the new OS Data Hub, users can connect to a number of APIs including the OS Maps, OS Vector Tile, OS Features, OS Names, OS Linked Identifiers and OS Downloads APIs. These resources range from raster and vector map tiles to vector features, naming and identification data or even full OS datasets.

image

codepoint-open-base-none-wales-small

With the OS Maps, OS Vector Tile and OS Features APIs, users authenticate their request with an API key created on their OS Data Hub account where they can dynamically fetch the data they need when they need it. This is powerfully useful in web maps, data science workflows and desktop GIS applications. These APIs enable developers and spatial analysts to create and customise interactive maps, and to fetch location data with rich attribution — information about a feature useful for visualisation and analysis purposes.

The OS Downloads API makes a number of OS OpenData datasets available for automated download over the internet. This new service makes collecting useful spatial data from Ordnance Survey as easy as pasting a URL, or sending an HTTP GET request. The OS Downloads API documentation has sample code, and we’ve created a tutorial to help JavaScript developers fetch data programmatically with NodeJS.

image

Unlocking value by improving access

The OS Data Hub APIs are the result of a major effort at OS to make our National Geographic Database easier to access than ever before. A broad range of users in Great Britain from civil servants to NGO workers, academic researchers, entrepreneurs and large companies, could enhance the quality and efficiency of their work with the insights they could derive from OS data. We’re working hard to make sure the path from data to decision-making is as short as possible. Automated data downloads are one of the ways we’re doing that.

We’ve written a tutorial to get you started with automated data downloads from the OS Downloads API using NodeJS — find it here.

Top comments (0)