Using AWS to batch process slippy map tiles from aerial photographs
The following is an excerpt from 2pi Software Director, Carsten Eckelmann’s presentation to the 28 August Geoscience meetup in Australia.
- Create a set of slippy map tiles from aerial photographs
- Current procedure takes weeks to finish
- Goal was to perform this in just days
- Use AWS Batch for parallel processing
- Python/GDAL for spatial image transformations programming by Dr. Adam Steer from Spatialised
The tasks for the Python scripts were to Load aerial photographs to S3 (GeoTIFF) (manual), create a tile index from photograph metadata. Once completed, it would build a mosaic (WGS84) in GeoJson format and finally create slippy map tiles for zoom levels 12 – 19.
The following services from Amazon Web Services were utilised:
- Input S3 bucket contains the image files to be processed
- Working storage S3 bucket
- Output S3 bucket
- Lambda function to start things off
- CodeCommit repo containing software and configurations
- Batch Job Definitions, one for each Batch Job
- Batch Compute specifications, one for each Batch Job
- Batch Job Queue for the Batch Compute Environment
- Docker image for use by the Batch Compute Environment
- SNS entry to send completion notifications
The different jobs were uploaded to ECS Batch a service that uses a managed compute environment to run docker images on a fleet of container instances. ECS Batch manages auto scaling and distribution of jobs to the configured number of instances and can even utilise spot instances for cost efficiencies. The ECS Batch console provides a great way to monitor the process of jobs.
All of ACT accounts for 44 tiles. Since all processing is done in parallel, all 44 tiles can theoretically be processed to Zoom Level (ZL) 18 in 15 hours and to ZL 19 in 50 hours.
The total cost for mapping all of ACT using 1 EC2 instance per tile, per zoom level with each process completing a map square was barely over $2,000 AUD as of June 2019.
As an AWS Select Consulting Partner, 2pi Software has many experienced cloud systems architects who are skilled in delivering cost efficient, scalable and reliable infrastructures designed to handle just about anything. If your business or organisation is looking to migrate to the cloud, give us a call today!