Hacking to liberate RADARSAT-1 data
How four individuals came together at the recent Ottawa Space Apps Challenge to create intuitive access to the RADARSAT-1 Catalog
Imagine the possibilities
Recently (April 2019), it was with great excitement that I learned that the program that helped to launch my career (RADARSAT-1), gave birth to the commercial SAR market, and collected hundreds of thousands of images for so many Canadian and International users over an amazing time period of 17 years would be releasing data to the public through free and open access. The Canadian Space Agency and Natural Resources Canada announced that over 36,000 images would be distributed freely through the Earth Observation Data Management System (EODMS). Immediately following this announcement, I eagerly headed to the EODMS website to discover what images were available and began to dream of many big data projects that could be possible given the availability of these datasets. While RADARSAT-1 was operated mainly as a tasked satellite, unlike the newest breed of satellites that collect standard coverages, and are “always on” (Sentinel-1, Sentinel-2, Planet, etc), advancements in processing power, reductions in storage costs, and ability to handle big data in EO would mean that many new multi-temporal applications leveraging this collection could be possible.
Reality check
A few hours later, reality set in while trying to search, discover and download images from the EODMS system. While the EODMS system was developed over a number of years, and engineered to cover several uses cases in support of the Federal Geospatial Platform (FGP), as a user I was left frustrated using the browser based graphical User Interface (GUI), which presented several limitations. Limits in search results, inability to export the entire catalog for exploration in other GIS tools, and a click intensive user experience did not lead me to answer a simple question – show me the location of all of the data, and let me discover for myself what datasets could be of interest. Answering this fundamental question would be of critical importance before dedicating any further efforts spent downloading images to process big EO datasets. After several hours, I did manage to extract all of the footprint information from the 36,000 free images – however this had to be done through dozens (nearly 100) individual queries using the GUI. Surely, I thought, there must be a better way. EODMS did advertise access to their catalog through machine to machine (M2M) application programming interfaces (APIs), however I had reached the limits of my abilities and did not venture further. Months went by, and I often thought about the possibilities of working with these rich datasets.
Hackers gonna hack
One day, as I went through my usual routine, I loaded up on coffee, said hi to co workers, and checked email – then social media. I stumbled upon an interesting post which piqued my curiosity – the Canadian Space Agency announced that a RADARSAT-1 related data challenge would be one of the available topics for the 2019 Space Apps Challenge – a hackathon. Although I do have some ability to write simple Python scripts and automate certain tasks using commercial off the shelf software, I have never considered myself a developer / coder or hacker. My intuition told me to stay away, and that this was not the place for me. Then I thought, why not. What have I got to lose?
I registered and headed to the opening evening of the Ottawa Space Apps Challenge, hosted by the Technology Information Management (TIM) program at Carleton University, and organized by a dedicated group of local volunteers. The event is one location among hundreds of international locations, all participating over the same weekend in conjunction with the International NASA Space Apps challenge.
When I arrived I was not sure what to do, who to talk to, or how to convince others to join me on my journey to liberate the RADARSAT-1 imagery. Students were already hard at work, with laptops open and furious typing occurring. I saw some of them already producing code, with their favourite integrated development environments (IDEs) up on their screens. After walking past those eager coders, I was greeted by Adam Simonini, who introduced himself to me and we started chatting. I mentioned to him that I had this idea on how to address one of the CSA challenges, but did not know how to code. He showed signs of being interested, so I continued explaining the project to him. He quizzed me, and challenged my assumptions. He spent time with other people from CSA and NRCan who were present at the hackathon to confirm his understanding of the challenge and started to formulate ways in which he could participate. I thought to myself, looks like I’ve managed to recruit one coder! As the evening went on, two other talented developers began asking what Adam and I were up to, and soon after, we had formed a team. By the next day, the team was solidified – we were the “RADARSAT-1 Liberators”. In retrospect, it was Adam’s persistence to understand the problem set before writing a single line of code that was a key ingredient to our success.
Exhilaration, hallucinations and progress
As this was my first hackathon, I didn’t know the culture. I quickly discovered that once hackers sink their teeth into a problem they are passionate about, they tend to stop at nothing to meet challenging objectives. I mostly watched and coached while three skilled developers did their thing. Adam, Nick Kellett and Alan Higginson and I made a rough plan. Adam would work on the front end, Nick would create the back end of the application, and Alan would help glue it all together given his knowledge and experirence with coding but also the RADARSAT program. I created them test datasets pulled from EODMS. They created a code repository, we registered a domain, stood up a database, and created a web service and within a few hours, we had created our minimum viable product (MVP). https://radarsat-1-archive.space was live. Our concept was simple – we would build: “Realtor.ca, for RADARSAT-1”. Let the user pan, zoom, and filter to discover what data is available, rather then forcing them to enter rigid search queries and only see a limited subset of the collection.
The hours fly by – several times we passed on food and water to keep pushing our prototype application forward. At one point a decision was made to pull an all nighter by one of our team members, and surely some hallucinations were experienced! Rather than sleep on the floor, a “bed” was fashioned out of three chairs…it didn’t matter, progress was happening. 48 hours is not a lot of time – our team was firing on all cylinders. By Sunday morning, our application was in great shape, we worked feverishly to add as many features as possible with the amount of time we had left.
Crossing the finish line
By Sunday, 3pm, all coding stopped. We finalized our pitch deck, did some quick dry runs in the back of the room, and prepared to present our project. In the end, our application was chosen to move forward to the national level judging, along with one other Ottawa Space Apps Challenge teams (Carleton University Cyber Security Club – a group of talented computer science students from Carleton University). The four of us formed a bond that lasted past the weekend, with communication continuing over the coming days and weeks. We continued to improve the app, recorded a demo video, and submitted our project for national judging.
Hackathons have the power to move ideas forward, faster
With this event behind me and while returning to work, I reflected on the Space Apps hackathon weekend. A business case I had brought to this team of skilled developers had turned into a working prototype application, all within a 48 hour time span. The format of the hackathon, and the people it attracted is what made it possible. The sheer fact of having a short time window, and establishing a strong connection with motivated individuals had created an impressive outcome. It would not have been possible without the hackathon organizers, sponsors, judges, mentors, coaches and volunteers. Hackathons can be a powerful force to move ideas forward, and I am thrilled to have discovered the Space Apps Challenge. I can truly say that we formed a high performing team through this experience, it was an absolute pleasure to work with Nick, Adam and Alan. We think our idea has great potential, and we’re continuing to pursue it. We believe that the RADARSAT-1 data can be liberated, or at least more easily searched and discovered through an operational application such as the one we made through this experience. This can only help CSA and NRCan meet their objectives with the Open Data initiative, which is to increase utilization and development of innovative information products and applications. Who knows, our concept could really improve the way many other datasets can be explored!
1 https://www.canada.ca/en/space-agency/news/2019/03/open-data-over-36000-historical-radarsat-1-satellite-images-of-the-earth-now-available-to-the-public.html
2 https://en.wikipedia.org/wiki/Radarsat-1
3 https://www.spaceappsottawa.com/
4 https://www.spaceappschallenge.org/
5 https://github.com/adamsimonini/rs1
6 https://www.dropbox.com/s/7qgtcu6ioceuosl/R1_Liberators_rc3.pdf?dl=0
