Flong

Printed from www.flong.com
Contents © 2017 Golan Levin and Collaborators

Golan Levin and Collaborators

Projects

Sort by : Author | Date | Name | Type

< project thumbnails


Terrapattern

2016 | Levin, G., Newbury, D., McDonald, K., Alvarado, I., Tiwari, A., and Zaheer, M.

Terrapattern

Terrapattern is a tool for exploring the unmapped and the unmappable: a system for finding 
"more like this, please" in satellite photos. It can also be described as "a visual search engine for satellite imagery", "similar-image search for satellite photos", or "a prototype for geospatial query-by-example".

Terrapattern is an open-source software project created by a collaborative team of artists, creative technologists, and students. It is an experimental research provocation, developed in a university setting, with the aim of presenting a new way of exploring, understanding, and visually organizing the world. Developed at the Frank-Ratchye STUDIO for Creative Inquiry at Carnegie Mellon University with support from the John S. and James L. Knight Foundation, Terrapattern went live on May 24th, 2016.

Overview and demonstration video, 9’01" [VimeoYouTube]

Terrapattern is intended to democratize geospatial intelligence, and to prompt new thinking about how we see our planet. Our system is particularly well-suited to locate things that aren’t usually indicated on maps, such as specialized nonbuilding structures and other forms of soft infrastructure. We hope it will help citizen scientists, artists, humanitarians, journalists, and other curious people to identify and track "patterns of interest".

Terrapattern provides an open-ended interface for geospatial query-by-example. Simply click an interesting spot on Terrapattern’s map, and it will find places that look similar. While related research projects focus on detecting specific things (like logging roads or strip-mines), Terrapattern is designed for general-purpose search, allowing users to discover categories for which it wasn’t specifically trained. Our goal is to provide a geospatial software tool that makes it easy for everyday people to specify a thing that they are interested in; to automatically find more examples of that thing; and to provide the locations of those instances in a common data format that easily allows for further examination.

We believe Terrapattern is important because it is open-source, open-access, and open-ended. In this way, Terrapattern also operates as a revelatory artwork: a "panoptic perceptron" for open-ended play, and an absurdist tool to lay bare the rhetoric of 20th-century formalist analysis in urban planning and architecture.


The Terrapattern interface.

Motivation

It has been predicted that, within the next three years, access to daily-updated, whole-earth satellite imagery with sub-meter resolution will become widely available online. There will be many important stories about natural and human activities latent in this data. Without special effort, however, it is doubtful that this imagery will also have the information layers necessary to make such stories discoverable.

In light of this, the Terrapattern prototype is intended to demonstrate a workflow by which users—such as journalists, citizen scientists, humanitarian agencies, social justice activists, archaeologists, urban planners, and other researchers—can easily search for visually-consistent patterns of interest. We are particularly keen to help people locate, characterize and track indicators which have not been detected or measured previously, and which have sociological, humanitarian, scientific, or cultural significance.

In developing Terrapattern, we sought to test the following hypotheses:

  • That there are undiscovered patterns of activity that are only visible from the vantage point of satellite imagery;
  • That these patterns can offer insight into trends, stories and phenomena of social, economic, ecological, anthropological, humanitarian, scientific, or other cultural interest;
  • That there exists a subset of such patterns in satellite imagery which, because of their repetition and visual consistency, lend themselves particularly well to automated detection and analysis by means of machine learning and computer vision;
  • That there is demand for information about these patterns among domain experts, as well as among journalists, citizen scientists, humanitarian NGO’s, and the general public;
  • And that it is possible to create software tools which make such workflows easy and reliable.

Raw screen recording of Terrapattern in use, 7’20" [VimeoYouTube]

Implementation

There has never been a more exciting time to observe human activity and study the patterns of humanity’s impact on the world. To develop Terrapattern, we sought to help people discover such patterns in satellite imagery, through the use of "deep learning" techniques.

Terrapattern uses a deep convolutional neural network (DCNN), based on the ResNet ("Residual Network") architecture developed by Kaiming He et al. We trained a 34-layer DCNN using hundreds of thousands of satellite images labeled in OpenStreetMap, teaching our neural network to predict the category of a place from a satellite photo. In the process, our network learned which high-level visual features (and combinations of those features) are important for the classification of satellite imagery.

We trained our DCNN using 466 of the OpenStreetMap categories (such as "airport", "marsh", "gas station", "prison", "monument", "church", etc.), with approximately 1000 satellite images per category. After training the model, we removed the final classification layer of the network and extracted the next-to-last layer of the DCNN. Using this layer of proto-features (a technique called "transfer learning"), we computed descriptions for millions more satellite photos that cover the metropolitan regions of Pittsburgh, Detroit, Austin, Miami, Berlin, San Francisco, and New York. When we want to discover places that look similar to a visitor’s query, we just have to find places whose descriptions are similar to those of the tile they selected. To perform this search in near real time, we use the CoverTree algorithm for K-Nearest Neighbors.


Examples


A collection of circular structures in the San Francisco region, discovered with Terrapattern.


Suburban cul-de-sacs in the Pittsburgh region, discovered with Terrapattern.

Terrapattern is ideal for discovering, locating and labeling typologies that aren’t customarily indicated on maps. These might include ephemeral or temporally-contingent features (such as vehicles or construction sites), or the sorts of infrastructure (like fracking wells or smokestacks) that only appear on specialist blueprints, if they appear at all.

We here share some discoveries of our own, made with the Terrapattern system. It is important to point out that the Terrapattern system was not trained on any of the categories shown below, but instead recognizes them because of their common visual features.

Here, for example, are golf course sand traps identified by our system in the Pittsburgh metro region:
Golf course sand traps in Allegheny county, identified by Terrapattern

Here, Terrapattern identified some of Pittsburgh’s finest school bus depots: School bus depots in Allegheny county, identified by Terrapattern

...certain attractive runway markings from various New York airports, Runway lines from various New York ariports, identified by Terrapattern

...and a collection of nautical wrecks in NYC-area coastal waters: Nautical wrecks in the NYC area, identified by Terrapattern


Press about Terrapattern

Terrapattern has been discussed in a range of publications, including The Atlantic, The New Yorker, Creative Applications, Der Standard, Popular Science, and WIRED:

Anzilotti, Eillie. "Mapping the Hidden Patterns of Urban Life". Citylab, 6/7/2016. [PDF]
Anzilotti, Eillie. "Mapeando los patrones ocultos de la vida urbana". Univision, 6/16/2016. [PDF]
Blakemore, Erin. "This New Satellite Project Helps People Find Patterns in City Spaces". Smithsonian, 6/9/2016. [PDF]
Bravo, Eduardo. "La web que encuentra lugares parecidos entre sí desde el cielo". Yorokobu.es, 5/27/2016. [PDF]
Brien, Jörn. "Satellitenbilder durchsuchen mit Terrapattern: Suchmaschine arbeitet mit Künstlicher Intelligenz". t3n.de, 6/6/2016. [PDF]
Brown, Eileen. "Terrapattern search engine finds patterns in the Google Earth landscape". ZDNet, 7/14/2016. [PDF]
Brownlee, John. "This Neural Network Reveals Your City's Secret Patterns". Fast Company, 6/1/2016. [PDF]
Chen, Qi. "Open source Terrapattern, the public use of artificial intelligence Nuggets big data from remote sensing (开源的Terrapattern, 让公众利用人工智能从遥感大数据中掘金)". 3sNews, 6/12/2016. [PDF]
Clarke, Keir. "Searching for Map Patterns". GoogleMapsMania, 5/26/2016. [PDF]
Cohen, Tanya. "НОВЫЙ ЛУЧШИЙ САЙТ: ПОИСКОВИК ПО ИЗОБРАЖЕНИЯМ СО СПУТНИКА". Knife.Media, 5/28/2016. [PDF]
Coldewey, Devin. "Terrapattern is reverse image search for maps, powered by a neural network". TechCrunch, 5/25/2016. [PDF]
"Cool Tech! TerraPattern, open-source tool for discovering 'patterns of interest'". GISUser, 5/31/2016. [PDF]
Dempsey, Caitlin. "Terrapattern: Search Engine for Satellite Imagery". GIS Lounge, 5/28/2016. [PDF]
"Geo-Suchmaschine vergleicht Satellitenbilder". MittelstandsWiki, 6/13/2016. [PDF]
Heller, Piotr. "Suchmaschine für Satellitenbilder: Wir wissen, wo euer Flugzeug steht". Spiegel Online, 6/3/2016. [PDF]
Hlács, Ferenc. "Az égbe költözik a gazdasági analitika". hwsw.hu, 6/13/2016. [PDF]
Ibáñez, Alvaro. "Terrapattern, el buscador de imágenes de satélite similares". Microsiervos, 5/28/2016. [PDF]
"Il motore di ricerca dei paesaggi". Il Post, 6/9/2016. [PDF]
"In addition to image recognition artificial intelligence can accurately locate GPS coordinates (除了图片识别 人工智能还能准确定位GPS坐标)". TechWeb, 6/12/2016. [PDF]
Kohlstedt, Kurt. "Terrapattern: Satellite Image Search Engine Matches Similar Places". Weburbanist, 5/30/2016. [PDF]
Kottke, Jason. "Satellite imagery search engine". Kottke.org, 5/25/2016. [PDF]
Manaugh, Geoff. "Subverting Our New Space Overlords". The Atlantic, 6/9/2016. [PDF]
Mandelbaum, Ryan F. "This website lets you find the hidden similarities in big cities". PopSci.com, Popular Science, 5/25/2016. [PDF]
Meyer, Robinson. "The Thrill of Terrapattern, a New Way to Search Satellite Imagery". The Atlantic, 5/27/2016. [PDF]
Merkezi, Haber. "Terrapattern uydu haritasında görsel arama yapıyor". TechInside, 5/30/2016. [PDF]
"Neuronales Netz: Mit Terrapattern die Welt erkunden". Der Standard, 6/4/2016. [PDF]
Nickel, Katharina. "Mit diesem Tool könnt ihr Satelliten-Bilder für eure Arbeit nutzen". WIRED.de, 8/29/2016. [PDF]
Parlangeli, Diletta. "Le città nascoste su Terrapattern, il motore di ricerca di immagini satellitari". WIRED.it, 6/23/2016. [PDF]
Pluta, Werner. "Terrapattern erkennt Strukturen auf Satellitenbildern". Golem.de, 6/6/2016. [PDF]
Reese, Hope. "Google Maps meets AI: Carnegie Mellon's Terrapattern can find and map every pool in New York City". TechRepublic, 6/8/2016. [PDF]
Schieb, Jörg. "TerraPattern: Neue Such-Maschine für Satelliten-Aufnahmen". Scheib.de, 6/8/2016. [PDF]
Singleton, Micah. "Terrapattern is the first open-access visual search engine for satellite maps". The Verge, 5/27/2016. [PDF]
Stinson, Liz. "Terrapattern is Like a Search Engine for Satellite Imagery". WIRED.com, 5/27/2016. [PDF]
"Terrapattern Berlin". Nerdcore.de, 6/5/2016. [PDF]
"Terrapattern, il motore di ricerca intelligente per foto satellitari.". HDBlog.it, 6/9/2016. [PDF]
"Terrapattern: Webdienst durchsucht Satellitenbilder". Futurezone Technology News, 6/4/2016. [PDF]
Toobin, Adam. "Kill Some Time With Terrapattern, Which Turns Satellite Images Into Art". Inverse.com, 5/27/2016. [PDF]
Twilley, Nicola. "Meet Terrapattern, Google Earth's Missing Search Engine". The New Yorker, 5/25/2016. [PDF]
Visnjic, Filip. "Terrapattern – Neural network visual search tool for satellite imagery". CreativeApplications.net, 5/25/2016. [PDF]
"Visual search map advent of neural network Reverse Image Search (视觉搜索地图问世, 神经网络实现逆向图像搜索)". Lei Feng Network, 5/26/2016. [PDF]
Voon, Claire. "A Visual Search Engine for the Aerial Patterns of Cities". Hyperallergic, 7/6/2016. [PDF]
"When Google Maps experience AI: Terrapattern AI to find and locate the swimming pool in New York City (当谷歌地图遇到人工智能: Terrapattern AI能找到并定位纽约市的游泳池)". cnBeta.com, 6/12/2016. [PDF]
Whitehead, Timothy. "Terrapattern, the search engine for imagery". Google Earth Blog, 5/27/2016. [PDF]
Whitehead, Timothy. "Image recognition and Google Earth". Google Earth Blog, 7/28/2016. [PDF]
Wolfson, Elijah. "There’s now a tool you can use to scour satellite images for criminal activity and environmental devastation". Quartz, 8/25/2016. [PDF]
Worms, Anne-Cécile. "Terrapattern, le nouveau moteur de recherche des images satellite". Makery, 5/31/2016. [PDF]
Yau, Nathan. "Visual search tool for satellite imagery". FlowingData, 6/1/2016. [PDF]


Credits

Terrapattern was created by Golan Levin, David Newbury, and Kyle McDonald with the assistance of Irene Alvarado, Aman Tiwari, and Manzil Zaheer. The project was developed at the CMU Frank-Ratchye STUDIO for Creative Inquiry with support from the John S. and James L. Knight Foundation.


Citations

When citing this project in bibliographies, please use:
Levin, G., Newbury, D., McDonald, K., Alvarado, I., Tiwari, A., and Zaheer, M. "Terrapattern: Open-Ended, Visual Query-By-Example for Satellite Imagery using Deep Learning". http://terrapattern.com, 24 May 2016.


Project Media

Golan Levin premiered Terrapattern in this half-hour lecture presentation at the Alt-AI conference in New York City, 24 May 2016.

The Terrapattern software can be experienced interactively at http://terrapattern.com. Source code for Terrapattern can be found at the project Github repository. The Terrapattern code is free software and open source, made available under the MIT Licence

We maintain an archive of high-resolution Terrapattern images in this Flickr archive: http://flic.kr/s/aHskzPzVE7. These images are available under the Creative Commons CC-BY-SA 4.0 licence.

Images (click to enlarge)

Terrapattern Terrapattern Terrapattern Terrapattern