The Skies Epitomized

Art + Machine Learning showing how we see the sky.

The Skies Epitomized is a series of artworks exploring the essence of the sky from the perspective of humans gazing at it. The works were created in collaboration between Maja Petrić, an artist, and Nebojša Jojić, a machine learning researcher. The artwork is derived from big data through a machine learning algorithm that is used to create visual summarizations (epitomes) of sky images people posted on the Internet.

Exhibited: December 2015 – March 2016

Maja Petrić explains the motivation behind the art: I find it fascinating to see a visualization of how we see the world. Nebojša Jojić’s epitome algorithm actually paints the picture of the sky in the eyes of the Internet community, which includes most of us. I believe that the image of “the sky in our eyes” can unlock understanding about human relationship towards our origin, and our surroundings.

Nebojša Jojić’s epitome model is a generative model of images that assumes that all images of interest come from a large panorama in which individual pixels can change slightly. The individuals are small cutouts with small local perturbations. The panorama is much larger than the images in the set, but much smaller than a tiling of all images would be. The panorama is thus referred to as an epitome of the set, as it captures the variation seen in the data in a miniature form, but also expands on individual images by placing them in the context of other images. The learning algorithm iteratively maps each of the images from the set to the best matching locations in the epitome and then re-estimates the panorama and the variation parameters until it achieves convergence.

People have been turning their eyes towards the sky for meaning and guidance since prehistoric times. Our gaze directed towards the sky has shaped astronomy, mythology, religion, agriculture, art, physics, and world-views of cultures across this globe. As an artist, I am interested in the meaning of that gaze at the sky as an element that is both mundane and archetypal for the history of our civilization.

I was guided by the notion that on Earth, we share one sky, but each person looking at it has its’ own experience of it that depends on time and place they are observing it, but mostly it depends on their “eyes”, subjective view. The multitude of mental images of the sky reflects large numbers of perspectives about the space that surrounds us and our place in the universe.

I wonder what is the significance of our way of perceiving the sky and how is that perception of the sky shaping the environment and our behavior towards the surroundings, and in so other people. In this project, we used Nebojša Jojić’s epitome model to construct the visualization (epitome) of sky images people posted on the Internet. Here, a machine learning technique allows us to go inside the head of the Internet and have a peek of it’s mental image of the sky, and also of the process of constructing that image.


THE EPITOME OF EARTH SKIES, a world map of epitomized skies
maja_cover1The machine learning algorithm sifts through big data for representational images of the sky for each climate and uses it to create visual summarization (epitome) of people’s perception of the sky changing across the globe. It is a visualization of the sky from the human perspective. Each location in the world is covered by the most recognizable kind of sky in that area. It is meant to be reminiscent of the transient nature of the experience people have when they gaze up. The goal is to allow the viewer to grasp the wide range of experiences and memories from these geographic locations.



This series epitomizes the online images of the sky retrieved through web search for keywords corresponding to specific times and places. Through these epitomes, we depict the collective view of the sky across several locations in time and space. These views are facilitated by the vast collection of images people around the world have posted on the web, and are thus also a reflection of how the web might “comprehend” the sky in its boundless variation: Epitomes reveal archetypical skies defined by a global Internet community. With this series we are both inquiring about what people see when they gaze into the sky, and exploring ways in which human views of the sky shape our landscapes.

Maja posed a question: What is a collective perception of the polar sky?

In search of an answer, we used big data that was processed by a machine learning algorithm to construct a visual summarization (epitome) of the polar sky.

the_skies_epitomized_2To create the epitome of the polar sky, the algorithm sifted through big data in search of the images under the keywords sky, skyscape, north+pole+sky, south+pole+sky and used Nebojša Jojić’s epitome model to extrapolate parts in found images into an epitomized data set representing what people perceive to be a typical polar sky.

the_skies_epitomized_3Next to each sky epitome is a photograph of a person gazing at the sky. The photographs are retrieved from the web in the same manner: by web search for a person that represents that location, climate, or time point.

the_skies_epitomized_4Epitome of the Polar Skies

the_skies_epitomized_5Epitome of the Tropical Monsoon Skies

the_skies_epitomized_6Epitome of the Arid Climate Skies

the_skies_epitomized_7Epitome of the Moderate, Hot and Humid Skies

the_skies_epitomized_8Epitome of the Moderate, Mild and Rainy Skiesthe_skies_epitomized_9Epitome of the Tropical Wet Skies

the_skies_epitomized_10Epitome of the 9/11 Skiesthe_skies_epitomized_11Epitome of the Baghdad Skies



maja_portraitMaja Petrić is an artist working at the interface of science, technology, and art. Her work is about transforming poetic experience of space through light. During her art residency at Microsoft Research, she has been exploring the essence of the sky both thematically and experientially in collaboration with natural interaction researcher Hrvoje Benko and machine learning researcher Nebojša Jojić.

Maja holds the Doctorate in Digital Art and Experimental Media (DXARTS) from University of Washington and the Masters degree in new media art from New York University, Tisch School of the Arts, Interactive Telecommunications Program (ITP). Maja grew up in Croatia during the violent fragmentation of Yugoslavia. It is then that she became preoccupied with using art to transform the sense of her surroundings. Her work is about transforming the poetic experience of space through light.

Some of the venues where she exhibited include Henry Art Gallery, Matadero Madrid, Medialab Prado, and Amazon Gallery. Some of the awards she received include Richard Kelly Light Art Award, two Thunen Lighting Awards, and Doctoral Fellowship from Croatian Science Foundation. Most recently, The Centre for International Light Art nominated her for International Light Art Award.



Maja Petrić, studio99 Artist in Residence at Microsoft Research from Oct. – Dec. 2015
Nebojša Jojić, Principal Researcher at Microsoft Research