MIT develops MosAIc algorithm to find hidden connections between art across cultures

exploring the boundless trove of artwork documented worldwide, a team of researchers at MIT develops an art-recognizing algorithm ‘MosAIc’ to find hidden connections. working with microsoft, the group at MIT’s computer science and artificial intelligence laboratory (CSAIL) catalogues and explores paintings and sculptural art from the metropolitan museum of art (the met) and amsterdam’s rijksmuseum. the endeavor was first inspired by a rijksmuseum exhibit ‘rembrandt and velazquez’ which showcased the unexpected parallels between francisco de zurbarán’s portrait ‘the martyrdom of saint serapion’ and jan asselijn’s ‘the threatened swan,’ two works that portray an uncanny visual resemblance.

all images courtesy of MIT CSAIL
‘the dolly sisters,’ european, photographs (left) | untitled piece, chinese, precious stone

MosAIc is the result of a collaboration between microsoft and MIT’s computer science and artificial intelligence laboratory (CSAIL). MosAIc discovers analogous works of art — including such a range of mediums as painting, sculpture, ceramic, woodwork, and originating from different cultures, regions, and time periods. the works are paired through the use of deep networks to algorithmically understand how ‘close’ two images are. CSAIL PhD student mark hamilton, the lead author on a paper about MosAIc comments on the first pairing: ‘these two artists did not have a correspondence or meet each other during their lives, yet their paintings hinted at a rich, latent structure that underlies both of their works.’

francisco de zurbarán’s ‘the martyrdom of saint serapion’ (left) |  jan asselijn’s ‘the threatened swan’

in developing the MosAIc algorithm, the team at MIT notes the difficulty of the endeavor, finding images that were similar not just in color or style, but in meaning and theme. the group sought ‘dogs to be close to other dogs, people to be close to other people, and so forth.’ to achieve this, they probe a deep network’s inner ‘activations’ for each image in the combined open access collections of the met and the rijksmuseum. distance between the ‘activations’ of this deep network, which are commonly called ‘features,’ was how they judged image similarity.

‘tureen with cover in the form of a turkey,’ german, ceramics (left) | ‘turkey palma,’ various, stone

‘lachenalia aloides,’ various, drawings (left) | ‘ontwerp voor een kruisbeeld,’ roman, drawings

‘balearica regulorum,’ various, drawings (left) | ‘twee javaanse danseressen,’ southeast asian, uncategorized

‘helmet (khula khud) with horns,’ south asian, weapons (left) | ‘colin’s royal stag’ british, photographs

‘the great ruby watch,’ german, accessories (left) | ‘disk ornament,’ ancient american, metalwork

an example of what MosAIc does when given one image (left) and the similar object found through CIR