“Klimt vs. Klimt – The Man of Contradictions,” a new interactive hub from Google Arts & Culture, will provide visitors with information on Gustav Klimt’s biography, artistic inspiration, legacy, and more. It will also showcase scholarly articles written by experts from Austria’s top institutions, such as the Belvedere and the Wien Museum.
Gustav Klimt created some of the world’s most valuable works of art, yet around 20% of his work has gone missing. During the aftermath of WWII, the Nazis chose to burn down the Immendorf castle in Lower Austria, which housed rare antiques, because they did not want its valuable artwork to fall into the hands of the Russian army. Three masterpieces by Austrian painter Gustav Klimt were also lost in the fire. The three paintings, Medicine, Jurisprudence, and Philosophy, are known as the Faculty Paintings. There are just black and white photographs of the artworks left. Although the original artworks are unlikely to be seen again, machine learning has come close to resurrecting them. Google Arts & Culture collaborated with the Belvedere Museum in Vienna to reproduce the artworks in full color using machine learning techniques.
In addition, Google used artificial intelligence technology to colorize black-and-white photos of three of Klimt’s lost works from 1899 as part of its effort. Using “Pocket Gallery” they put some of his most renowned pieces into one’s living room in 3D and augmented reality.
The cutting-edge technology was trained on a data collection of Klimt’s paintings and was able to restore the ‘Faculty Paintings’ to its former brilliance. Once the color data was gathered, Emil Wallner, a Google Arts & Culture Lab resident, used an algorithm to recreate the Faculty Paintings using Dr. Smola’s study. Rather than coloring the paintings by hand, Wallner’s algorithm does a statistical study of Klimt’s previous works and learns how to imitate his colorization technique. From real-world photos, the algorithm extracted a feeling of skin tones and sky colors, as well as a sense of composition, object boundaries, and textures from paintings.
To train the algorithm on how to colorize the Faculty Paintings, the team at Google Arts & Culture Lab used 80 pictures of Klimt’s colorful artworks. When they simply utilized these photos to train the Pix2pix algorithm, the model learned about Klimt’s color palette, but not enough about the settings in the paintings to colorize them coherently. Then they attempted the DeOldify algorithm, which involves training the model on one million images of real-world objects such as people, animals, and buildings. In this case, the model provided more consistent colorization and closely resembled the real environment. This model, however, has no comprehension of art or Klimt’s colorizing technique.
Read More: Intertwined Intelligences: Introducing India’s First AI NFT Art Exhibition
Next, they employed guided colorization with human-made color annotations. A user adds a handful of color dots to a black and white painting in this approach, which tells the algorithm how to colorize it. The machine learning model recognizes textures and objects, then propagates color suggestions to nearby areas.
Finally, by integrating these techniques, the researchers created a unique model. The model uses a U-net with a pre-trained ResNet-34 with self-attention, spectral normalization, and a 3-channel RGB input with color hints. It shares a similar structure with DeOldify and is progressively trained with a custom feature loss from a pre-trained GAN critic.
Overall, 91749 artworks from Google Arts & Culture were used to train the new algorithm. This enabled the machine learning model to recognize object borders, textures, and common artwork compositions. It unifies colorization and learns how to adapt to different colorization techniques from tens of thousands of artists.
It was then trained on Klimt’s colorful paintings as a last step to develop a colorization bias towards Klimt’s color themes. The AI-colored pictures that resulted may be the closest we’ll ever get to viewing a complete image of those long-lost masterpieces.
