Siddharth Shashi: Project 1 - Colorizing the Prokudin-Gorskii photo collection

Description

The goal of this assignment is to take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible. In order to do this, the three color channel images needed to be extracted, placed on top of each other, and aligned so that they form a single RGB color image.

Core Approach

First, the image is split into three sections of equal height by splicing the image array. Then, using alignment functions, the red and green channels are compared against the blue channel to compute ideal alignment offsets. Finally, the channels are shifted according to their offsets, and then stacked on top of each other to create a single image. Note: large visual artifacts along the borders of images made effective alignment computations challenging, so I compared only the centers of the images to compute offsets.

Single-scale Alignment

For images with low resolutions, I implemented a search algorithm that would compute the norm between the two channel image vectors at each displacement within [-15, 15], and then return the offset of the displacement that yielded the least norm.

cathedral_aligned — Green: (-1, 1), Red: (-1, 7)

monastery_aligned — Green: (2, -3), Red: (2, 3)

tobolsk_aligned — Green: (2, 3), Red: (3, 6)

Multi-scale Pyramid Alignment

For high-resolution images, exhaustively searching would be slow and computationally expensive, so I implemented a pyramid search optimization. First, images were reduced in resolution until they were less than 500px tall, and then an exhaustive search was performed on every offset from [-15, 15]. Next, until the image was back to full resolution, the channels were doubled in resolution and then exaustively searched in just the [-2, 2] window, but centered at the previously optimal offset. When run on my laptop, this would take at most 30 seconds to finish.

train_pyramid_aligned — Green: (4, 43), Red: (31, 89)

three_generations_aligned — Green: (14, 52), Red: (11, 111)

Edge Detection Alignment

Though effective for some images, the pyramid alignment didn't work as well as I had wanted because high brightness values between channels made pixel intensities inaccurate to compare. To solve this, I used a Sobel edge detection filter in both the x and y directions for each channel, and then calculated the magnitude of the edges. I then phase correlated the edges between the images to find the optimal shift to align the edges of the channels.

With emir.tif in particular, there edge detection offers a very noticable improvement from pyramid alignment as can be seen below:

emir_pyramid_aligned — Pyramid: Green: (47, 96), Red: (-142, 0)

emir_aligned — Edge Detection: Green: (23, 49), Red: (40, 106)

Here are the other photos with edge alignment:

harvesters_aligned — Green: (17, 59), Red: (14, 123)

church_aligned — Green: (4, 24), Red: (-4, 58)

lady_aligned — Green: (9, 55), Red: (12, 119)

melons_aligned — Green: (10, 80), Red: (13, 176)

onion_church_aligned — Green: (28, 50), Red: (35, 107)

sculpture_aligned — Green: (-11, 33), Red: (-27, 139)

self_portrait_aligned — Green: (28, 77), Red: (36, 175)

Automatic Contrasting

Finally, I thought it would be interesting to automatically contrast the image by rescaling the pixel intensities. I did this simply by rescaling image intensities such that the darkest pixel is zero on its darkest color channel and the brightest pixel is 1 on its brightest color channel. Here's an example of the difference contrasting adds: