Super-resolution (also spelled as super resolution and superresolution) is a
term for a set of methods of upscaling video or images.
Terms such as "upscale", "upsize", "up-convert" and "uprez" also describe increase of resolution in either image
processing or video editing. Most super-resolution techniques are based on the same idea: using information from several different
images to create one upsized image. Algorithms try to extract details from every image in a sequence to reconstruct other
frames. This multiframe approach differs significantly from sophisticated image (single frame) upsizing methods which try to
synthesize artificial details.
"Super-resolution" is not a marketing buzzword, it's a mathematical term
used by scientists. First work on this topic was published in 1984
and the term "Super-resolution" itself appeared at around 1990
Methods usually discussed in scientific literature try to reproduce
process of losing quality when shooting video with low-res cameras and then
solve inverse problem of finding video which being downsized with that process
gives us known low-res material. This is an ill-posed inverse problem which
doesn't have straightforward solution and usually requires some additional
regularization (applying some artificial constraints) and huge CPU time
to check an awful lot of variants. Modern practical methods are usually
simpler but still effective.
Super-resolution (SR) works effectively when several low resolution images contain slightly different perspectives of the same
object. Then total information about the object exceeds information from any single frame. The best case is when an object
moves in the video. Motion detection and tracking are then employed to benefit upscaling. If an object doesn't move at all and is
identical in all frames, no extra information can be collected. If it moves or transforms too fast then it looks very different in
different frames and it's too hard to use information from one frame in reconstructing the other.
Super-resolution is used now in two tasks: extracting single frames from video (or a set of images/photos) with high quality, or upsizing the whole
video. SR methods are usually based on two important algorithms: high quality spatial (in-frame) upscaling, and motion
compensation for finding corresponding areas in neighbor frames. Many practical implementations of super-resolution software
upscale original material two times. If we need to upsize it four times, we usually apply SR twice (this can be done internally in
Let's consider a real example of upsizing video 4 times with super-resolution algorithm.
This example was made using Video Enhancer.
First, we have original low-resolution sequential frames of video (or consider them a sequence of slightly different low quality photos):
The same frames zoomed 4 times:
Step 1. Upsize all frames 2 times using good quality interpolation, in this
case Lanczos3 method (note: our current implementation doesn't use Lancsoz anymore, it uses a similar method but with a different kernel):
Step 2. Using sub-pixel accurate motion compensation, find similar areas in neighbor
frames and intelligently merge frames to combine information (and in some
cases reduce noise):
Now we have our video upsized 2 times using super-resolution. We repeat this
process to get 4x enlargement.
Step 3. Spatial interpolation again:
Finally, Step 4. Combine neighbor frames to get more details using motion compensation:
Now let's compare it with what we originally had:
Quite amazing, isn't it? But it's no magic, only a bit of math. ;)
One may ask: was that motion compensated merging really required? Weren't
spatial interpolated frames good enough? The answer will become obvious if
you look at
comparison of different upsize methods. Shortly, spatial interpolation
methods don't even come closer to a good super-resolution in terms of details
and objective metrics.
Note that super-resolution cannot always provide excellent results.
Read in the Infognition knowledge base when
super-resolution doesn't work.
1. R. Y. Tsai and T. S. Huang, "Multiframe image restoration and
registration," in Advances in Computer Vision and Image Processing,
vol. 1, chapter 7, pp. 317-339, JAI Press, Greenwich,
Conn, USA, 1984.
2. M. Irani and S. Peleg. 1991, "Super Resolution From Image Sequences"
ICPR, 2:115--120, June 1990.
Try this yourself with Video Enhancer
Get Super Resolution plugin for VirtualDub
Get Super Resolution plugin for Adobe After Effects and Premier Pro
Get Super Resolution plugin for AviSynth
OFX plugin coming soon.