Making Environment Maps
from Fisheye Photographs

Ipix has recently been enforcing its acquired patents relating to conversion of fisheye images to perspective images. However, before this time, there was widespread use of fisheye images, and dewarping these to perspective images by optical and image processing techniques has been practiced and published for decades.

There is a lot of current interest in the use of fisheye images to increase the vertical field of view of photographically-acquired environment maps in real-time environment renderers, such as QuickTime VR, offered by Apple Computer, Inc. It is the intent of this article to review and apply the technology developed prior to the relevant ipix patents to the authoring and rendering of environment maps acquired with fisheye lenses.

Of particular relevance is a pair of papers, widely published 5 years before ipix's earliest patent, that deal with the geometry and projection of fisheye images in the context of interactive three-dimensional computer graphics.

In 1986, Ned Greene published the seminal paper on environment mapping, Environment Mapping and Other Applications of World Projections, in IEEE Computer Graphics and Applications, November 1986, vol. 6, no. 11, pp. 21-29. In this paper he describes the generation of such environment maps both synthetically and captured with a fisheye lens. He further uses these environment maps in two ways within a 3D graphics system: (1) to render an infinitely far away backdrop, and (2) to render realistic reflections off chrome-like objects closer to the viewer.

With Greene's preferred representation of the environment map as a polyhedron, it was possible to use a special-purpose 3D computer, such as those manufactured by Silicon Graphics or Evans and Sutherland, to display polygons, texture-mapped with the environment map, at interactive rates. These computers used a variety of input devices, such as a joystick or a trackball, to control the view (i.e. pan, tilt, roll, and field-of-view). Greene espouses the use of cubic environment maps over the more obvious latitude-longitude spherical parametrization found in fisheye projections, because of their better fit with the capabilities of these widely available 3D graphics workstations.

The speed of computers has advanced (100-fold) to the point today where much of this interactive 3D viewing can be done on personal computers. Witness the largest demographic group of 3D users: computer game players. Modern 3D computer games are richly adorned with texture-mapped surfaces making up a space that the player navigates throughout.

VRML (Virtual Reality Modeling Language) was developed around 1994 as a means to share 3D environments, targeted primarily at the world-wide web. Its specification includes several environmental primitives, including a surrounding cube. The cube is texture-mapped, and provides a backdrop behind a scene composed otherwise of polyhedral objects. This cubic environment map is the one advocated by Greene in his 1986 paper.

In 1993, Apple Computer pioneered the use of cylindrical environment maps in an interactive setting with QuickTimeVR. Within the subsequent years perhaps a dozen other companies joined suit, providing an immersive experience to users by interactively panning and zooming around cylindrical, cubic, and spherical (both in polar or fisheye parametrizations, and Cartesian, equirectangular or Mercator-like projections) environment maps.

Fisheye lenses have traditionally been used to capture environments. This type of lens was first developed to allow astronomers to capture the entire hemisphere of the night sky with one photograph. Illumination engineers have used fisheye lenses to study the canopy provided by leafy trees. Photographers have used fisheye lenses to capture panoramic shots that could not otherwise be captured with a rectilinear lens.

Telerobotics International, Inc., in 1991 developed and patented the Omniview Motionless Camera System, in which a video camera with a fisheye lens was augmented with some special-purpose image processing hardware to project the fisheye-parametrized spherical image onto a plane, in perspective. The technology used here is similar to that described by Greene in his paper 5 years earlier, except that it makes use of a live video feed and specialized rendering hardware to generate perspective views of a dynamic scene in real-time.

In 1993, Telerobotics specialized this fisheye projection system to allow remote surveillance through the use of a fisheye lens in their Omniview Motionless Camera Surveillance System. The attractive thing about this invention, compared with other surveillance systems, is that it an entire hemisphere could be monitored without any moving parts. Prior surveillance systems were built with a standard video camera and lens mounted on a motor that panned back and forth throughout the area. Smart burglars could defeat these surveillance systems by simply staying in the blind zone of these moving cameras. The fisheye lens allowed the monitoring of the entire area without moving. Furthermore, Omniview developed some special purpose hardware to project the distorted fisheye image to a plane, resulting in the familiar perspective kind of view.

Also in 1993, Telerobotics developed the Omniview Motionless Camera Endoscopy System for minimally-invasive inspection of cavities deep within the human body without major surgery.

Several years later, Telerobotics transferred the rights to these patent to ipix. Since that point, ipix has been aggressively applying for additional patents regarding the use of fisheye lenses in interactive environment maps. During the same period, ipix has been threatening anyone that performs digital image warping of fisheye images with telephone calls, e-mail, postal mail, and subpoenas from their lawyers. This aggressive, threatening behavior has caused several immersive imaging companies to go out of business, to withdraw products, to close their web sites, and the like. Additional historical information can be found at the International QuickTime VR Association and at Kathy Wheeler's web site.

This behavior of ipix has stifled innovation in the immersive imaging industry, and has upset the community of immersive content creators enough to boycott the use of ipix's products.

One of the reasons regards field-of-view. All lenses have some amounts of distortion, but especially those with a shorter focal length. It is these shorter focal length lenses that have the wider fields of view. Algorithms for correcting this distortion have been around for some time (refs here). At a certain point, around a 15 mm lens, it is just too hard to compensate for the distortion in the lens design, so lenses are instead designed with a predictable, analytical distortion: the fisheye lens.

Ordinary rectilinear lenses can be thought of as imaging to a planar photosensitive surfaces. Fisheye lenses, on the other hand, should be though of as imaging to a spherical surface; this spherical surface is then flattened to record it on planar film. Think of a tennis ball: first cut the ball into two hemispherical pieces, then take one of these hemispheres and squash it down into the table with the heel of your hand.

It is impossible for a rectilinear image to encompass a field of view of 180°, because it would require an infinitely large imaging surface. This is easy for a fisheye lens, however. In fact 180° is the most common field of view found in fisheye lenses. Nikkor does make a 220° lens though. The focal length of a 220° lens for a 35 mm camera is about 6 mm, and that for a 180° lens is about 8 mm. This dimension is the radius of the virtual imaging sphere for fisheye lenses. The circumference of this sphere determines the field of view recorded on the 24 mm X 36 mm imaging area of a standard 35 mm camera film frame. There is no reason other than manufacturing capability why there isn't a 360° fisheye lens. It is, however, relatively easy to construct an optical system to capture maybe 359° in a fisheye image: merely photograph a silver mirrored Christmas tree ornament with a telephoto lens.

In Ned Greene's seminal environment paper, he mentions the use of a fisheye image as the texture for an environmental map. He shows a picture of 5 polygons texture-mapped using a fisheye image as a source texture, arranged in a hemicube formation, to illustrate the fisheye's ability to enclose space. This is exactly the same kind of projection described by Telerobotics (ipix) in their subsequent 1993 patent, the first in a series of patents relating to the use of fisheye lenses.

A companion paper, actually published several months before, described the geometry of the fisheye lens, as applied to synthesizing Omnimax movies.

In [Greene86jun], Ned Greene gives the formulas for converting the fish-eye coordinates into spherical coordinates:

A 180° fish-eye lens (round type) projects half of the environment into a circular image. … The limiting circle of the 180° fish-eye projection is assigned a unit radius, and its center is chosen as the image origin. … Points in the image are assigned polar coordinates (r, theta), and are converted to spherical coordinates with angular coordinates theta and phi, where theta is longitude and phi is the angle from the axis of projection. The transformation from polar to spherical coordinates keeps theta the same and transforms r into phi. The expression for phi as a function of r, adapted from Max [Max83dec], is phi(r) = 1.411269 r - 0.094389 r3 + 0.25674 r5

We show this radial distortion function in magenta in the graph below. The ideal fisheye, with

phi_ideal(r) = pi/2 r

is shown in green for comparison.

Personally, I prefer to use coordinates of the image that correspond to the ideal fisheye. Instead of the fisheye radius extending from 0 to 1, I instead prefer it to go from 0 to pi/2 radians, and allow arbitrary distortion correction from an ideal fish-eye image.

In [Greene86jun], Greene shows the part of the fisheye image that is used in an OmniMax frame:

and how the portions of the hemicube map to the fisheye image.