(more projects added January 12)
Most of these ideas projects require a programmable camera. That is, after all, the idea behind this course. For this purpose, each of you has been loaned a Stanford-Nokia Franken-N900 for the duration of the course. The "Hello, camera" assignment will give you some experience programming this camera.
Alternatively, we have three Frankencameras hanging around the lab - in various states of health and nakedness. You're welcome to use one of these prototypes for your project, but it may require some patience. We also have a menagerie of Canon and Nikon SLRs and point-and-shoot cameras. The SLRs are programmable using SDKs from the manufacturer, but the metering, focusing, and post-processing pipelines remain black boxes. The Canon point-and-shoot cameras can be partly programmed using the Canon Hack Development Kit (CHDK), but you're on your own here. (Actually, Natasha Gelfand (ngelfand@gmail.com) of Nokia can help you.) Still other projects might require you to be handy with a soldering iron. We'll pay for any hardware you need to buy, but talk to us before you go on a buying spree.
Finally, if you want clarification of a project idea, or you have an idea we haven't listed, come talk to us! In general, Marc and Fredo are the best people to talk to to figure out if your idea is novel, plausible given the time frame, and interesting, and Jongmin (or one of our other students, as noted below) is the best person to talk to find out if your idea is possible given the hardware and programming constraints of the devices. Don't consider the list below as your only options. Any of us would love to brainstorm with you about your ideas.
Finally, many of these project ideas could be Siggraph, Eurographics, or CVPR/ICCV/ECCV papers. (In fact, two student projects from the 2004 version of this course did become SIGGRAPH papers.) To keep you and your fellow students from being scooped, please don't distribute this handout, or the URL of this web page, or its ideas, beyond the Stanford community.
- Metering for HDR imaging
High dynamic range imaging consists of capturing images with different exposures, merging them together, and tone mapping the result to compress its range for display. Although HDRI has a long history, nobody has addressed the question of automatically deciding which exposures to capture. Of course, nobody has had a programmable camera with access to the light meter. The Frankencamera and Franken-N900 don't have light meters, but they have fast access to the main sensor and plenty of computational power. Using one of these platforms, explore metering for HDR imaging. What technique works best?
- Motion-sensitive low-noise imaging
Two years ago we developed an algorithm for aligning successive viewfinder frames on a cell phone in real time [Adams 08]. One application we explored was aligning and averaging multiple frames to reduce noise in low-light environments, assuming you held the camera reasonably stationary. However, this technique fails for moving objects; they become blurry. Can you extend the technique to detect motion, then locally reduce the number of frames that are combined in these regions? This is essentially photography (or videography) with adapative per-pixel exposure time. As such, it's related to [Bennett 05].
- Painted aperture, variant #1
The images from multiple side-by-side cameras can be combined to simulate a single camera with a larger aperture and hence a shorter depth of field [Vaish 05]. If the depth of field is short enough, objects lying off the in-focus plane become so blurry that they effectively disappear. This allows you to look through foreground occluders such as bushes [Vaish 04]. It should also be possible to move a single small camera through a range of positions, and 'paint' a synthetic aperture. The resulting images should be aligned to each other on the background and combined in such a way that removes foreground occluders. Think carefully about sampling pattern and density. Using this technique to take photographs of sunbathing celebrities through dense foliage is optional.
- Painted aperture, variant #2
In portrait photography, large apertures are useful to blur out background detail, leaving only the person's face in focus. Using the technique described in the previous project idea, but aligning on the foreground instead of the background, make large aperture portrait photos possible on cameras with small apertures, like the N900. For extra fun on this or the previous project, implement a touchscreen user interface for choosing the object on which to align, and hence where to focus. Making either project run interactively on a camera will be challenging. We know, because we've tried. Talk to Abe Davis, abedavis@stanford.edu.
- Moment Camera
A moment camera stores a circular buffer of recent frames over the past n seconds. When the user presses the shutter, they can select as their output any frame(s) from that n-second window. The Casio EX-F1 has this capability. Implement a moment camera on the Frankencamera or Franken-N900, using (low-resolution) viewfinder frames for your circular buffer. Alternatively, use high-resolution photographs taken by an SLR in non-real time. Instead of simply selecting the best frames, try combining them in creative ways [Agarwala 04, Cohen 06]. It would be great to combine a low-resolution burst of frames with a single high-resolution frame captured at the instant the user presses the trigger, but our current Frankencameras can't change resolutions fast enough.
- Applications of approximate depth maps
By capturing images at two focus settings (or a sequence of settings) and testing the resulting images for sharpness, one can estimate distance to the scene at each pixel. In computer vision this is called depth-from-defocus or depth-from-focus, respectively. Although these depth maps are inaccurate, they might be useful in photography. For example, if you estimated a depth map during aiming, could you use it to capture an all-focus image by moving the lens only to depths containing interesting features? Or to simulate a tilted focal plane? Or to help you extract a matte (see the "Flash matting" project)?
- User interfaces for controlling camera settings
The settings of digital cameras (aperture, shutter, ISO, focus, white balance, etc.) are not very intuitive. When people look at a photograph they think instead of brightness, contrast, noise (or grain), depth of field (features they want to be sharp versus blurry), motion blur, color balance, etc. Using the touch screen on the Frankencamera or Franken-N900, or using the Frankencamera and sliders or dials added using its Phidgets subsystem, explore alternative ways to control camera settings. This problem is more challenging (and interesting) than it sounds. For example, depth of field depends on the lens, sensor, zoom and focus settings, so work these variables into your algorithms. Similarly, our notion of brightness depends on absolute scene luminance and our state of visual adaptation (see Reinhard's book on HDR imaging for relevant papers); can these variables be worked in? For the user interface, consider capturing bursts of images with alternative settings and displaying them as thumbnail images in a grid on the viewfinder, using some variant on "Design Galleries" [Marks 97].
- Adjusting a camera based on your photo collection
An increasingly common theme in computational photography is clever ways to leverage existing collections of images when performing image editing. One example that might be of interest to the user of cell phones is improving a photograph, or adjusting a camera before you capture a photograph, by looking at other shots of that same object, animal, or person among a user's existing images - on the device or in the cloud. Examples of improvements are adjusting white balance or exposure, removing camera shake, increasing resolution, or removing shadows or occlusions. Even the images a user recently deleted from their camera might provide helpful information about the user's preferences! Try implementing one or more of these ideas. Depending on your approach, algorithms from machine learning might be appropriate.
- Foveal imaging
The Frankencamera allows us to alternate between reading the entire sensor at reduced resolution and reading out a small region of interest at full sensor resolution. The goal is to use this interleaved pair of video streams to synthesize a single full-rate stream while reading back fewer pixels per second. If the region of interest is centered in the field of view, this should produce a video with greater sharpness and smoother motion in the center. Alternatively, the region of interest could be placed over parts of the scene that are well-exposed, in-focus, or that change quickly from frame to frame. Or it could be scanned slowly across the scene. Challenges include deciding where to place the ROI and knowing how to process the imagery afterwards. See [Bhat 07] and [Gupta 09] for inspiration. One of our PhD students is working on this problem, but it's hard. Care to try your hand at it? Talk to Sung Hee Park, shpark0707@gmail.com.
- Re-photography
Re-photography means guiding the user with images and arrows to the viewpoint of a reference historical photograph, so they can take a matching picture. Fredo Durand has implemented this technique at MIT using an SLR tethered to a laptop, but this arrangement is awkward. We've tried porting his technique to the Franken-N900, but it's challenging to make it run fast enough. Some of the image processing techniques Andrew Adams covered in CS 448F might be useful. Another challenge is removing foreground objects that weren't present in the historical photograph, for example by translating the camera and combining multiple photographs [Agarwala 04]. Care to help? Talk to PhD student Jongmin Baek, baekjm@stanford.edu.
- Computer-assisted lucky imaging
The term comes from a related technique in astronomy. In photography, the muscle tremors that cause handshake are cyclical. If you shoot a burst of frames, some of them will fall into the minima of these cycles and be relatively free of handshake. PhD student Jen Dolson, jdolson@stanford.edu has bolted an inertial measurement unit (IMU) to an N900 (it already has one, but of limited quality), and she's using it to detect these lucky moments, saving only those frames captured when handshake was minimal. We're wondering whether it would be possible to perform this selection more accurately (although probably not faster) using image-based motion estimation. One challenge is that image-based techniques will also detect (and confuse) handshake with object motion. Perhaps a hybrid of IMU and image-based techniques could solve this confusion. As an extension, could you use this information to set the shutter speed for the next frame - maximizing exposure while avoiding motion blur?
- Removing handshake using deconvolution
Handshake can be modeled as convolution of an unknown (sharp) scene by an unknown blur kernel (representing the shaking of your hands) to produce the recorded (blurry) image. In theory one can remove handshake using deconvolution [Levin 09], either blind - without knowing the blur kernel, or non-blind - for example using an IMU to estimate camera motion. Jen has investigated the IMU approach, but calibrating the device and synchronizing it with a camera have proven challenging. One idea we haven't tried yet is blind deconvolution using a burst of images, since they share the same unknown sharp scene. Using the Frankencamera (or Franken-N900), these images can be captured very rapidly. Since cameras have mass, the handshake in one frame might "connect" to the handshake in the next, providing another constraint for deconvolution.
- Real-time deconvolution of ROIs
Deconvolution to remove handshake is slow. None of the published algorithms are anywhere near real-time, even on a desktop. But there's a strong motivation for removing handshake in a camera; it tells you if you need to take another shot. How about removing handshake from only a few regions of interest, like the center and corners (to cover the possibility of in-plane rotation), and presenting these results in the viewfinder. Could this be done in a few seconds? Using which algorithm(s)? How about a progressive refinement scenario, where the result improves as you watch it, but you can give up at any time and just take another shot.
- Flash matting
A fun photographic game is to replace the background in a picture. This requires extracting a matte that specifies which pixels in the picture are foreground and which are background. One technique for performing this so-called matte extraction is to record a flash-noflash image pair, then look for pixels that change the most. If you then record a replacement background picture using the same camera, you can perform the replacement right on the camera. Implement this pipeline in your Franken-N900, starting with the flash matting paper [Sun 06].
- Borrowed flash
Pictures taken using on-camera flash look awful. If red-eye doesn't ruin your shot, those deer-in-the-headlights specular highlights will. Web sites like Strobist tout the virtues of moving the flash away from the camera, but this requires buying (and carrying around) a detachable flash unit, which are only available for SLRs. For cell phones (or future point-and-shoot cameras with radios), why not borrow a second camera from a friend, slave it to your phone using Wifi or Bluetooth, hold it in your other hand, and program it to flash when your camera takes a picture. We've never tried this, but it should work on the Franken-N900s. How about key and fill flash, using two extra N900s? We have some ideas on synchronize multiple cameras to a global clock; come talk to us.
- Flash strips
Many companies sell strips of LEDs that can be controlled from a computer. Can these be used as flash units? If the LEDs are arranged in a line or cluster, this softens the hard shadows that plague conventional flash photography. LEDs are also available in color; that's easier than taping gels to your flash unit. If you shot a collection of pictures with different LEDs turned on, can you mix them on the viewfinder? Or adjust their levels interactively on the viewfinder in preparation for a "final shot" using those levels? How should could one incorporate multiple LED flashes into a camera body? Could one design a camera body with a pull-out spool of LEDs? This project will obviously involve some hardware hacking. We have a laser cutter in the lab for prototyping camera bodies.
- Shaped flash using pico-projectors
Flash units offer little control over the angular distribution of illumination. Video projectors offer such control, and they're getting smaller, brighter, and cheaper, but aside from one unpublished tech report [Ziegler 08] nobody has explored their use in photography. Future cell phones may incorporate pico-projectors; could these be used as flash units? For example, if a pilot snapshot taken by the camera finds a person's face centered in the field of view, it could reshape the illumination to avoid overexposing their skin, or it could carve two holes in the illumination pattern to prevent red-eye. If you're in a small room, by controlling the shape and color of light bouncing off each wall maybe you can create a virtual lighting studio! For this project we could either buy pico-projectors (if the lead time isn't too long), or you could use a full-size projector (we'll loan you one) lashed to a camera. You might be able to drive the projector from our Frankencamera(s); otherwise, use a laptop.
- Artistic effects using iris and focus
Sharp features that are out of focus have a certain look, sometimes called bokeh, which depends on the shape of the camera's iris (aperture). By manipulating the iris quickly during an exposure, you can change the bokeh for artistic effects. By manipulating the focus during an exposure, you can create other effects. A special case is moving the focus smoothly from front to back during the exposure, sometimes called focus sweep [Nagahara 08]. Design an interactive interface for experimenting with these effects - one that is both expressive and easy to control. Which effects work well? Since the Franken-N900 doesn't have an iris, you'll have to do this project on the Frankencamera. Both cameras have control over focus. However, neither camera gives you control over speed at which the iris or focus moves, so you'll have to be clever about how you use them.
- Ad-hoc stereo photography
3D photographs and movies are all the rage. One shouldn't need a specialized camera rig to capture them. Using Wifi or Bluetooth, try synchronizing two Franken-N900s to capture a stereo picture or video. After alignment, display the pair of videos in cross-eyed or red-green stereo or using shutter glasses (we have a pair), or use computer vision algorithms to estimate 3D shape. What camera separations work best for different kinds of scenes? Or borrow everyone's N900 and capture a synchronized bullet-time video.
- Viewfinder visualizations
The small LCDs at the back of cameras don't make it possible to judge what is in focus. Especially when playing with depth of field effects or a tilt-shift lens, it would be very useful to have a real-time visualization of what is in focus in the image, for example with a heat map. Alternatively, straight lines are important elements for composition. A photographer might want them to stay horizontal or vertical, or to avoid converging parallels (for architectural photography in particular). Real-time analysis and visualization on the viewinder could help a photographer achieve these goals. For more ideas about visualizing focus, see this thought-provoking Open Letter to Leica by Luminous Landscape's Michael Reichmann.
- Computational metering
Exposure metering still follows techniques inherited from the film era. Digital, however, has different constraints and parameters. For example, digital photographers often seek to "expose to the right" and get the intensity histogram as far right as possible so as to increase SNR but avoid saturation. A convenient interface for this would allow the user to simply decide which percentile gets mapped to the maximum sensor value. Michael Reichmann's open letter to Leica, cited in the previous project, has interesting things to say about this issue as well. Another difference between analog and digital photography is that ISO can be changed easily with digital. The "P" program in cameras seeks to automatically chose the shutter speed / aperture pair with simple and fixed rules. It would be desirable for photographers to be able to specify their own program based on their shooting habits (e.g., I want a shutter speed above 1 over focal length with an aperture of f/2 or above, and an ISO that does not go over 1600. If this isn't possible, give up on aperture first, then ISO, then shutter speed). One solution would be to have a general programming interface where the user writes rules for their exposure program. But maybe a visual interface is also possible (drawing curves of shutter speed, aperture and ISO as a function of light level). This problem can then be generalized to flash photography, where both the background and flash exposure must be set.
- Photomontage helper
Merging multiple shots that contain the same person or object in a different pose in each shot is fun (see the "multi-Fredo" image at the top of the course home web page), but they require planning. The task would be much easier if the camera could pre-visualize the montage while framing the shots. For example, if one wants to make a picture of a person boxing himself or herself, the placement would be greatly facilitated if the LCD displayed the current view superimposed on the first one. Quality need not be perfect and simple background subtraction can probably be used. For more ideas, see http://photoextremist.com/2009/11/how-to-clone-yourself-using-multiplicty-photos/. Similarly, real-time visualization can help the creation of time lapse movies such as http://photoextremist.com/2009/09/captin/, by the same guy, by creating visualizations that help the alignment when taking a new shot. One can also take multiple shots and automatically chose the one that best matches the previous frames. Other montages and video effects can probably be helped by real-time visualization. Be creative!
- Lighting
The borrowed flash project can be generalized to borrowed light. If rooms become "intelligent" and have a networked lighting control system, the camera should be able to get access to these controls and modify the scene lighting for the brief amount of time where a picture is taken. The same idea applies to professional studio lighting. Studio lighting is a complex task, in particular for portraits. A computational camera could control a number of light sources to achieve a given lighting. Can you set up a room full of studio lights and control them from the camera? We have lots of this sort of equipment lying around. Controlling these lights from a camera may involve some hardware hacking. For an additional challenge, the user could provide a picture as a template, and the camera would seek to match the light distribution.
As long as this list is, don't consider these your only options. In fact, another long list, which overlaps only partially with this list, can be found on the web site of the Spring 2008 version of this course. If you have an idea for another project, or a variation on one of these ideas, come talk to us; we'd love to brainstorm with you about it!