Low-noise

  - Recall sources of noise from earlier lecture
      (draw signal chain)
  - For each source of noise we can reduce it by going through extra 
      effort and expense:
      - photon shot noise (principal effect on precision)
         - proportional to sqrt
         - relatively less with more photons
         - therefore accumulate more photons
            - use more silicon area (expensive)
            - collect more light (long exposures / high light levels)
            - somewhat analogous to slow film
      - dark current shot noise (contributes to DR)
         - proportional to sqrt dark current
         - therefore reduce dark current
            - short exposure (at odds with previous requirements)
            - lower temperature
               - thermoelectric cooling (Peltier effect)
               - heat removal: convection, forced air, liquid coolant
                  (in order of inconvenience)
               - cool chip below dew point => keep under vacuum or dry atm.
                  => extra optical window in system
      - fixed pattern noise
         - two flavors: gain and bias
         - discussed this before: handle with post-processing
            - dark frame subtraction to cancel bias
            - flat field calibration to cancel gain
               - lots of work => esoteric situations only
                  (telescopes, satellites)
      - readout noise (contributes to DR)
         - generally reduced at slower readout speeds
           
      - extreme quantitative example: Apogee instruments AP7
         - sensor = SITe SI-502AB: 512x512 24um pixels
         - well depth: ~350,000 e-
         - read noise: ~9 e-
         - dynamic range: 39,000:1, or 91 dB (14-15 bits)
         - dark current: 1800 e-/pixel/sec @ 20 C
            (can tolerate about 80 e- to equal readout noise)
            => dark current dominates S/N above 40 ms exposure time
         - dark current doubles every 5-7 degrees
            => can double exp time for every 5-7 (say 6) degs of cooling
            => about  -7 C for clean 1 sec exposure
            => about -27 C for clean 10 sec exposure
            => about -42 C for clean 1 min exposure

      - very different example: CoolSnap CF
         - sensor = Sony ICX205: 1392 x 1040 4.65um pixels
         - well depth: ~9700 e-
         - read noise: ~15 e-
         - dynamic range: 670:1, or 56 dB (9-10 bits)
         - dark current: 1.3 e-/pixel/sec @ 20 C
            => not a problem up to 2-3 min at 20 C


High dynamic range

  - dynamic range: highest to lowest signal observed
      (or often, highest signal to RMS noise floor)
      (example: specs for audio equipment)
  - dynamic range can be increased by increasing precision
      (analogy to adding bits to an integer)
  - can also increase DR by keeping same *relative* precision
      (analogy to floating point numbers)

  - multiple images can give FP-like data
      (think of pixel value as mantissa and exposure as exponent)

  - exposure series example illustrating high DR

  - combining exposures
     - could take top half of each image
        - analogous to radix-2 floating-point: mantissa > .5
           (otherwise the exponent would be smaller)
        - bad plan in practice though
            - discontinuities are always bad
            - throwing away useful data
     - use weighted average
        - weight depends on pixel value to select properly exposed pixels
           - Mann & Picard propose derivative of response
           - Debevec & Malik propose hat function
           - Mitsunaga & Nayar propose f(M) / f'(M)
           - I propose just the exposure time with saturation rolloff
              - if noise is additive and happened in the linear domain
                  this makes sense

  - radiometric response
     - we have to relate the numbers in the image back to sensor 
         irradiance (up to a scale factor)
       - electronic image sensors are usually quite linear, but often
           someone has introduced a mapping before we get the data
       - film-based capture is often attractive but is quite nonlinear
     - assuming the mapping from number of electrons to pixel value is
         repeatable and the same for all pixels (watch this assumption!):
       I (pixval) = f ( E (irrad) )
     - f is the "opto-electronic conversion function (OECF)" or "radiometric
         response function"
     
  - radiometric response calibration
     - using a reference
        - gray chart
           (sensitive to illumination and flare)
        - fine exposure series (unknown but stable scene)
           (requires fine exposure control)
           (requires an awful lot of images)
     - self-calibration
        - use more information from unknown scene
        - version 0: known exposure series (above)
        - version 1: get exposure series from 2 images (Mann & Picard)
           pixel A_i has irradiance I_1
           pixel B_i has irradiance k I_1
           find A_j with same pixel value as B_i (thus same irrad)
           pixel B_j has irradiance k^2 I_1
           etc.
           - this gets several samples
              (can fit simple function e.g. gamma)
           - more samples for smaller k but also more uncertainty
           - limit of small k gives the derivative...
        - version 2: align lots of exposure series (Debevec & Malik)
           - make a sparsely sampled curve from each of several pixels
           - minimal assumptions about f
           - solve for I_k that make them all line up
           - Debevec does this with linear minimization
        - version 3: unknown exposure times (Mitsunaga & Nayar)
           - formulate error: sum ( f(I_i,j) - r_j f(I_i,j+1) )^2
              - with known rs, expand f in a basis and this is
	          linear least squares.
              - with unknown rs, not so simple
                 - they represent f with a polynomial and use an iteration
                    that alternates between fixing the rs and fixing
                    the coefficients
                 - there are ambiguities so need to start with good guess
           - note that this avoids explicitly finding the irradiances

  - combining images for HDR
     - Mann & Picard's weighting
        - weight pixels proportional to f'(E)
        - argument: values more reliable in steep part of curve
     - Debevec & Malik's weighting
        - simple hat function
        - right general behavior but short on justification
     - Mitsunaga & Nayar's weighting
        - weight proportional to E F'(E)
        - argument: want to weight by S/N, not just 1/N
     - My weighting
        - just weight by exposure time (while rejecting saturated pixels)
        - the others implicitly assume noise happens after nonlinearity
        - in digital world noise (assuming sufficient quantization levels)
            happens in analog before nonlinearity)
        - weight proportional to E, but E is always in the same ratio
        - avoids bias due to value-dependent weight


High dynamic range image at one time

  - analogy to color imaging

  - SVE as analog of CFA

  - reconstruction
     - easier than CFA b/c recovering one function, not 3
        (they don't address color -- they have RGB per pixel)
     - their approach
        - define an interpolation from inter-pixel grid to pixel grid
        - for each off-grid pixel:
           - look at 4x4 neighborhood of on-grid pixels
              [16x1 vector M_r]
           - look at 7x7 neighborhood of off-grid pixels that influences M_r
              [49x1 vector M_o]
           - consider the linear transformation defined by interpolation
              [16x49 vector F : M_r = F M_o]
           - solve for the 7x7 neighborhood of off-grid pixels that,
               when interpolated, will best for the valid on-grid pixels
              [First toss invalid rows from that equation, then solve
               M~ = F~ M_o using pseudoinverse]
           - take the center pixel from that neighborhood
              [So we really just need one 1x16 row of F~]
        - and then interpolate the whole off-pixel grid to get
            the final image
     - they don't discuss implementation
        - boils down to a 4x4 filter that depends on the validity pattern
        - one could imagine a lookup table -- 16 bits but 8fold symmetry...
        - maybe a table that only includes more common patterns...


Multi-exposure depth of field enhancement

  - show Haeberli's example