That’s Not Real – Algorithm testing and validation using synthetic data

NIST Hyperspectral Image Projector

Example synthetic image of Enrique Reef, Puerto Rico using the NIST Hyperspectral Image Projector (HIP)

An important aspect of developing new algorithms or analysis techniques involves testing and validation. In remote sensing this is typically performed using an image with known characteristics, i.e. field measurements or other expert on-the-ground knowledge. However, obtaining or creating such a dataset can be challenging. As an alternative, many researchers have turned to synthetic data to address specific validation needs.

So what are the challenges behind using “real” data for validation? Let’s consider some of the common questions addressed through remote sensing, such as classifying images into categories describing the scene (e.g. forest, water, land, buildings, etc…) or identifying the presence of particular objects or materials (e.g. oil spill, active fire areas, coastal algae blooms, etc…). To validate these types of analyses, one needs knowledge of how much and where these materials are located in the given scene. While this can sometimes be discerned through experience and familiarity with the study area, in most cases this requires physically visiting the field and collecting measurements or observations of different representative points and areas throughout the scene. The resulting data is extremely useful for testing and validation, and recommended whenever feasible; however, conducting thorough field studies is not always practical, particularly when time and budget is limited.

Here we explore a few options that researchers use for creating synthetic images, from the simple to the complex:

  • A simple approach is to create an image with a grid of known values, or more specifically known spectra, where each cell in the grid represents a different material. Subsequent validation analysis can be used to confirm that a given methodology accurately categorizes each of the known materials. To add greater variability to this approach, different levels of noise can be added to the input spectra used to create the grid cells, or multiple spectra can be used to represent each of the materials. While seemingly simplistic, such grids can be useful for assessing fundamental algorithm performance.
  • The grid concept can be further extended to encompass significantly greater complexity, such as creating an image using a range of feasible parameter combinations. As an example from the field of coral reef remote sensing, a model can be used to simulate an image with various combinations of water depth, water properties, and habitat composition. If water depth is segmented into 10 discrete values, water properties are represented by 3 parameters, each with 5 discrete values, and habitat composition is depicted using just 3 categories (e.g. coral, algae and sand), this results in 3750 unique parameter combinations. Such an image can be used to test the ability of an algorithm to accurately retrieve each of these parameters under a variety of conditions.
  • To add more realism, it is also feasible to utilize a real image as the basis for creating a synthetic image. This becomes particularly important when there is a need to incorporate more realistic spatial and spectral variability in the analysis. From the field of spectral unmixing, for example, an endmember abundance map derived from a real image can be used to create a new image with a different set of endmembers. This maintains the spatial relationships present in the real image, while at the same time allowing flexibility in the spectral composition. The result is a synthetic image that can be used to test endmember extraction, spectral unmixing and other image classification techniques.
  • Another approach based on “real” imagery is the NIST Hyperspectral Image Projector (HIP), which is used to project realistic hyperspectral scenes for testing sensor performance. In other words, the HIP is used to generate and display synthetic images derived from real images. As with the above example, a real image is first decomposed into a set of representative endmember spectra and abundances. The HIP then uses a combination of spectral and spatial engines to project these same spectra and endmembers, thereby replicating the original scene. The intent here is not necessarily to use the synthetic data to validate image processing techniques, but rather to test sensor performance by differentiating environmental effects from sensor effects.

Even though it’s a powerful tool, keep in mind that synthetic data won’t solve all your validation needs. You still need to demonstrate that your algorithm works in the “real world”, so it’s best to also incorporate actual measured data in your analysis.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s