This is part of a series on tips for getting the most out of your geospatial applications. Check back regularly or follow HySpeed Computing to see the latest examples and demonstrations.
Objective: Open a HICO dataset stored in HDF5 format and transform the data into standard ENVI formatted files ready for further processing and analysis.
(Update 14-June-2014): A companion post is now available describing how to open HDF5 HICO scenes using an IDL application prepared by the U.S. Naval Research Laboratory.
Scenario: This tip demonstrates two methods for opening a HDF5 formatted HICO scene in ENVI: (i) one method uses the new HDF5 tool released in ENVI 5.1 and (ii) the other method uses the H5_BROWSER routine in IDL. Subsequent steps are then used to save the data in standard ENVI format and prepare the resulting data for further analysis.
- HICO – Hyperspectral Imager for the Coastal Ocean – is an imaging spectrometer located on the International Space Station. Visit the HICO website at Oregon State University for detailed information on data characteristics and tips for working with this data.
- HDF5 – Hierarchical Data Format v5 – is a file format used to organize and store large multifaceted data collections. Visit the HDF Group for more information.
Why two methods? Not only is it informative to describe both methods, but since the new HDF5 tool in ENVI is generic, it does not always support access to all data files within a given dataset. Hence, for users who wish to remain within the ENVI/IDL system, it is often necessary to leverage the H5_BROWSER routine to open all available data in a particular HDF5 scene.
(Update 5-Mar-2014): Exelis VIS has released ENVI 5.1 Hotfix 1, which addresses the issue of being able to read all data files within a given HICO dataset. The new HDF5 tool is now fully functional for the entire dataset. See below for details.
Data: HICO data is currently available from two different online sources: (i) as ENVI files through the HICO website, which requires users to submit a brief proposal to become a registered HICO Data User, or (ii) as HDF5 files through the NASA GSFC Ocean Color website, which requires users to register with NASA’s EOSDIS User Registration Service.
This example utilizes a HICO scene of the Florida Keys (H2011314130732.L1B_ISS) downloaded from the NASA GSFC Ocean Color website.
The HDF5 formatted HICO scenes contain six core datasets: (i) products (L1B top of atmosphere radiance), (ii) images (three-band true color image), (iii) navigation (latitudes, longitudes, sensor azimuth and zenith, and solar azimuth and zenith), (iv) quality (scan-line quality flags), (v) data (L0 uncorrected data), and (vi) metadata (FGDC and HICO metadata information). Note: In more recent files the L0 uncorrected data is not included and the HDF5 files therefore only contain five core datasets.
Tip 1 – HDF5 Tool in ENVI 5.1: Below are steps to open the navigation and quality datasets in ENVI using the new HDF5 tool:
- Start by unpacking the compressed folder (e.g., H2011314130732.L1B_ISS.bz2). If other software isn’t readily available, a good option is to download 7-zip for free from http://www.7-zip.org/.
- Rename the resulting HDF5 file with a *.h5 extension (e.g., H2011314130732.L1B_ISS.h5). This allows the HDF5 tools to more easily recognize the appropriate format.
- Open the ENVI generic HDF5 reader by selecting File > Open As > Generic Formats > HDF5, and then select the desired *.h5 file in the file selection dialog.
- The left column of the HDF5 reader lists the Available Datasets and the right column lists the Raster Builder. The right column is where you will create and populate the two HICO datasets to open in ENVI.
- First, for the quality dataset, rename the default raster in the right column of the Raster Builder (e.g., rename Raster 1 to H2011314130732_L1B_ISS_quality). This can be easily accomplished by right clicking on the raster name and selecting Rename Raster. Next, add the quality flags to this raster by clicking on the <flags (512×2000)> dataset in the left column and then clicking the arrow in the center of the HDF5 tool to add this data to the recently created raster.
- Now add a second raster to the right column using the New Raster button at the bottom of the right column (it’s the rightmost icon with the small green plus sign). Rename this raster (e.g., rename Raster 2 to H2011314130732_L1B_ISS_navigation), and use the center selection arrow to add all six data layers from the navigation dataset in the left column to the new raster in the right column.
- Note that if you will be opening data from more than one HICO scene, then you can also build a Template from these settings and use the Template as the basis to open more datasets.
- Your HDF5 dialog window should look similar to the following:
- Now select Open Rasters at the bottom of the dialog window to open both of the rasters as data layers in ENVI.
- These data layers can then be saved in ENVI format by selecting File > Save As…, selecting a given data layer in the file selection dialog, and assigning an Output Filename in the Save File As Parameters dialog.
(Update 5-Mar-2014): Follow the same steps as above to open and save the L1B product using the HDF5 reader. However, for the three band color image, note that the HDF5 metadata itself misidentifies this data as band interleaved by pixel (BIP) when in fact it is band sequential (BSQ). To save this data in ENVI format, first open and save the data using the same steps as above, and then use Edit ENVI Header to adjust the band interleave of the resulting ENVI file from BIP to BSQ.
Tip 2 – H5_VIEWER in IDL: Below are steps to open the L1B product in ENVI using the IDL H5_BROWSER tool:
- As evident in the current example, the left column of the generic HDF5 reader does not indicate any data is present within the products data layer. Hence, we use an alternative approach here to access the L1B product data. You can also use this same approach to open other datasets within the HDF5 file.
- This tip requires both ENVI and IDL, so to get started you will need to make sure you have launched ENVI+IDL rather than just ENVI. By launching ENVI+IDL you will start both the standard ENVI interface and the IDL Workbench.
- In the IDL Workbench, within the IDL command-line console, enter the following commands to start the HDF5 Browser (substituting your own filename as appropriate):
- The HDF5 Browser window should now appear, where the left column of the window lists the available datasets (now with data listed under the products dataset) and the right column shows details about any given selected dataset.
- Within the H5 Browser, select the Lt data in the products dataset (as shown above), and then select Open.
- Returning now to the IDL command-line console, enter the following lines (again substituting your own filename as appropriate) to save the data to disk in ENVI format and write an associated ENVI header file.
- Alternatively, you can also use the following code to create your own simple IDL program. To do so, simply start a new program by selecting File > New File in the IDL Workbench, copy and paste text from this file (hy_open_hico.pdf) into the new program, save the program with the same name as listed in the code (i.e., hy_open_hico), replace the hardcoded filenames with your own filenames, compile the code, and then run.
Data Preparation: Below are the final steps needed to prepare the HICO data for further processing:
- As noted on the HICO website: “At some point during the transit and installation of HICO, the sensor physically shifted relative to the viewing slit. The edge of the viewing slit was visible in every scene.” This effect is removed by simply cropping out affected pixels in each of the above data files (i.e., quality, navigation and products). For scenes in standard forward orientation (+XVV), cropping includes 10 pixels on the left of the scene and 2 pixels on the right. Conversely, for scenes in reverse orientation (-XVV), cropping is 10 pixels on the right and 2 on the left.
- If you’re not sure about the orientation of a particular scene, the orientation is specified in the HDF5 metadata under Metadata > HICO > Calibration > hico_iss_orientation.
- Spatial cropping can be performed by selecting Raster Management > Resize Data in the ENVI toolbox, choosing the relevant input file, selecting the option for Spatial Subset, cropping the image for Samples 11-510 for forward orientation (3-502 for reverse orientation), and assigning a new output filename. Repeat as needed for each dataset.
- The HDF5 formatted HICO scenes also require spectral cropping to reduce the total wavelengths from 128 bands to a 87 band subset from 400-900 nm. The bands outside this subset are considered less accurate and typically not included in analysis.
- Spectral cropping can also be performed by selecting Raster Management > Resize Data in the ENVI toolbox, only in this case using the option to Spectral Subset and selecting bands 10-96 (corresponding to 404.08-896.69 nm) while excluding bands 1-9 and 97-128. This step need only be applied to the hyperspectral L1B product data.
- The header file for the L1B product data should now be updated to include information for the wavelengths (87 bands from 404.08-896.69 nm), fwhm (10 nm for 400-745 nm; and 20 nm for 746-900 nm) and gain values (0.020 for all bands).
- Editing the header can be accomplished using the option for Raster Management > Edit ENVI Header in the ENVI toolbox, or by directly editing the actual header file using a standard text editor. For simplicity, as illustrated below, you can cut and paste the relevant header info from the following file (example_header.pdf) into the header.
- The HICO scene is now ready for further processing and analysis in ENVI.
- As an example, the latitude and longitude bands in the navigation file can be used to build a GLT to geolocate the scene using the rough coordinates provided with the data distribution. The HICO website provides a step-by-step overview of this geolocation process.
For more information on the sensor, detailed data characteristics, ongoing research projects, publications and presentations, and much, much more, HICO users are encouraged to visit the HICO website at Oregon State University. This is an excellent resource for all things HICO.