NEON EDUCATION bio photo

NEON EDUCATION

Devoted to open data and open source in science and education.

View All Tutorials

Tags

LiDAR (9)
R programming (70)
Remote Sensing (12)
Data Visualization (4)
Hyperspectral Remote Sensing (7)
Hierarchical Data Formats (HDF5) (24)
Spatial Data & GIS (18)
Time Series (15)
Phenology (7)
Raster Data (8)
Vector Data (6)
Metadata (1)
Git & GitHub (6)
(1) (1) (1)

Tutorial by R Package

dplyr (8)
ggplot2 (17)
h5py (1)
lubridate (time series) (6)
maps (1)
maptools (3)
plyr (2)
raster (32)
rasterVis (raster time series) (3)
rgdal (GIS) (23)
rgeos (5)
rhdf5 (21)
sp (7)
scales (4)
gridExtra (4)
ggtheme (0)
grid (2)
reshape2 (3)
plotly (6)

View ALL Tutorial Series




Twitter Youtube Github


Blog.Roll

R Bloggers
Date(s):

Explore the Operational NEON HDF5 Format

Date: 11 May 2015

This NEON internal brownbag introduces the concept of Hierarchical Data Formats in the context of developing the NEON HDF5 operational file format. Look here to discover resources on HDF5, code snippets in R, Python and Matlab to use H5 files and some example H5 files for Remote Sensing Hyperspectral data and time series temperature data.

Background Materials

Please review, download and setup the following, prior to attending the brownbag.

Data to Download

Download NEON Teaching Data Subset: Sample Tower Temperature - HDF5
These temperature data were collected by the National Ecological Observatory Network's Flux Towers . The entire dataset can be accessed from the NEON website.
Download NEON Teaching Data Subset: Imaging Spectrometer Data - HDF5
These hyperspectral remote sensing data provide information on the National Ecological Observatory Network's San Joaquin Experimental Range field site. The data were collected over the San Joaquin field site located in California (Domain 17) and processed at NEON headquarters. The entire dataset can be accessed from the NEON website. </ul>

Download the Free H5 Viewer

The free H5 viewer will allow you to explore H5 data, using a graphic interface.

More on the viewer here

Please review the following:

SCHEDULE

Time Topic Instructor
12:00 Hand-on exploration of the HDF5 Data Format  
12:20 Working with HDF5 in Python - live demo.  
~12:30 NEON HDf5 Format - what’s next  

Useful HDF5 Resources

Python resources for HDF5:

  1. H5 Python Documentation
  2. O’Reilly book on Python and HDF5! The modern stamp of legitimacy for programming.
  3. Python examples from the HDF5 people themselves!

Python Code to Open HDF5 files

The code below is starter code to create an H5 file in Python.

if __name__ == '__main__':
	#import required libraries
	import h5py as h5
	import numpy as np
	import matplotlib.pyplot as plt

	# Read H5 file
	f = h5.File("NEON-DS-Imaging-Spectrometer-Data.h5", "r")
	# Get and print list of datasets within the H5 file
	datasetNames = [n for n in f.keys()]
	for n in datasetNames:
		print(n)
	
	#extract reflectance data from the H5 file
	reflectance = f['Reflectance']
	#extract one pixel from the data
	reflectanceData = reflectance[:,49,392]
	reflectanceData = reflectanceData.astype(float)

	#divide the data by the scale factor
	#note: this information would be accessed from the metadata
	scaleFactor = 10000.0
	reflectanceData /= scaleFactor
	wavelength = f['wavelength']
	wavelengthData = wavelength[:]
	#transpose the data so wavelength values are in one column
	wavelengthData = np.reshape(wavelengthData, 426)

	# Print the attributes (metadata):
	print("Data Description : ", reflectance.attrs['Description'])
	print("Data dimensions : ", reflectance.shape, reflectance.attrs['DIMENSION_LABELS'])
	#print a list of attributes in the H5 file
	for n in reflectance.attrs:
	print(n)
	#close the h5 file
	f.close()

	# Plot
	plt.plot(wavelengthData, reflectanceData)
	plt.title("Vegetation Spectra")
	plt.ylabel('Reflectance')
	plt.ylim((0,1))
	plt.xlabel('Wavelength [$\mu m$]')
	plt.show()
    
	# Write a new HDF file containing this spectrum
	f = h5.File("VegetationSpectra.h5", "w")
	rdata = f.create_dataset("VegetationSpectra", data=reflectanceData)
	attrs = rdata.attrs
	attrs.create("Wavelengths", data=wavelengthData)
	f.close()