Introduction
This page describes a case study of cloud computing applied to the Ocean Observing Intiative (OOI) in the broader context of managing oceanographic data. We are following the work of Friedrich Knuth in visualization and work by Vaughn Iverson on cruise data harmonization problem.
CC*IIE Remarks
Objective and Approach
Solution
Results
Links
Starting material provided by Friedrich Knuth
HD Video Data
Tim Crone at LDEO is working with Aaron Marburg at APL. Tim has set up a platform to provide co-located processing and storage of HD video data. Transfer to cloud would require a cost evaluation; and likewise with broadband hydrophone data.
OOI M2M with real-time data visualization
…and other inspirational material provided by Friedrich Knuth…
IPython notebook, documentation pending. The notebooks gets at the basic data and the metadata request functionality. Bear in mind that real-time requests are called ‘synchronous’ and come back as JSON with a maximum of 1000 data values; whereas ‘asynchronous’ requests produce .nc files off a THREDDS server. In both cases we are seeing raw data processed on-request to a data product.
The real-time plotting function is built on top of the above notebook.
from concurrent.futures import ThreadPoolExecutor
is the key to sending continuous requests that re-initiate after each prior request is completed. For more on this see the OOI Data Team portal.
To go further: D3 is highly favored at this time (elegant plotting/visualization; plus open source). D3 was developed by Mike Bostock who formerly worked in Jeff Heer’s lab, now at UW. See also this idl link. To continue in this vein we mention further that D3 data interpolation examples can be tied to smart data decimation; see this link associated with Simen Brekken.
Another item: The FK et al paper on OOI HD video data. This work illustrates derived scientific data products and automated QA/QC (Quality Assurance / Quality Control) methods. This QA/QC code would ideally be run on the cloud; and would have the ability to work through the entire video archive. See this pdf.
Visualization inspiration: A beautifully illustrated data story written in D3.