Published

OGC Other

•
CF-netCDF Core and Extensions Primer
Ben Domenico Editor
Submission Date: 2011-04-05 Approval Date: 2011-04-05 Publication Date: 2011-04-05 External identifier of this OGC® document: http://www.opengis.net/doc/IS/netcdf/Primer/1.0 Internal identifier of this OGC® document: 10-091r3
Version: 1.0
OGC Other

Published

Notice

This document is not an OGC Standard. This document is an OGC Other and is therefore not an official position of the OGC membership. It is distributed for review and comment. It is subject to change without notice and may not be referred to as an OGC Standard.

Further, an OGC Other should not be referenced as required or mandatory technology in procurements.

Document number:10-091r3
Document type:OGC Other
Document subtype:
Document stage:Published
Document language:English

Copyright notice

Copyright © 2011 Open Geospatial Consortium
To obtain additional rights of use, visit http://www.ogc.org/legal/

Note

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.


License Agreement

Permission is hereby granted by the Open Geospatial Consortium, (“Licensor”), free of charge and subject to the terms set forth below, to any person obtaining a copy of this Intellectual Property and any associated documentation, to deal in the Intellectual Property without restriction (except as set forth below), including without limitation the rights to implement, use, copy, modify, merge, publish, distribute, and/or sublicense copies of the Intellectual Property, and to permit persons to whom the Intellectual Property is furnished to do so, provided that all copyright notices on the intellectual property are retained intact and that each person to whom the Intellectual Property is furnished agrees to the terms of this Agreement.

If you modify the Intellectual Property, all copies of the modified Intellectual Property must include, in addition to the above copyright notice, a notice that the Intellectual Property includes modifications that have not been approved or adopted by LICENSOR.

THIS LICENSE IS A COPYRIGHT LICENSE ONLY, AND DOES NOT CONVEY ANY RIGHTS UNDER ANY PATENTS THAT MAY BE IN FORCE ANYWHERE IN THE WORLD. THE INTELLECTUAL PROPERTY IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE DO NOT WARRANT THAT THE FUNCTIONS CONTAINED IN THE INTELLECTUAL PROPERTY WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION OF THE INTELLECTUAL PROPERTY WILL BE UNINTERRUPTED OR ERROR FREE. ANY USE OF THE INTELLECTUAL PROPERTY SHALL BE MADE ENTIRELY AT THE USER’S OWN RISK. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR ANY CONTRIBUTOR OF INTELLECTUAL PROPERTY RIGHTS TO THE INTELLECTUAL PROPERTY BE LIABLE FOR ANY CLAIM, OR ANY DIRECT, SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM ANY ALLEGED INFRINGEMENT OR ANY LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR UNDER ANY OTHER LEGAL THEORY, ARISING OUT OF OR IN CONNECTION WITH THE IMPLEMENTATION, USE, COMMERCIALIZATION OR PERFORMANCE OF THIS INTELLECTUAL PROPERTY.

This license is effective until terminated. You may terminate it at any time by destroying the Intellectual Property together with all copies in any form. The license will also terminate if you fail to comply with any term or condition of this Agreement. Except as provided in the following sentence, no such termination of this license shall require the termination of any third party end-user sublicense to the Intellectual Property which is in force as of the date of notice of such termination. In addition, should the Intellectual Property, or the operation of the Intellectual Property, infringe, or in LICENSOR’s sole opinion be likely to infringe, any patent, copyright, trademark or other right of a third party, you agree that LICENSOR, in its sole discretion, may terminate this license without any compensation or liability to you, your licensees or any other party. You agree upon termination of any kind to destroy or cause to be destroyed the Intellectual Property together with all copies in any form, whether held by you or by any third party.

Except as contained in this notice, the name of LICENSOR or of any other holder of a copyright in all or part of the Intellectual Property shall not be used in advertising or otherwise to promote the sale, use or other dealings in this Intellectual Property without prior written authorization of LICENSOR or such copyright holder. LICENSOR is and shall at all times be the sole entity that may authorize you or any third party to use certification marks, trademarks or other special designations to indicate compliance with any LICENSOR standards or specifications. This Agreement is governed by the laws of the Commonwealth of Massachusetts. The application to this Agreement of the United Nations Convention on Contracts for the International Sale of Goods is hereby expressly excluded. In the event any provision of this Agreement shall be deemed unenforceable, void or invalid, such provision shall be modified so as to make it valid and enforceable, and as so modified the entire Agreement shall remain in full force and effect. No decision, action or inaction by LICENSOR shall be construed to be a waiver of any rights or remedies available to it.

None of the Intellectual Property or underlying information or technology may be downloaded or otherwise exported or reexported in violation of U.S. export laws and regulations. In addition, you are responsible for complying with any local laws in your jurisdiction which may impact your right to import, export or use the Intellectual Property, and you represent that you have complied with any regulations or registration procedures required by applicable law to make this license enforceable.



I.  Abstract

This OGC primer provides an overview of the OGC CF-netCDF standards suite by describing the CF-netCDF core and extensions. The CF-netCDF standard defines how to encode digital geospatial information representing space/time-varying phenomena

II.  Keywords

The following are keywords to be used by search engines and document catalogues.

ogcdoc, netcdf, cf-netcdf


III.  Preface

The intended target audience is developers that wish to implement CF-netCDF encoding for servers and/or clients. This document provides an overview of the many possible components of the CF-netCDF suite and how those components fit together into a coherent whole. This document also provides useful hints and best practices beyond the pure standards texts. In the era of modular standards, such primer documents are especially important so that potential users can get a sense of the overall landscape.

As such, the content of this document is informative and not normative.

IV.  Security considerations

No security considerations have been made for this document.

V.  Submitting Organizations

The following organizations submitted this Document to the Open Geospatial Consortium (OGC):

VI.  Document Contributor Contact Points

Name Organization
Ben Domenico UCAR Unidata

VII.  Changes to the OpenGIS® Abstract Specification

The OpenGIS® Abstract Specification does not require any changes to accommodate the technical contents of this document.

VIII.  Future Work

This document needs to be updated whenever a new extension is added to CF-netCDF. It may need an update if and when significant functionality changes are made to existing CFnetCDF components.

IX.  Foreword

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium Inc. shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

X.  Introduction

The CF-netCDF standard defines how to encode digital geospatial information representing space/time-varying phenomena.

CF-netCDF consists of a set of normative specifications, collectively referred to as “the CFnetCDF suite”. These specifications encompass:

This document provides an overview on the CF-netCDF encoding suite by describing CF-netCDF core and extensions. As such, the content of this document is informative and not normative.

CF-netCDF Core and Extensions Primer

1.  Scope

Scope of this document is to provide a primer on the use of the CF-netCDF suite of standards.

2.  Conformance

This Primer document does not contain any normative statements, hence no compliance is defined. In case there are any deviations between the standards mentioned and this document the standards texts shall prevail.

3.  Normative references

There are no normative references in this document.

4.  Terms and definitions

No terms and definitions are listed in this document.

This document uses the specification terms defined in Subclause 5.3 of OGC 06-121r3, which is based on the ISO/IEC Directives, Part 2, Rules for the structure and drafting of International Standards. In particular, the word “shall” (not “must”) is the verb form used to indicate a requirement to be strictly followed to conform to this standard.

5.  Conventions

5.1.  UML notation

All the diagrams that appear in this specification are presented using the Unified Modeling Language (UML) static structure diagram, as described in Subclause 5.2 of OGC Web Services Common OGC 06-121r3.

5.2.  Namespace prefix conventions

None defined here.

6.  CF-netCDF Package Overview

6.1.  Core and extensions

Based on the CF-netCDF model defined in this document and the NetCDF Classic core data model specification, many extension packages are conceivable which:

  • add specific functionality to the NetCDF classic data model and encoding, or

  • constrain aspects of the core NetCDF classic model and encoding.

The list presented below contains existing, planned, and possible CF-netCDF extensions. It makes use of a grouping which appears reasonable at the time of this writing; however, this structuring is by no means normative and shall not be used to draw any conclusions on the functionality a particular specification provides.

For illustration purposes, some possible extensions are listed in Clause 7.

NetCDF Classic Core and each extension specify, as normative requirements, which prerequisite specifications they require. Frequently, options are possible in some specific group of extensions; for example, every netCDF implementation must support at least one encoding extension.

Figure 1 — CF-netCDF specification hierarchy graphical overview

This constitutes a dependency graph as shown in Figure 1, where the dark green boxes represent the initial targets for standardization and light green are the next targets.

6.2.  Application Profiles

An Application Profile (AP) trims or constrains CF-netCDF functionality for some particular purpose. To this end, two mechanisms are available:

  • The AP can make a choice among the extensions it declares mandatory (note that the core has to be included in any case).
    Note that there are cross dependencies among some extensions which have to be respected. For example, the core requires at least one encoding extension, and the CF Conventions extension requires a Data Sampling Type.
    As an example, a Satellite Imagery AP might require a Swath Data Type whereas Forecast Model Output AP requires the Gridded Data Type.

  • Additional restrictions can be imposed on core and extensions.
    In the above example, certain high resolution Model Output APs might required the 64-bit offset binary encoding.

Consequently, a conformance test for an AP will in turn inspect the NetCDF Classic Core,any extension listed, and finally the specific requirements of the AP in question.

7.  Possible extensions

In this section, several CF-netCDF extensions are listed which have been discussed by the CF-netCDF Working Group in the course of developing the CF-netCDF specification. Some of them are already available as adopted or proposed specifications by other groups (NASA and NOAA in particular), but nearly all of them are available as part of the netCDF packages which have been implemented and used widely for decades. The CF conventions are in fact a separate entity, but they too evolved as part of the netCDF community of practice and were initially implemented for use with netCDF. By no means is this list prescriptive or comprehensive; some extensions listed here may never be written, and others not listed at this time may be added and developed later.

Following the presentation in Clause 6, extensions are grouped into abstract classes:

These abstract classes will not be standardized and implemented as such, but concrete extensions within the classes will be implemented and standardized.

7.1.  Data model extensions

7.1.1.  Purpose

This category of extensions focuses on adding information content to the netCDF dataset model.

7.1.2.  Identification

The netCDF core specification and each CF-netCDF extension is identified by a unique HTTP URI which, by convention, is specified in the first formal requirement of an extension specification.

This primer document is identified by OGC URI http://www.opengis.net/doc/primer/cfnetcdf/1.0.

Currently, the CF-netCDF suite of (proposed) standards includes:

The NetCDF Core Specification. OGC Document 10-091r3. This specification is identified by OGC URI http://www.opengis.net/spec/netcdf/1.0. The document has OGC URI http://www.opengis.net/doc/IS/netcdf/1.0.

The NetCDF Classic and 64-bit Binary Encoding Extension. The specification is identified by OGC URI http://www.opengis.net/spec/netcdf-binary/1.0. The document has OGC URL http://www.opengis.net/doc/IS/netcdf-binary/1.0.

7.1.3.  List of extensions

7.1.3.1.  Enhanced netCDF data model

The netCDF classic data model has been extended by an enhanced model that includes “groups,” an expanded list of primitive data types, and a provision for user defined data types.

http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#Data-Model

7.1.3.1.1.  Other data model extensions

Other extensions to the core data model may be added later.

7.2.  Conventions extensions

7.2.1.  Purpose

This category of extensions describes conventions that may constrain the general netCDF data model. Many netCDF conventions have been defined for use in a diverse set of disciplines. Among them are:

  • CF Conventions (Recommended, if applicable)

  • COARDS Conventions (1995 standard that CF Conventions extends and generalizes)

  • GDT Conventions (1999 standard that CF Conventions extends and generalizes)

  • CDC Conventions (for gridded data, compatible with but more restrictive than COARDS)

  • NCAR-RAF Conventions for Aircraft Data

  • AMBER Trajectory Conventions for molecular dynamics simulations

  • ARGO netCDF conventions for data centers

  • National Oceanographic Data Center NetCDF Conventions

  • NUWG Conventions (1992-1995 effort to create some observational data conventions)

7.2.2.  List of extensions

While it is important to be aware of the fact that many netCDF conventions exist, the initial focus of the CF-netCDF SWG is on the conventions for Climate and Forecast (CF) metadata.

7.2.2.1.  CF conventions

The conventions for climate and forecast (CF) metadata are designed to promote the processing and sharing of files created with the NetCDF API. The conventions define metadata that provide a definitive description of what the data in each variable represents, and the spatial and temporal properties of the data. This enables users of data from different sources to decide which quantities are comparable, and facilitates building applications with powerful extraction, regridding, and display capabilities.

CF Conventions
http://www.cfconventions.org

Proposed NASA CF Standard:
http://www.esdswg.org/spg/rfc/esds-rfc-021

7.2.2.2.  Other conventions extensions

Other conventions extensions may be added later.

7.3.  Binary encoding extensions

7.3.1.  Purpose

This category of extension describes the structure of the encoded datasets.

7.3.2.  List of extensions

NetCDF encoding formats are defined in format encoding extensions. Some of these encodings are binary and are to be specified in extensions to the netCDF core.

Unidata netCDF encoding documentation
http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#Classic-File-Parts

7.3.2.1.  NetCDF classic and 64-bit offset binary encoding formats

In particular, the first extension is to be the NetCDF Classic and 64-bit Offset File Formats which have been previously adopted as a NASA Standard:

NASA Standard: NetCDF Classic and 64-bit Offset File Formats
http://www.esdswg.org/spg/rfc/esds-rfc-011/ESDS-RFC-011v1.00.pdf

7.3.2.2.  HDF 5 binary encoding format

The HDF 5 encoding format is used in conjunction with the netCDF enhanced data model. It is no doubt appropriate to have the general HDF 5 encoding format defined by another standards group and only define the constraints on its use in conjunction with the netCDF data model defined here.

HDF 5 Encoding for netCDF Enhanced Data Model
http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#NetCDF_002d4-File-Parts

7.3.2.3.  Other binary encodings

Other binary encoding extensions may be added later.

7.4.  XML encoding extensions

7.4.1.  Purpose

XML encoding extensions can serve several purposes. The entire contents of a netCDF dataset can be encoded in dialects of XML. But XML dialects can also be used to augment the metadata associated with a binary encoded netCDF dataset. It can also be used to define virtual datasets that consist of aggregations of data that exist in multiple netCDF binary files.

7.4.2.  List of extensions

This open-ended list is likely to encompass at least ncML and ncML-GML. It remains to be seen whether and how CSML fits with netCDF encoding.

7.4.3.  NcML

NcML Documentation
http://www.unidata.ucar.edu/software/netcdf/ncml/

NcML is an XML representation of netCDF metadata, (approximately) the header information one gets from a netCDF file with the “ncdump -h” command. NcML is similar to the netCDF CDL (network Common data form Description Language), except, of course, it uses XML syntax. The simplest use of NcML is to describe the metadata and structural content of a netCDF file. A more advanced use is to modify existing NetCDF files, as well as to create “virtual” NetCDF datasets, for example through aggregation.

7.4.3.1.  NcML-GML

NcML-GML is:

  • an Abstract and Content Model reconciliation schema for ES and GIS info realms

  • a Mediation Markup Language between ncML (netCDF Markup Language) and GML

  • an extension of ncML core schema, based on GML grammar

At the moment, to support some legacy software packages, ncML-GML is not a standard GML profile. This will be fixed in a future release.

NcML-GML Documentation
http://zeus.pin.unifi.it/joomla/index.php?option=com_content=view=50=78

7.5.  Text encoding

7.5.1.  Purpose

It is often useful to represent the contents of a netCDF file or just the header metadata information in a simple, east for humans to read, text form.

7.5.2.  List of extensions

7.5.2.1.  CDL (network Common data form Description Language)

CDL (network Common data form Description Language) is a tiny language that makes it possible to represent either the metadata or the entire contents of a netCDF dataset in an intuitive, easily understandable textual form.

CDL syntax is described at:
http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#CDL-Syntax

7.6.  Application Programming Interfaces (APIs)

7.6.1.  Purpose

The powerful but relatively simple netCDF APIs are often cited at a primary reason for the wide adoption and usage of netCDF. For that reason, they are seen as a facilitator of interoperability of data systems within the community. Because of that, there have been suggestions that establishing the most commonly used APIs as a standard would greatly benefit interoperability in a wider community. Hence, they are included here as future possibilities, but are not the initial focus of the CF-netCDF SWG.

7.6.2.  List of extensions

7.6.2.1.  C language

The netCDF C Interface Guide:
http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c.html#Top

7.6.2.2.  Fortran

The netCDF Fortran 77 Interface Guide:
http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-f77.html#Top

The netCDF Fortran 90 Interface Guide:
http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-f90.html#Top

7.6.2.3.  C++

The netCDF C++ Interface Guide:
http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-cxx.html#Top

7.6.2.4.  Java

The netCDF Java Library:
http://www.unidata.ucar.edu/software/netcdf-java/

7.6.2.5.  Other APIs

Other Applications Programming Interface extensions may be added later


Annex A
(informative)
Revision history

Date Release Author Paragraph modified Description
2010-08-27 1.0.0 Ben Domenico All Created
2010-10-20 1.0.1 Bruce Wright and Ben Domenico 7.3.2, 7.3.2.1, 7.3.2.2 Edited for clarity
2011-01-05 1.0.1 Ben Domenico All Changed r2 to r3 in document numbers
2011-01-22 1.0.1 Carl Reed Various Prepare for publication
2011-02-16 1.0.1 Ben Domenico Various Prepare for publication

Bibliography

[1]  Clemens Portele: OGC 07-036, OpenGIS Geography Markup Language (GML) Encoding Standard. Open Geospatial Consortium (2007). https://portal.ogc.org/files/?artifact id=20509.

[2]  Ben Domenico: OGC 10-092r3, NetCDF Binary Encoding Extension Standard: NetCDF Classic and 64-bit Offset Format. Open Geospatial Consortium (2011). https://portal.ogc.org/files/?artifact id=43734.

[3]  Arliss Whiteside: OGC 06-121r3, OpenGIS Web Service Common Implementation Specification. Open Geospatial Consortium (2007). https://portal.ogc.org/files/?artifact id=20040.

[4]  Ben Domenico: OGC 10-090r3, OGC Network Common Data Form (NetCDF) Core Encoding Standard version 1.0. Open Geospatial Consortium (2011). https://portal.ogc.org/files/?artifact id=43732.

[5]  Rew R., Heimbigner D., Davis E., Caron J.: NASA ESDS-RFC-011v2.00, NetCDF Classic and 64-bit Offset File Formats. http://www.esdswg.org/spg/rfc/esds-rfc-011/ESDS-RFC-011v2.00.pdf.

[6]  Unidata UCAR, NetCDF Reference Document, 2009, http://www.unidata.ucar.edu/software/netcdf/docs/

[7]  Unidata UCAR, NetCDF User Guide, http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html

[8]  Unidata UCAR, NetCDF Reference Implementations, ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf.tar.gz

[9]  CF Conventions, http://www.cfconventions.org

[10]  Proposed NASA CF Standard, http://www.esdswg.org/spg/rfc/esds-rfc-021

[11]  NASA Standard: NetCDF Classic and 64-bit Offset File Formats, http://www.esdswg.org/spg/rfc/esds-rfc-011/ESDS-RFC-011v1.00.pdf

[12]  Unidata netCDF encoding documentation, http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#Classic-File-Parts

[13]  HDF 5 Encoding for netCDF Enhanced Data Model, http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#NetCDF_002d4-File-Parts

[14]  NcML Documentation, http://www.unidata.ucar.edu/software/netcdf/ncml/

[15]  NcML-GML Documentation, http://zeus.pin.unifi.it/joomla/index.php?option=com_content=view=50=78

[16]  CDL Syntax Description, http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#CDL-Syntax

[17]  The netCDF C Interface Guide, http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c.html#Top

[18]  The netCDF Fortran 77 Interface Guide, http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-f77.html#Top

[19]  The netCDF Fortran 90 Interface Guide, http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-f90.html#Top

[20]  The netCDF C++ Interface Guide, http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-cxx.html#Top

[21]  The netCDF Java Library, http://www.unidata.ucar.edu/software/netcdf-java/