New version of neonUtilities (v1.3.7)

neonUtilities 1.3.7 is now available on CRAN! Updates in this version are primarily to keep current with updates in package dependencies.

Changes in this version include:

  • Fixes stackByTable() bugs related to data.table package updates
  • Download by time interval now uses timeIndex= rather than avg= in zipsByProduct()

If you have any issues or questions about this release, please visit our issues log on GitHub.

Resolved: Notice of incorrect soil CO2 concentration data (DP1.00095.001)

This is a resolution to the following past news posts:

A fix was developed for the issue of missing atmospheric pressure corrections for the soil CO2 concentration data product (DP1.00095.001). All data have now been republished and are available on the NEON Data Portal.

Sample IDs updated in sediment data products

Sample IDs have been updated in Sediment data products. They had been incorrect, with station IDs that did not match the named location. Approximately 200 sample IDs were affected, across most sites and through several years. The sample IDs were updated across all data tables.

The affected products include:

  • Sediment Chemical Properties (DP1.20194.001)
  • Sediment Physical Properties (DP1.20197.001)

The sample IDs are primarily used for linking across the various tables, and that linkage has always been preserved; when they were incorrect, they were still consistent across the tables. Therefore the only issue for the data user was if they were extracting the station ID from the character string of the sample ID rather than from the named location field. All data downloaded after March 2020 have the correct sample IDs which match the named locations.

Please use the data portal contact form for further inquiries please select DP1.20194.001 or DP1.20197.001 in the Data Product dropdown list.

Water Chemistry external lab data transformed from wide to long format

All external lab data has been transformed from wide format to long format for the water chemistry data products. The wide format table (externalLabData_pub) has been deprecated, and replaced with this table: externalLabDataByAnalyte_pub.

The affected products include:

  • Chemical properties of Surface Water (DP1.20093.001)
  • Chemical Properties of Groundwater (DP1.20092.001)

All previous data have been transferred to the new table as of April 2020. There should be no further updates.

Please use the data portal contact form for further inquiries please select DP1.20093.001 or DP1.20092.001 in the Data Product dropdown list.

Quality Assurance updates and fixes to 2017 AOP data

The sites flown in 2017 did not have a uniform QAQC document available for the L3 tiles – a data quality measure implemented at the start of the 2018 flight season. This quality check revealed that the L3 Albedo tiles did not include a 'No Data' value and that several sites had a flight line in which the weather metadata was missing from the HDF5 file, which prevented weather information from being propagated to downstream QA processes. The sites missing with weather metadata missing from a flight line were BART, NOGP, WOOD, and JORN. The weather data was updated using the ASO-provided flight logs and the Albedo tiles had the appropriate 'No Data' value assigned. The L3 mosaiced tiles were recreated for the sites that were affected by missing weather data in case the previously omitted line was flown in better weather conditions. Once these corrections had been made, the QAQC document was generated for each site flown in the 2017 flight season. As of August 19, 2020, these data have been corrected and available for download on the Data Portal and through NEON's API.

Please use the data portal contact form for further inquiries and include the term “INC0037086“ in the body of your request

NEON aquatic dissolved oxygen sensors percent saturation normalized to sea level pressure

NEON dissolved oxygen (DO) sensors are calibrated using local atmospheric pressure.  However, the DO % saturation currently reported in the water quality data product (DP1.20288.001) have been normalized to sea level pressure (760 mmHg) by the YSI EXO2 multiparameter sonde.  NEON has begun the process of reconfiguring the multisondes to also report DO % saturation relative to local atmospheric pressure, which will be published as a new parameter in the water quality data product.  In the interim, NEON has developed the localPressureDO R-package that uses barometric pressure from the streamside meteorological station to post-correct historical data and can be accessed in the NEONscience/NEON-water-quality GitHub repo. The DO concentration in mg/L data stream is not impacted by the % saturation reporting settings, and is correct. 

Please use the data portal contact form for further inquires: please select DP1.20288.001 in the Data Product dropdown list and include the term “INC0040000“ in the body of your request

Bundling terrestrial biogeochemistry data products

In order to improve usability of the Observational System terrestrial biogeochemistry data products, we are repackaging several data products to deliver all of the relevant data for sample sets in one download. For example, end users will no longer need to download DP1.10067.001, DP1.10102.001 and DP1.10099.001 to access root biomass standing stocks, root chemistry, and root stable isotopes (respectively), but will instead be able to get all of these data from downloading just DP1.10067.001. The same will be true for leaf litter, canopy foliage, and soil samples. Once all plant and soil data products have been updated, we will update this news item. End users may need to update code routines for pulling data from the portal, but we hope this improves the overall user experience for NEON terrestrial biogeochemistry data products.

Please use the data portal contact form for further inquiries and include the term “RITM0025882“ in the body of your request

New version of neonUtilities (v1.3.6)

neonUtilities 1.3.6 is now available on CRAN! Updating is recommended. Changes in this version include:

  • Fixes stackByTable() bugs that caused errors when stacking large numbers of site by month files
  • sensor_positions files in new format stack correctly
  • zipsByURI() is compatible with fastq files
  • stackByTable() stacks lab-specific files as well as site-specific files

If you have any issues or questions about this release, please visit our issues log on GitHub.

Rodent-borne Pathogen Data Product Scope Change

NEON recently obtained approval to begin testing small mammal blood and ear tissue samples for tick-borne pathogens rather than hantaviruses. This change will be reflected in the data product beginning in the 2020 field season.

The details of this adjustment in scope are the result of extensive dialogue with NEON users and advisory groups stemming from user requests to improve the utility of this data product. This will shift the focus to a set of emerging pathogens that are more abundant and widespread than hantaviruses, and that link the tick vector and small mammal datasets. This change will also improve the public health relevance, capacity to detect continental- and decadal-scale patterns, and overall data utility. Blood samples collected in excess of our capacity to test will continue to be archived at the NEON Biorepository and can be requested by individual researchers.

The 2020 sampling year will consist of a pilot project in which ear and blood samples from 5 sites will be tested for a range of tick-borne pathogens. The list of pathogens is likely to include: Borrelia burgdorferi, Borrelia miyamotoi, Borrelia mayonii, Anaplasma phagocytophilum, Rickettsia ricketsii, Rickettsia parkeri, Francisella tularensis, and Babesia microti. The list of sites included in the pilot study will depend in part on the number of samples available given fluctuating field sampling efforts due to the pandemic; however the preference is to test samples from HARV, SCBI, TREE, TALL and SJER. Pending results from this pilot study, full implementation of tick-borne disease testing in small mammals is anticipated beginning in the 2021 field season. Moving forward, (from the 2020 field season onwards) all rodent pathogen testing will be for tick-borne diseases, and no additional hantavirus tests will be conducted by NEON.

Please use the data portal contact form for further inquiries and include the term “INC0039430“ in the body of your request

Timestamp mismatch between the Picarro G2131 and L2130 instruments

We have recently discovered multiple instances where timestamps became out of sync between the Picarro G2131 and L2130 instruments and computers, and what was assigned to the data points upon transmission to NEON Headquarters. This has resulted in offsets ranging from 20 seconds to 12 minutes in data products DP1.00036 Atmospheric CO2 isotopes and DP1.00037 Atmospheric H2O isotopes. Many sites and time periods have been affected, we will update with specific instances by the end of July 2020.

There are two root causes:

  1. The instrument becomes out of sync with the location controller at the site. This issue has been resolved and is not expected to impact data going forward.
  2. The instrument becomes out of sync with its computer. When detected, this can be resolved by a hard restart of the instrument. We are working with the manufacturer on further troubleshooting and to find a permanent resolution.

When this problem occurs, it often leads to high variance and offset mean values in reference gas measurements because the averaging window in data processing covers two span gasses.

We are raising the science review flag for the affected time periods and investigating the possibility of data correction. Please use the data portal contact form for further inquiries; select DP1.00036 or DP1.00037 in the Data Product dropdown list and include the term “PRB0040670“ in the body of your request

LI-840 Infrared Gas Analyzer downtime resulting in data flags and gaps

The LI-840 Infrared Gas Analyzer requires more maintenance and has an observed failure rate higher than other NEON sensors measuring atmospheric carbon dioxide and water vapor. This has led to multiple data gaps and periods of flagged data.

Many sites and time periods have been affected, we will update this post with specific instances within the next month. Increased monitoring has reduced the detection time of these issues, but downtime has been magnified by the suspension of field work due to COVID-19.

NaN values and raised quality flags occur in these data products, which are contained in the bundled eddy-covariance file (DP4.00200.001):

  • DP1.00100 H2O concentration - storage
  • DP1.00099 CO2 concentration - storage
  • DP2.00008 CO2 concentration rate of change
  • DP2.00009 H2O concentration rate of change
  • DP3.00009 CO2 concentration rate of change profile
  • DP3.00010 H2O concentration rate of change profile
  • DP4.00067 Carbon dioxide flux
  • DP4.00137 Latent heat flux

Please use the data portal contact form for further inquiries; select DP4.00200.001 or any of the constituent data products listed above in the Data Product dropdown list and include the term “PRB0040480“ in the body of your request

Update 4: Microbial Community Composition Data Products Being Reprocessed

This is an update to the following past news posts:

In order to be more agile in identifying and correcting quality issues with the microbial community composition data products, we are developing an in-house data processing pipeline to generate the microbial community composition data products. Testing of the pipeline is currently underway and will be followed by parameter optimization. A firm timeline for completing testing and optimization is unclear: this news item will be updated periodically as development progresses.

This issue will be updated again August 17, 2020. Please use the data portal contact form for further inquiries and include the term “INC0033894“ in the body of your request.

Resolved: Faulty algorithm in Windspeed and direction above water on-buoy (DP1.20059.001)

Update to: Faulty algorithm in Windspeed and direction above water on-buoy (DP1.20059.001).

Buoy wind speed and direction data have been reprocessed with the wind direction-averaging algorithm fix through 2019 and instrument data reprocessing is presently continuing into 2020 as of July 1, 2020. However, an investigation on the reprocessed wind direction data identified another issue of inconsistent 2D wind sensor install orientations, causing unreliable buoy wind direction data. This issue will be addressed in further detail in a separate data portal news announcement. Buoy wind speed data remains unaffected. We recommend data users continue to use the 2D wind speed and direction (DP1.00001.001) at lake sites for reliable wind direction data.

Update: Notice of incorrect soil CO2 concentration data (DP1.00095.001)

Update: to Notice of incorrect soil CO2 concentration data (DP1.00095.001).

A fix has been developed for the issue of missing atmospheric pressure corrections for the soil CO2 concentration data product (DP1.00095.001). This will result in the pressure correction being performed if the necessary inputs are available or flagging the data if the pressure correction inputs are unavailable. Reprocessing previously collected data has begun with the expectation that the corrected data will start to be published by early July. In the meantime, all previously published soil CO2 concentration data has been flagged on the NEON data portal. An update will be provided in early July 2020.

Please use the data portal contact form for further inquiries; please select DP1.00095.001 in the Data Product dropdown list and include the term "PRB0040660" in the body of your request.

Spurious trace precipitation in Primary Precipitation data product (DP1.00006.001)

A review of the precipitation data product (part of to DP1.00006.001) identified that spurious trace precipitation is recorded frequently for all primary precipitation sensors (weighing gauges). While actual precipitation events are typically distinguishable from spurious events by visual inspection of the time series, the spurious events can add significantly to precipitation totals when aggregated over long intervals. Sums of spurious precipitation range from < 0.05 mm/day to > 0.5mm/day, with higher sums generally corresponding to sites and time periods of greater evaporative demand.  

We are working on additional quality control algorithms in order to detect and remove spurious trace events in the primary precipitation data product. An updated algorithm is expected sometime in 2021. An update to this issue will be provided by March, 2021. In the meantime, data remain available and notice of this issue has been added to the to data product description. Users are advised to manually inspect and clean the dataset of spurious trace precipitation events.

For further inquiries, please use the data portal contact form; please select DP1.00006.001 in the Data Product dropdown list and include the term "PRB0040676" in the body of your request.

Instrumented data reprocessed

The full timeseries of most instrumented data products (exceptions below) were reprocessed with the most recent algorithms, quality control thresholds, and/or other metadata to improve overall data coverage and quality. Specific algorithm updates and data corrections are noted in other news items and in the issue log for each data product.  

Publication of reprocessed data will begin Wednesday, June 17, 2020 and will likely complete in early July, 2020. Reprocessed data can be identified by the publication date embedded in the file name of each downloaded data file. Dates on or after June 17, 2020 indicate reprocessed data.  

Publication of June 2020 data may be delayed until mid to late July while this process completes.  

Products not included in the current reprocessing effort include surface atmosphere exchange data (DP4.00200.001: Bundled data products - eddy covariance) and stream discharge products (DP4.00130.001: Stream discharge, and DP4.00133.001: Stream discharge rating curve). These products are reprocessed on separate schedules.

Please use the data portal contact form for further inquiries; please include the term “CHG0032696“ in the body of your request.

Resolved: Correction of 2D wind direction data

Resolution to Upcoming corrections to 2D wind direction data.

Previously, we identified an issue with the 2D anemometers on the NEON towers that were producing 2D wind data that did not correlate with 2D wind data collected at NEON meteorological stations nearby. The data have been reprocessed, and, the corrected data will be available via the data portal in July 2020. Please use the data portal contact form for further inquiries and include the term “INC0033824“ in the body of your request.

Faulty algorithm in Windspeed and direction above water on-buoy (DP1.20059.001)

NEON recently discovered an error in the algorithm that creates the  Wind speed and direction above water on-buoy (DP1.20059.001) data product. As a result, all published buoy wind direction values averaged at both 2-minute and 30-minute intervals have been faulty to present day. Buoy wind speed was not affected by this algorithm error, so this data product has not been removed from the portal. All past and present-day buoy wind speed and direction data from the corrected algorithm will be republished following a large data republication effort. An update will be provided by July 1. The nearby, on-shore aquatic met station  2D wind speed and direction (DP1.00001.001) at buoy sites remains unaffected by the buoy’s wind direction algorithm and may serve as a suitable alternative in the meanwhile.

For further inquiries, please use the data portal contact form; please select DP1.20059.001 in the Data Product dropdown list and include the term “INC0038276“ in the body of your request.

API Rate Limiting Issue

Large data downloads using neonUtilities may fail to download all available data due to the new API rate limit. If your data download returns messages in the form "No files found for site SITE and month YYYY-MM", try downloading again, specifying only the sites and months that weren't found on the first attempt.

We are working to release a new version of neonUtilities that handles this situation smoothly. In the meantime, downloading small amounts of data and/or downloading using an API token will reduce the problem.

Resolution: Incorrect Geolocation in 2019 Toolik AOP Spectrometer Data

Resolution to Notice of Incorrect Geolocation in 2019 Toolik AOP Spectrometer Data.

Previously, we identified an issue with the 2019 Toolik airborne (AOP) spectrometer (NIS) data products. The data were originally processed with an incorrect GPS timing offset and, as a result, were not correctly geolocated. The data have been reprocessed, and, as of 30 April 2020, the corrected data are available via the data portal. Please use the data portal contact form for further inquiries and include the term “INC0036345“ in the body ofyour request.