API Rate Limiting Issue

Large data downloads using neonUtilities may fail to download all available data due to the new API rate limit. If your data download returns messages in the form "No files found for site SITE and month YYYY-MM", try downloading again, specifying only the sites and months that weren't found on the first attempt.

We are working to release a new version of neonUtilities that handles this situation smoothly. In the meantime, downloading small amounts of data and/or downloading using an API token will reduce the problem.

Resolution: Incorrect Geolocation in 2019 Toolik AOP Spectrometer Data

Resolution to Notice of Incorrect Geolocation in 2019 Toolik AOP Spectrometer Data.

Previously, we identified an issue with the 2019 Toolik airborne (AOP) spectrometer (NIS) data products. The data were originally processed with an incorrect GPS timing offset and, as a result, were not correctly geolocated. The data have been reprocessed, and, as of 30 April 2020, the corrected data are available via the data portal. Please use the data portal contact form for further inquiries and include the term “INC0036345“ in the body ofyour request.

Update 3: Microbial Community Composition Data Products Being Reprocessed.

This is an update to the following past news posts:

We have determined that significant modifications to the bioinformatics pipeline are needed to ensure the highest quality microbial community composition data products. To that end, we are performing optimization of analytical parameters in the pipeline. There is no change to data availability for the microbial community composition data products.

This issue will be updated again June 15, 2020. Please use the data portal contact form for further inquiries and include the term “INC0033894“ in the body of your request.

Notice of incorrect soil CO2 concentration data (DP1.00095.001)

We have discovered that the atmospheric pressure correction for the soil CO2 concentration data product (DP1.00095.001) is not being performed, causing incorrect values to be reported. This issue affects all TIS sites and, while the exact start date is not currently known with certainty, it started prior to 4 Jan 2019 and likely affects all previously published soil CO2 concentration data. The magnitude of the impact is variable among sites (less pronounced for sites close to sea level and more substantial at high elevation sites), but also varies, albeit less markedly, with local conditions (e.g., temperature and CO2 concentration). As an example, omitting the pressure correction at atmospheric pressures of 100, 90, 80, and 70 kPa (roughly corresponding to elevations of 100, 1000, 1900, and 3000 m, respectively) resulted in an underestimate of true soil CO2 concentration by 2%, 15%, 28%, and 41%, respectively. These values were calculated assuming 10 °C temperature, 75% relative humidity, 20.95% oxygen concentration, and an uncorrected CO2 sensor reading of 1000 ppm. Work has begun to correct this issue with the expectation of having it resolved by approximately July 2020. An update will be provided by the end of May 2020. Please use the data portal contact form for further inquiries; please select DP1.00095.001 in the Data Product dropdown list and include the term “PRB0040660“ in the body of your request.

Resolved: Timestamp mis-match discovered for sensor data products at WLOU and SYCA for part of 2019

This is the final update to Timestamp mis-match discovered for sensor data products at WLOU and SYCA for part of 2019.

Current Status: Data has been re-streamed, re-processed, and re-published for both SYCA and WLOU with the correct timestamps. Data is now available for download on the Portal and API. Please use the data portal contact form for further inquiries; please include the phrase “INC0034443“ and “INC0035951“ in the body of your request.

Update #1: Timestamp mis-match discovered for sensor data products at WLOU and SYCA for part of 2019

This is an update to Timestamp mis-match discovered for sensor data products at WLOU and SYCA for part of 2019.

Current Status: Data has been re-streamed for both SYCA and WLOU with the correct timestamp. The re-processing is in a testing phase with a plan to have the data re-processed for re-publication by the end of April. An update will be posted when the data is re-published on the portal or by April 30th, 2020. Please use the data portal contact form for further inquiries; please include the phrase “INC0034443“ and “INC0035951“ in the body of your request.

Timestamp mis-match discovered for sensor data products at WLOU and SYCA for part of 2019

We have identified that the timestamp of the on-site server was set to Mountain Time instead of Coordinated Universal Time (UTC) resulting in all sensor data streaming with the wrong timestamp for SYCA and WLOU from 6/4/2019 - 12/5/2019 and 7/23/2019 - 2/17/2020, respectively. All impacted data have been removed from the NEON data portal and will be re-streamed, re-processed, and re-published. We will provide an update iby March 20, 2020 with information about when data will be on the portal. An automated script is being put in place to prevent this issue in the future. Impacted data products include:

  • AIS data products:
    • NEON.DP1.20100: Elevation of groundwater
    • NEON.DP1.20015: Specific Conductivity in groundwater
    • NEON.DP1.20217: Temperature of groundwater
    • NEON.DP1.20016: Elevation of surface water
    • NEON.DP1.20033: Nitrate in surface water
    • NEON.DP1.20042: PAR at water surface
    • NEON.DP1.20053: Temperature (PRT) in surface water
    • NEON.DP1.20288: Water quality
  • Shared AIS/TIS data products:
    • NEON.DP1.00001: 2-D windspeed and direction
    • NEON.DP1.00002: Single aspirated air temperature
    • NEON.DP1.00004: Barometric pressure
    • NEON.DP1.00024: PAR
    • NEON.DP1.00098: Relative humidity
    • NEON.DP1.00023: Shortwave and longwave radiation (net radiometer)
    • NEON.DP1.00006: Precipitation
    • NEON.DP1.00038: Stable isotope concentrations in precipitation
    • NEON.DP1.00013: Wet deposition chemical analysis

Please use the data portal contact form for further inquiries; please include the phrase “INC0034443“ and “INC0035951“ in the body of your request.

Update 2: Microbial Community Composition Data Products Being Reprocessed

This is an update to the following past news posts:

Data quality analysis for 90% of the existing fungal ITS sequencing data sets is complete. In addition, draft quality criteria and data flagging rules are under review by members of the external user community. We expect to finalize the quality criteria and flagging rules by February 29, 2020 and begin flagging sequence data immediately thereafter, beginning with fungal ITS datasets. No changes to data availability have been made.

Data sets will be re-posted incrementally as quality flagging is completed. This issue will be updated again March 30, 2020. Please use the data portal contact form for further inquiries and include the term “INC0033894“ in the body of your request.

Orthorectification errors identified for camera images for 2018_YELL_1 and 2019_JERC_5

We have identified issues with the camera image and mosaic data for 2018_YELL_1 and 2019_JERC_5. The products involved are:

  • L1
    • DP1.30010.001: High-resolution orthorectified camera imagery
  • L3
    • DP3.30010.001: High-resolution orthorectified camera imagery mosaic

In both cases, the orthorectified images for a single flight line (about 40 images for each site) are off by about 200 meters due to an error in the raw timing data. These errors are not correctable, but only a small portion of the total site coverage is affected. To correct for these issues, we deleted the misaligned images and recomputed the mosaics which will result in no loss of coverage due to spatial overlap. Updated imagery will be re-posted as soon as correction is completed. This issue will be updated again March 19, 2020. Please use the data portal contact form for further inquiries and include the term “INC0036413“ in the body of your request.

Notice of Incorrect Geolocation in 2019 Toolik AOP Spectrometer Data

We have identified an issue with a portion of the 2019 airborne (AOP) spectrometer (NIS) Level 1, Level 2 and Level 3 data products for the Toolik site, TOOL. The NIS data for one of the 5 flight days were originally orthorectified with an incorrect timing offset and, as a result, 4 flight lines (L1 and L2) and a portion of the mosaic data (L3) are not correctly geolocated. The following products are affected:

  • L1
    • DP1.30008.001: Radiance (spectrometer, flightline)
    • DP1.30006.001: Reflectance (spectrometer, flightline)
  • L2
    • DP2.30011.001: Albedo (spectrometer, flightline)
    • DP2.30016.001: Total biomass map (spectrometer, flightline)
    • DP2.30014.001: PAR (spectrometer, flightline)
    • DP2.30012.001: LAI (spectrometer, flightline)
    • DP2.30026.001: Vegetation indices (spectrometer, flightline)
    • DP2.30022.001: Canopy lignin (spectrometer, flightline)
    • DP2.30018.001: Canopy nitrogen (spectrometer, flightline)
    • DP2.30019.001: Canopy water content (spectrometer, flightline)
    • DP2.30020.001: Canopy xanthophyll cycle (spectrometer, flightline)
  • L3
    • DP3.30011.001: Albedo (spectrometer, flightline)
    • DP3.30016.001: Total biomass map (spectrometer, flightline)
    • DP3.30014.001: PAR (spectrometer, flightline)
    • DP3.30012.001: LAI (spectrometer, flightline)
    • DP3.30026.001: Vegetation indices (spectrometer, flightline)
    • DP3.30022.001: Canopy lignin (spectrometer, flightline)
    • DP3.30018.001: Canopy nitrogen (spectrometer, flightline)
    • DP3.30019.001: Canopy water content (spectrometer, flightline)
    • DP3.30020.001: Canopy xanthophyll cycle (spectrometer, flightline)

As of February 2020 the data are being re-orthorectified. We will update this message by the end of March 2020 with further information or location and availability of the corrected data.  Please use the data portal contact form for further inquiries and include the term “INC0036345“ in the body of your request.

 

Upcoming corrections to 2D wind direction data

In March of 2017 the terrestrial instrument system (TIS) team identified that the 2D anemometers on the NEON towers were producing 2D wind data that did not correlate with 2D wind data collected at NEON meteorological stations nearby. At first this was assumed to only be an issue at a handful of sites. However, as time progressed it was evident that this was an issue at nearly all operational sites. The North Transducer head of the 2D anemometers at the TIS sites listed below were not oriented consistently and all 2D wind direction data were offset from 90 to 270 degrees from 'truth'. The deviation from truth is a function of site and measurement level. The TIS team is in the process of identifying site- and measurement-level--specific dates of incorrect anemometer orientation so that these data can be corrected. The process of identifying the orientation of each 2D anemometer, the corrective action, and reprocessing the data should be completed by Spring 2020. We will provide a more specific completion date in February 2020. Please use the data portal contact form for further inquiries; please select DP1.00001.001 in the Data Product dropdown list and include the term "INC0033824" in the body of your request.

Resolved: Notice of the needed re-orthorectification of 2018 BARR and KONZ spectrometer data

This is an update to Notice of the needed re-orthorectification of 2018 BARR and KONZ spectrometer data.

Previously, we identified an issue with the 2018 KONZ and BARR airborne (AOP) spectrometer (NIS) data products. The data were originally processed with an incorrect GPS timing offset and, as a result, were not correctly orthorectified. The data have been reprocessed. As of 31 January 2020 the corrected data are available via the data portal. Please use the data portal contact form for further inquiries and include the term “INC0034943“ in the body of your request.

Resolved: Notice of the re-scaling of 2019 spectrometer Albedo (L2 and L3)

This is an update to Notice of the re-scaling of 2019 spectrometer Albedo (L2 and L3).

Previously, we identified an issue with the early 2019 airborne (AOP) spectrometer (NIS) Level 2 and 3 Albedo data products for the sites TALL, LENO, DELA, OSBS, DSNY, REDB, ONAQ, CLBJ, and OAES. The data were originally processed with an incorrect scale factor and, as a result, ranged from 0 to 10 instead of the usually distributed 0 to 1. As of January, 31 2020 the data have been re-scaled and re-published to the data-portal. Please use the data portal contact form for further inquiries and include the term “INC0035396“ in the body of your request.

Notice of the re-scaling of 2019 spectrometer Albedo (L2 and L3)

We have identified an issue with the early 2019 airborne (AOP) spectrometer (NIS) Level 2 and 3 Albedo data products for the sites TALL, LENO, DELA, OSBS, DSNY, REDB, ONAQ, CLBJ, and OAES. The data were originally processed with an incorrect scale factor and, as a result, range from 0 to 10 instead of the usually distributed 0 to 1. The following products are affected for the sites:

  • L2
    • DP2.30011.001: Albedo (spectrometer, flightline)
  • L3
    • DP3.30011.001: Albedo (spectrometer mosaic)

As of January 2020 the data are being re-scaled. We will update this message by the end of February 2020 with further information or location and availability of the corrected data.  Please use the data portal contact form for further inquiries and include the term “INC0035396“ in the body of your request.

Notice of the needed re-orthorectification of 2018 BARR and KONZ spectrometer data

We have identified an issue with the 2018 KONZ and BARR airborne (AOP) spectrometer (NIS) data products. The data were originally processed with an incorrect GPS timing offset and, as a result, are not correctly orthorectified. This affects all spectrometer products (radiance, reflectance, spectral indices, and mosaics) for the two sites’ 2018 collections:

  • L1
    • DP1.30006.001: Spectrometer orthorectified surface directional reflectance (flightline)
    • DP1.30008.001: Spectrometer orthorectified at-sensor radiance (flightline)
  • L2
    • DP2.30019.001: Canopy water content (flightline)
    • DP2.30020.001: Canopy xanthophyll cycle (flightline)
    • DP2.30014.001: fPAR spectrometer (flightline)
    • DP2.30012.001: LAI spectrometer (flightline)
    • DP2.30016.001: Total biomass map (spectrometer, flightline )
    • DP2.30026.001: Vegetation indices (spectrometer, flightline)
  • L3
    • DP3.30011.001: Albedo (spectrometer mosaic)
    • DP3.30022.001: Canopy lignin (mosaic)
    • DP3.30018.001: Canopy nitrogen (mosaic)
    • DP3.30019.001: Canopy water content (mosaic);
    • DP3.30020.001: Canopy xanthophyll cycle (mosaic)
    • DP3.30014.001: fPAR (spectrometer, mosaic)
    • DP3.30012.001: LAI (spectrometer, mosaic)
    • DP3.30006.001: Spectrometer orthorectified surface directional reflectance (mosaic)
    • DP3.30016.001: Total biomass map (spectrometer, mosaic)
    • DP3.30026.001: Vegetation indices (spectrometer, mosaic)

The data are being reprocessed. We will update this message by the end of January 2020 with further information or location and availability of the corrected data.  Please use the data portal contact form for further inquiries and include the term “INC0034943“ in the body of your request.

neonUtilities 1.3.3 now available on CRAN

neonUtilities 1.3.3 is now available on CRAN! Changes in this version are fairly minor, primarily meant to fix bugs in version 1.3.2.

  • loadByProduct() used with the `avg` input to filter by averaging interval is working again after breaking in the 1.3.2 update.
  • variables files processed by stackByTable() now include the fields added by stackByTable()
  • variables,validation, readme, and sensor_positions files processed by stackByTable() now include the data product number in the file name, e.g. variables_10098. This avoids the problem of duplicating file names when downloading multiple data products.
  • Package citation is regularized. To get citation information for the neonUtilities package, run citation('neonUtilities')

If you have any issues or questions about this release, please visit our issues log on GitHub.

Update: Microbial Community Composition Data Products Being Reprocessed

This is an update to Microbial Community Composition Data Products Being Reprocessed.

The quality issues are currently under investigation, with evaluation of the raw sequence data underway. For some data sets, the read quality for the reverse read was not sufficient to allow paired-end read merging, suggesting that the reverse reads for an unknown number of runs may be unsuitable for analysis using the existing workflow. QC analysis of individual sequencing runs is underway and will be used to flag low-quality sequence data. Re-analysis of microbial community composition data products will begin after this analysis is complete. Thus far, no changes have been made to data available on the Data Portal. This issue will be updated again Feb 1, 2020.

Water Quality Software Updates

Water quality – DP1.20288.001 fDOM (all Aquatic sites): Data processing software improvements will now use SUNA absorbance data to estimate corrected fDOM, as described in NEON.DOC.004931vA NEON Algorithm Theoretical Basis Document (ATBD): Water Quality, rather than only correcting fDOM using temperature. This improvement will also considerably reduce the number of absorbance Quality Flags corresponding to fDOM data. NEON plans to test and implement the update within the next month. Corrections in published data should be fully reflected in 2020 February’s publication of 2020 January’s data, and partially reflected in 2020 January’s publication of 2019 December data. All data collected prior to the software deployment will be re-processed with the correct averaging windows and re-published over the next 6-12 months.

Aquatic instrument data product corrections: Update 1

Recent updates in the algorithms that generate some of our aquatic instrument data will improve published data products by fixing erroneous data. The updates will address the following issues with each of these data products at all sites where they are collected:

  1. Nitrate in Surface Water - DP1.20033.001 (all Aquatic sites): A new algorithm was deployed on October 24th, 2019 that improved data quality and publication format, see NEON.DOC.002181vC (the algorithm theoretical basis document) for details on the revised processing algorithm. Data collected and processed after October 24th, 2019 should be reflected on the data portal with the December 2019 publication of late October and November 2019 data. Data collected prior to October 24th, 2019 will be re-processed and re-published over the next 6-12 months and eventually all nitrate data collected by NEON will be available in the new format. Until the re-publication of older data is complete, users can disregard the superfluous blank rows with flags every 30 seconds to one-minute and use the nitrate data published every 15 minutes for the older format.
  2. Windspeed and direction above water on-buoy - DP1.20059.001 (all Aquatic sites with buoys): Software algorithm fixes have been tested and should start to be reflected in new data within the next month. A data portal news update will accompany this algorithm update. Data collected and processed after the deployment will have the correct time averaging windows. Corrections on the data portal should be fully reflected in 2020 February’s publication of 2020 January’s data. All data collected prior to the deployment will be re-processed with the correct averaging windows and re-published over the next 6-12 months. The time shift begins at approximately zero and progresses to a positive shift of approx. 1 hour at the end of 4 hours. It then resets to zero. On the daily and higher timescale, the differences between actual and reported statistics are small.
  3. Temperature at specific depth in surface water – DP1.20264.001 (all Aquatic sites with buoys): Software algorithm fixes have been tested and should start to be reflected in new data within the next month. A data portal news update will accompany this algorithm update. Data collected and processed after the deployment will have the correct time averaging windows. Corrections on the data portal should be fully reflected in 2020 February’s publication of 2020 January’s data. All data collected prior to the deployment will be re-processed with the correct averaging windows and re-published over the next 6-12 months. We recommend using the instantaneous 1-minute temperature at specific depth in surface water data until we announce the fixes for averaged data.
  4. Water quality – DP1.20288.001 (all Aquatic sites): Uncertainty estimates corresponding to each data stream from the multisonde have been corrected by November 9th, 2019 and any data collected and processed on or after that will have valid uncertainty estimates published. Data collected and processed between July 16th, 2019 and November 8th, 2019 will be re-processed and re-published over the next 6-12 months. Any data collected and processed prior to July 16th, 2019 was not part of this issue and was published with appropriate uncertainty estimates. Data streams other than uncertainty were not impacted by this issue.
  5. Elevation of groundwater - DP1.20100.001 (all 30-minute data, beginning Jan 1, 2018): A software algorithm fix has been identified and NEON plans to internally test and deploy the updated algorithm within the next month. A data portal news update will accompany this algorithm update. Data collected and processed after the deployment will have the correct data included in the 30-minute averages. Corrections on the data portal should be fully reflected in 2020 February’s publication of 2020 January’s data. All data collected prior to the deployment will be re-processed with the correct averaging windows and re-published over the next 6-12 months. We recommend using the 5-minute groundwater elevation data product until we announce the algorithm update.

At the time of this posting, the algorithm updates are not reflected in published data. Data are being re-processed and will be corrected. We will announce when these corrected data have been republished.

neonUtilities 1.3.2 now available on CRAN

neonUtilities 1.3.2 is now available on CRAN! There are several updates in this version, primarily around improving performance and retaining more complete metadata from the original download.

  • stackByTable has optional parallel processing. Use the input nCores to specify the number of cores to use in stacking the data, for speedier processing. This is helpful primarily in dealing with long time series of sensor data. The function may override nCores if it determines parallelization will not significantly improve performance.
  • stackByTable retains a version of t he readme file. In downloads, readme files are generated for each site by month combination, so the readme file provided by stackByTable deletes the site- and date-specific information.
  • stackByTable also retains the sensor_positions files, which are provided with downloads of streaming sensor data. sensor_positions files are specific to a site, so the version provided by stackByTable is stacked across sites, in the same way as the data tables.
  • stackByTable no longer requires the ‘folder’ input to distinguish between portal and zipsByProduct downloads. It won’t fail if ‘folder’ is included, so any existing code using it will continue to work.
  • stackByTable adds a new column to each data table, called publicationDate, populated with the date-time stamp of publication for each record of data. These dates are provided with the raw downloaded files, but were not tracked during stacking in previous versions.
  • Publication date is also taken into consideration in the variables and validation files - the file kept by stackByTable is now reliably the most recently published version.
  • A new function, readTableNEON , takes a NEON data table and a NEON variables file as inputs and assigns classes to data frame columns based on the data types specified in the variables file. Dates are stored as POSIXct, and URLs are stored as strings.
  • loadByProduct incorporates readTableNEON , so data accessed by loadByProduct are read into R according to data type.

If you have any issues or questions about this release, please visit our issues log on GitHub.

— 20 Items per Page
Showing 1 - 20 of 83 results.