User Guide

Introduction

In this package we define higher-level functions using packages ‘rOmniDriver’ and the ‘r4photobiology’ suite as bases. These functions conform a sort of “grammar for spectral data acquisition”. We provide a separate function for each conceptual step in the process leading from raw counts to final spectral data—e.g. spectral irradiance. Some higher level functions are included as well, serving also as examples of how to implement specific protocols.

The package includes functions for direct data acquisition and near real-time processing of spectra, plus functions that read raw-detector-counts and instrument settings from text files output by programs and instruments from Ocean Optics. After acquisition or file reading the processing of the raw spectral data is the same.

For data acquisition, the initial steps are opening a connection to an instrument and retrieving a descriptor including model type, optical configuration and possibly calibration data from its non-volatile memory. These first steps in most cases are needed only at the start the R session where data will be acquired. The drivers from Ocean Optics support multichannel instruments and simultaneous connection to multiple spectrometers. This is also supported by this package.

The steps for the acquisition of one raw spectrum are to 1) define the settings to be used, 2) optionally adjust or tune some of these settings automatically, 3) acquire the spectrum, 4) convert the raw counts into linearised counts per second.

In most cases we need acquire more than one raw spectrum, frequently following a certain protocol. There are two possibilities: a) they are conceptually a single observation, such as multiple scans to be averaged to reduce noise or acquired with different integration times (bracketing) and to be merged to increase the dynamic range, and b) sets of spectra in which each spectrum represents a different observation, such as a ‘measurement’ spectrum and a ‘dark’ spectrum to be used to correct it. The distinction may be not that easy to grasp, but in case a) we are measuring a single thing, what we are measuring remains unchanged for all spectra acquired in a set—e.e. we are measuring repeatedly, possibly changing the instrument settings. In case b), although the different spectra may contribute to a single processed spectrum, they are based on measuring a different thing, say a light source and a paired dark reading.

In case a) all the acquired raw spectra are stored as columns in the same raw_spct object, while in case b) the different conceptual spectra are each stored in a separate raw_spct object, member of a raw_mspct collection of spectral objects. As a) and b) are usually combined, this approach allows and easy distinction of the two situations, and greatly simplifies the coding of later data processing steps.

When spectral data are acquired with other software and read from files into R, this same approach is used for raw data storage. The role of the different files is determined by the structure of a named list used as argument. Files do not contain a full descriptor of the instrument and consequently the instrument descriptor need to be supplied as an additional argument.

Following the approach of the ‘r4photobiology’ suite, we store all available metadata as attributes to the same objects where the acquired spectral data themselves are stored. The stored metadata includes a descriptor of the instrument and the instrument settings used. These attributes are an addition to metadata that is normally held in spectral objects of classes defined in package ‘photobiology’, such as time, location, and label for the measured data. Furthermore, metadata is added at each step, allowing the tracing of the origin and processing of the data. This allows the functions to detect mistakes like an user attempting to linearise raw-counts data that have been already linearised. This safeguard works even for data linearised in the spectrometer software and read from files.

In all cases a wavelength calibration is used to map the pixels to wavelengths. Wavelength calibration can be that retrieved from the instruments’ non-volatile memory or supplied directly by to user. Different further steps are needed depending on the quantity measured and method. In the case of irradiance, a calibration needs to be applied to the counts-per-second spectral data. Usually calibration data is also stored in the non-volatile memory of the spectrometer, but in some cases it is convenient to apply special algorithms or calibration data that cannot be retrieved from the instrument itself. Functions for both the normal and special cases are also included in the package. Here it is important to remember that a calibration will be valid only if the raw spectral data processing steps are exactly the same for the spectral data used for calibration and the measured data. This in practice means that calibrations supplied by instrument manufacturers will almost never be usable with all the methods implemented in this pacakge. However, methods are provided also for this case. When doing a calibration “manually” by simply measuring a standard lamp you will need the reference lamp spectrum and the raw detector-counts data from the calibration event. Doing the calibration manually allows the use of any of the different measuring protocols and correction algorithms implemented. This permits one to obtain valid calibration coefficients for the data-acquisition protocol one adopts for measurements.** On the other hand, this also means that the functions in the present package, as each of them fulfils a simple and well defined step of data processing, are ideal for in silico testing of alternative data processing approaches and measuring protocols.

Functions

function name role
data acquisition
start_session search for connected instruments and obtain a handle to the driver
end_session close all connections and release handle
list_instruments list all connected instruments
get_oo_descriptor get a descriptor for one of the possibly several connected instruments
set_descriptor_wl replace the wavelength values
set_descriptor_bad_pixs replace the indexes to bad array detector pixels
set_descriptor_nl replace the function used to correct for array non-linearity
set_descriptor_calib_mult replace the vector of irradiance calibration multipliers
set_descriptor_integ_time replace the vector defining the range of valid integration times
get_oo_settings query the instrument to obtain its current settings
acq_settings build a list of setting values
tune_acq_settings tune the integration time and number of scans to the current radiation conditions
acq_raw_spct take one measurement (data point)
acq_raw_mspct take a sequence of spectral measurements (several data points)
hs_acq_raw_mspct take a sequence of spectral measurements at high speed
acq_irrad_interactive interactively acquire spectral irradiance or fluence and save data and plots to disk
acq_fraction_interactive interactively acquire spectral transmittance or reflectance and save data and plots to disk
acq_fraction_pulsed_interactive as above but using a pulsed light source
data procesing
trim_counts replace values from saturated array pixels with NAs
bleed_nas replace values from pixels neighbouring NAs with NAs
linearize_counts linearise the raw counts
skip_bad_pixs replace bad pixels with interpolated raw counts
ref_correction subtract a reference (e.g. dark signal) from a measured spectrum
photobiology::fshift can be used to subtract a region of the same scan as an internal “dark” reference
raw2cps methods to convert raw counts into counts per second. The method specialization for raw_spct returns a cps_spct object
merge_cps method to do HDR merge of bracketed scans
slit_function_correction apply a correction for the ‘long tails’ of the slit function
filter_correction apply a correction for stray light based on the use of UV-cut-off or other selective filters
no_filter_correction apply a correction for stray light not based on the use of UV-cut-off or other selective filters
uvb_corrections apply corrections for reducing the impact of stray light and slit function on counts-per-second estimates. This function intelligently adjusts the processing method used based on the spectral data item supplied as x argument
which_descriptor select an instrument descriptor from a list based on the dates between which a calibration is valid
photobiology::cps2irrad methods to convert counts per second into spectral irradiance
photobiology::cps2Tfr methods to convert counts per second into transmittance
photobiology::cps2Rfr methods to convert counts per second into reflectance
calibration
compute_irrad_calibration compute irradiance calibration multipliers from raw counts data and polynomial coefficients from standard lamp certificates using any of the different measuring protocols and data processing methods implemented in the package
oo_calib2irrad_mult compute irradiance calibration multipliers from Ocean Optics provided calibration
file reading
read_oo_ssdata read data from a ‘SpectraSuite’ output file with raw-counts data
read_oo_ovdata read data from a ‘OceanView’ output file with raw-counts data
read_oo_pidata read data from a ‘XXXXXXXXX’ output file with raw-counts data from a Raspberry Pi board
read_files2mspct read multiple data files and assemble from them into a raw_maspct object compatible to that obtained through direct data acquisition
read_oo_caldata read irradiance calibration data from an Ocean Optics supplied calibration file
set_oo_ssdata_settings set the instrument settings attributes by decoding the header of text files saved from SpectraSuite
set_oo_ssdata_descriptor set the instrument descriptor attribute by decoding the header of text files saved from SpectraSuite and optionally merging these incomplete data with that in a descriptor supplied as argument
merge_raw_mspct Merge raw_spct objects members of a single raw_mspct object into a raw_spct object with multiple columns
high level method
raw2corr_cps higher level function that returns corrected counts per second useful for calculating spectral reflectance and spectral transmittance.
s_irrad_corrected high-level method with specializations for files and raw_mspct objects returning spectral irradiance in a source_spct object.
s_fraction_corrected high-level method with specializations for files and raw_mspct objects returning spectral transmittance in a filter_spct object or spectral reflectance in a reflector_spct object.

Workflow

The workflow consists in several steps. First all one needs to start a new connection to the driver by means of start_session() and disconnect at the end of the measuring session with end_session(). To learn which instruments are connected and which numerical index pints to each one uses list_instruments(). The indexes start at zero, so if there is only one instrument connected one can rely on its index being 0. The same applies to channels, for multichannel instruments.

Once we know which instrument we want to address and its index, we must create an instrument descriptor with function get_oo_descriptor(). From this point onwards we will use this descriptor to address the instrument. The descriptor contains information about the instrument obtained initially by querying it. Instruments may have some pieces of information unavailable (e.g. not stored in there non-volatile memory), or the user may want to override the available information with that from a more recent or better calibration. set_descriptor_nl(), set_descriptor_wl(), set_descriptor_bad_pixs() and set_descripto_integ_time() can be used to set or override the function used for correcting for the non-linearity of the sensors, the wavelength values corresponding to each pixel, a vector of indexes (starting at one) corresponding to bad pixels in the array (pixel known to return bad data) and the range of integration times in ms that are valid input for the instrument. __It is always good to check against the specifications of the instrument whether the values stored in its

As there are several settings needed for the acquisition of a spectrum, we will store sets of settings into objects that can reuse and modify. To create these objects we use function acq_settings(). The settings include both actual parameters that need to be set in the instrument by sending commands and parameters that our software may use to calculate suitable values for such settings. This function is quite flexible and allows the use of bracketing. In other words a single observation can consist in more than one data acquisition, each of these using different setting, with the aim of merging them into a single spectrum.

Some settings need to be adjusted according to the current irradiance level so as to make full use of the dynamic range of the instrument. For this we use function tune_acq_settings(), which updates the settings stored in a list as created by acq_settings() or as returned by a previous call to tune_acq_settings().

Function get_oo_settings() can be used to query the instrument for the settings currently in use. To acquire spectral data we can use one of two functions, acq_raw_spct() to acquire a single spectrum, or acq_raw_mspct() to acquire a collection of spectra according to a user supplied protocol. In cases when we use HDR (bracketing) or overexpose some part of the spectrum the first data-processing step is to replace the data from the saturated pixels with NA’s using function trim_counts()—as the settings used in the acquisition and descriptor of the instrument are stored in the spectral object, this function needs only to be supplied a raw_spct object, unless the user wants to discard more pixels. As pixels neighbouring overexposed pixels are disturbed by the charge overflowing from the saturated wells, these pixels need also to be set as NAs by means of function bleed_nas. If the instrument array contains bad pixels, it is best to replace the data produced by them at this early stage. Function skip_bad_pixs() is useful here and it replaces the bad data with the average of the readings from the neighbouring pixels. The third step after data acquisition, unless the linearisation is done by the instrument, is to linearise the acquired valid raw counts by means of method linearize_counts(). The non-linearity of the detector is related to how “full” are its electron wells, irrespective of whether the electrons originate from measured photons or any other source, consequently the linearisation function should be applied to raw detector counts at this point.

CCD arrays, especially if not cooled, have a significant dark signal. The best approach to remove this background signal is to measure it—to take a spectral reading with the array not exposed to light. This dark spectrum is them subtracted from the one measured in the light. An alternative, with advantages and disadvantages, is to use the signal from pixels corresponding to wavelengths known a priori to be absent in the light source being measured. This calculation is done with the linearized raw counts by means of method photobiology::fshift() with a pair of suitable wavelength values for range. In the case of measuring transmittance (and absorbance) and reflectance we need to also measure a clear and white reference respectively.

The raw counts after linearisation followed by subtraction of the dark signal are still expressed in number of counts per integration period. As the duration of the integration time is variable, we need to re-express the raw counts as counts-per-second (cps). Once more, the integration time used for the acquisition is stored in the cps_spct object and the conversion can be done by method raw2cps() with just this object as input. The values returned, expressed as counts per second, are linearly proportional to the rate at which photons imping each detector pixel. Be aware that as detector quantum yield depends on wavelength, a different multiplier is needed for each pixel to be able to convert the counts-per-second data to units of photons-per-unit-time.

If bracketing for HDR has been used during acquisition, at this point, the values of the cps for each of the bracketed spectra are expressed on the same scale. This means that they can simply be spliced together. The simplest approach possible, used by method merge_cps() is to replace the clipped data from saturated pixels (and possibly their neighbours) in the longer integration with data from the same pixels obtained with shorter integration. As we used trim_counts() and bleed_nas() on the raw counts, those saturated pixels have already been replaced with NA’s making the operation easy to implement.

After the preceding steps, if the cps or merged cps values are from an irradiance measurement the spectral data are now ready to be multiplied by the calibration constants corresponding to each individual pixel. If the values are to be used for calculation of transmittance or reflectance, all what is left to do is simple arithmetic.

Additional corrections

Apply slit function correction

The slit function describes how monochromatic light “reaches” adjacent pixels which should have been in total darkness had the monochromator been perfect. This is a subtle correction, which will affect only regions of a spectrum where the change in signal strength with wavelength is very steep, such as the UV region of a solar spectrum or a very narrow absorption peak.

The slit function of a spectrometer can be estimated by measuring laser beams at different wavelengths and fitting suitable functions to the observed data. Here we explain how to apply such a function. The slit function is the result of an optical effect, and consequently independent of integration time and other detector settings. Consequently, can be applied to either the merged cps values or after application of a calibration. Obviously, the stage at which the slit correction is applied needs to match the stage of processing of the calibration data used for estimating the slit function.

The correction to apply to a given pixel depends on the readings of nearby pixels. Some authors recommend to apply this correction recursively, however, we follow the simple approach of applying it only once as an approximation.

Code examples

All examples in vignettes use data and files included in the package. Data files in formats foreign to R, are stored in the "extdata" folder, as is the norm for R packages.

To reproduce any of the examples which use files as data input, it is best to make a local copy of the whole "extdata" folder. The files used in examples are organized into sub-folders, but all example code assumes that its in being run with "extdata" or a copy of it as the current working folder. The chunk below sets the working directory to point to the exdata folder in the installed package.

folderpath <- system.file("extdata", package="ooacquire")

Measurement of spectral irradiance

Irradiance estimates are affected by calibration errors, dark electrical noise, dynamic range and stray light. In this section we assume that the available calibration is valid. Random electrical and thermal noise can be averaged out by repeated measurement. Dynamic range depends both on the noise floor and on the resolution of the analogue to digital conversion electronics of the instrument. A way to increase the effective dynamic range is be bracketing integration time and splicing/merging the resulting spectra. This is what in digital photography is called HDR or high dynamic range images obtained by merging sets of images obtained at a series of different exposure values. In addition to improved optical design, stray light can be corrected for if it is possible to measure it by itself. This can be achieved by using an optical filter that transmits radiation of wavelengths causing stray light, but that blocks radiation of wavelengths we are interested in. An additional correction that can be applied is for the slit function of the instrument. The slit function describes how a single wavelength peak (e.g. similar to a laser beam) broadens at the array detector.

Which corrections are needed and which make little difference depends on the type of instrument, the characteristics of the light source and the region of the spectrum we are interested in. Consequently, depending on the circumstances we may want to use different measurement protocols. The high-level functions in the package “guess” the protocol from the set of file names or set of spectra passed as argument. For irradiance calculations three named members are recognized as different measurements in a protocol: "light", "filter" and "dark", each of these can be bracketed for integration time. In the case of file names for any of these three members, a vector of file names, is interpreted as a bracketed measurement. In the case of collections of spectra, raw_spct with multiple counts columns are interpreted as bracketed. Of course, at least a "light" member is required as input. If it is missing an empty source_spct object is returned.

Protocol light spct filter spct dark spct signal level signal range speed
minimal short - - high < 5e1 ?? 1/1
normal short - short high < 1e2 1/2
corrected short short short medium < 1e3 1/3
cor. brkt short + long short + long short + long low < 1e4 1/33

Examples

We first load the packages to be used, and set the working directory to the location of the example data files included in the package.

library(photobiology)
library(photobiologyWavebands)
library(ggspectra)
library(ooacquire)

From files

In special cases we may need to set a locale matching the language used in the files for month names is not English. In most cases month names are in English, an time zones and decimal markers will be inferred from the data files themselves, making it unnecessary to explicitly pass a locale definition as argument to functions.

The first example is the one listed as “minimal” in the table above, we use a single spectrum, using for the dark correction pixels expected to receive no true excitation. For example, those in the UVC region of the spectrum when the light source is sunlight at ground level.

We set a list of file names to be read, with members named according to their role in the measuring protocol.

file_names <- list(light = paste(folderpath, "irrad-files/light-short.txt", sep = "/"))

We subsequently compute the spectral irradiance.

one_file.spct <- 
  s_irrad_corrected(x = file_names,
                    descriptor = which_descriptor("2016-10-11" , 
                                                  MAYP11278_descriptors),
                    correction.method = MAYP11278_ylianttila.mthd)

As these measurements are of sunlight in a greenhouse we can remove noise from regions known to be zero by replacing with zeros all spectral irradiance values form wavelengths shorter than 290 nm.

one_file.spct <- trim_wl(one_file.spct, 
                         range = c(290, NA), 
                         use.hinges = FALSE, 
                         fill = 0)
one_file.spct
## Object: source_spct [1,423 x 2]
## Wavelength range 250.21-898.81 nm, step 0.43-0.48 nm 
## Label: light: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/light-short.txt 
## Measured on 2016-10-11 14:23:05 UTC 
## Variables:
##  w.length: Wavelength [nm]
##  s.e.irrad: Spectral energy irradiance [W m-2 nm-1] 
## --
## # A tibble: 1,423 × 2
##    w.length s.e.irrad
##       <dbl>     <dbl>
##  1     250.         0
##  2     251.         0
##  3     251.         0
##  4     252.         0
##  5     252.         0
##  6     253.         0
##  7     253.         0
##  8     254.         0
##  9     254          0
## 10     254.         0
## # ℹ 1,413 more rows

We plot the result from “minimal” protocol.

autoplot(one_file.spct, unit.out = "photon")

getWhenMeasured(one_file.spct)
## [1] "2016-10-11 14:23:05 UTC"
cat(getWhatMeasured(one_file.spct))
## light: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/light-short.txt
getWhereMeasured(one_file.spct)
## # A tibble: 1 × 3
##     lon   lat address
##   <dbl> <dbl> <chr>  
## 1    NA    NA <NA>
cat(comment(one_file.spct))
## Processed on 2024-11-01
## with 's_irrad_corrected()' from 'ooacquire' ver. 0.5.1.1
## 
## from files:
## light: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/light-short.txt
getInstrDesc(one_file.spct)
## Data acquired with 'MayaPro2000' s.n. MAYP11278
## grating 'HC1', slit '010s'
## diffuser 'cosine'
getInstrSettings(one_file.spct)
## integ. time (s): 1.6
## total time (s): 4.8
## counts @ peak (% of max): 76

At the other extreme, we can use bracketing, a dark measurement, plus a measurement with a UV-absorbing filter (polycarbonate). The UV absorbing filter allows us to estimate stray light in the UV region. This is the protocol listed as corrected and bracketed (cor. brkt.) in the table above.

We go exactly through the same steps as before, the only difference is the list of file names passed as argument to parameter x.

file_names <- list(light = paste(folderpath, c("irrad-files/light-short.txt",
                             "irrad-files/light-long.txt"), sep = "/"),
                   filter = paste(folderpath, "irrad-files/flt-long.txt", sep = "/"),
                   dark = paste(folderpath, c("irrad-files/dark-short.txt",
                            "irrad-files/dark-long.txt"), sep = "/"))
five_files.spct <- 
  s_irrad_corrected(x = file_names,
                    descriptor = which_descriptor("2016-10-11", 
                                                  MAYP11278_descriptors),
                    correction.method = MAYP11278_ylianttila.mthd)
## HDR CPS ratio = 0.791; replacing 'cps_2' by 'cps_1' instead of splicing.

As these measurements are of sunlight in a greenhouse we can remove noise from regions known to be zero as we did above.

# force to zero wavelengths < 290 nm use only for sunlight, and after checking plot
five_files.spct <- trim_wl(five_files.spct, 
                           range = c(290, NA), 
                           use.hinges = FALSE, 
                           fill = 0)
five_files.spct
## Object: source_spct [1,423 x 2]
## Wavelength range 250.21-898.81 nm, step 0.43-0.48 nm 
## Label: light: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/light-short.txt, /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/light-long.txt
## filter: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/flt-long.txt
## dark: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/dark-short.txt, /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/dark-long.txt 
## Measured on 2016-10-11 14:23:05 UTC 
## Variables:
##  w.length: Wavelength [nm]
##  s.e.irrad: Spectral energy irradiance [W m-2 nm-1] 
## --
## # A tibble: 1,423 × 2
##    w.length s.e.irrad
##       <dbl>     <dbl>
##  1     250.         0
##  2     251.         0
##  3     251.         0
##  4     252.         0
##  5     252.         0
##  6     253.         0
##  7     253.         0
##  8     254.         0
##  9     254          0
## 10     254.         0
## # ℹ 1,413 more rows

We can now plot the result from complex protocol.

autoplot(five_files.spct, unit.out = "photon")

getWhenMeasured(five_files.spct)
## [1] "2016-10-11 14:23:05 UTC"
getWhatMeasured(five_files.spct)
## [1] "light: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/light-short.txt, /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/light-long.txt\nfilter: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/flt-long.txt\ndark: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/dark-short.txt, /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/dark-long.txt"
getWhereMeasured(five_files.spct)
## # A tibble: 1 × 3
##     lon   lat address
##   <dbl> <dbl> <chr>  
## 1    NA    NA <NA>
cat(comment(five_files.spct))
## Processed on 2024-11-01
## with 's_irrad_corrected()' from 'ooacquire' ver. 0.5.1.1
## 
## from files:
## light: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/light-short.txt, /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/light-long.txt
## filter: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/flt-long.txt
## dark: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/dark-short.txt, /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/dark-long.txt
getInstrDesc(five_files.spct)
## Data acquired with 'MayaPro2000' s.n. MAYP11278
## grating 'HC1', slit '010s'
## diffuser 'cosine'
getInstrSettings(five_files.spct)
## integ. time (s): 1.6, 7
## total time (s): 4.8, 21
## counts @ peak (% of max): 76

These data are all from the same set of measurements, as one can apply the calculations for the “minimal” protocol to a subset of the data from the “corrected bracketed” protocol. We calculate the differences of the estimates from the “minimal” protocol compared to the complex one.

(q_irrad(one_file.spct) / q_irrad(five_files.spct) - 1) * 100
##   Q_Total 
## 0.2663931 
## attr(,"time.unit")
## [1] "second"
## attr(,"radiation.unit")
## [1] "total photon irradiance"

Differences expressed as percentage for different wavelength ranges.

knitr::kable(
  t((q_irrad(one_file.spct, c(UV_bands(), VIS_bands())) / 
     q_irrad(five_files.spct, c(UV_bands(), VIS_bands())) - 1) * 100),
  digits = 2
)
Q_]UVC.ISO Q_UVB.ISO Q_UVA.ISO Q_Purple.ISO Q_Blue.ISO Q_Green.ISO Q_Yellow.ISO Q_Orange.ISO Q_Red.ISO
NaN 80.29 1.54 0.52 0.16 0.16 0.14 0.16 0.13

From OceanView files

Files output by SpectraSuite and OceanView have different header format, but they can be recognized automatically. Consequently there is no difference in the function calls. The filter measurement is not used as it is not suitable for the source spectrum measured, and would be ignored with a warning.

file_names <- list(light = paste(folderpath, "irrad-files/light_MAYP112785.txt", sep = "/"),
#                    filter = paste(folderpath, "irrad-files/filter_MAYP112785.txt", sep = "/"),
                    dark = paste(folderpath, "irrad-files/dark_MAYP112785.txt", sep = "/"))

In this case we use a different descriptor definition only because the files acquired with SpectraSuite and OceanView originated from different instruments.

ov_files.raw_mspct <- 
  ooacquire::read_files2mspct(file_names,
                              descriptor = 
                                which_descriptor("2017-01-05", MAYP112785_descriptors))
summary(ov_files.raw_mspct[[1]])
## Summary of raw_spct [2,068 x 2] object: anonymous
## Wavelength range 198.408-1115.677 nm, step 0.406-0.472 nm
## Label: File: light_MAYP112785.txt 
## Measured on 2017-01-05 16:22:55 UTC 
## Data acquired with 'MayaPro2000' s.n. MAYP112785
## grating 'HC1', slit '010'
## diffuser 'cosine'
## integ. time (s): 2
## total time (s): 2
## counts @ peak (% of max): NAVariables:
##  w.length: Wavelength [nm]
##  counts: Raw detector counts [number] 
## --
##     w.length          counts        
##  Min.   : 198.4   Min.   :  -82.17  
##  1st Qu.: 439.5   1st Qu.: 1625.58  
##  Median : 673.9   Median : 1738.33  
##  Mean   : 668.3   Mean   : 2302.09  
##  3rd Qu.: 900.0   3rd Qu.: 1972.08  
##  Max.   :1115.7   Max.   :46912.83
summary(ov_files.raw_mspct[[2]])
## Summary of raw_spct [2,068 x 2] object: anonymous
## Wavelength range 198.408-1115.677 nm, step 0.406-0.472 nm
## Label: File: dark_MAYP112785.txt 
## Measured on 2017-01-05 16:23:55 UTC 
## Data acquired with 'MayaPro2000' s.n. MAYP112785
## grating 'HC1', slit '010'
## diffuser 'cosine'
## integ. time (s): 2
## total time (s): 2
## counts @ peak (% of max): NAVariables:
##  w.length: Wavelength [nm]
##  counts: Raw detector counts [number] 
## --
##     w.length          counts      
##  Min.   : 198.4   Min.   : -79.5  
##  1st Qu.: 439.5   1st Qu.:1585.2  
##  Median : 673.9   Median :1661.5  
##  Mean   : 668.3   Mean   :1681.7  
##  3rd Qu.: 900.0   3rd Qu.:1753.5  
##  Max.   :1115.7   Max.   :3665.5

In this case we use a different correction.method definition only because the files acquired with SpectraSuite and OceanView originated from different instruments.

ov_files.spct <- 
  s_irrad_corrected(x = ov_files.raw_mspct,
                    correction.method = ooacquire::MAYP112785_ylianttila.mthd)
## Warning in trim_counts(x): Negative raw counts in data!
## These are not true raw detector counts.
## A dark correction may have been applied.
## Warning in trim_counts(x): Negative raw counts in data!
## These are not true raw detector counts.
## A dark correction may have been applied.
ov_files.spct <- 
  s_irrad_corrected(x = file_names,
                    descriptor = which_descriptor("2017-01-05", 
                                                  MAYP112785_descriptors),
                    correction.method = ooacquire::MAYP112785_ylianttila.mthd)
## Warning in trim_counts(x): Negative raw counts in data!
## These are not true raw detector counts.
## A dark correction may have been applied.
## Warning in trim_counts(x): Negative raw counts in data!
## These are not true raw detector counts.
## A dark correction may have been applied.
ov_files.spct
## Object: source_spct [1,439 x 2]
## Wavelength range 251.116-899.874 nm, step 0.428-0.47 nm 
## Label: light: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/light_MAYP112785.txt
## dark: /tmp/RtmpV9z5wm/Rinst12da1e6289f9/ooacquire/extdata/irrad-files/dark_MAYP112785.txt 
## Measured on 2017-01-05 16:22:55 UTC 
## Variables:
##  w.length: Wavelength [nm]
##  s.e.irrad: Spectral energy irradiance [W m-2 nm-1] 
## --
## # A tibble: 1,439 × 2
##    w.length  s.e.irrad
##       <dbl>      <dbl>
##  1     251. -0.000187 
##  2     252.  0.0000196
##  3     252.  0.00113  
##  4     253. -0.000277 
##  5     253. -0.000949 
##  6     253. -0.000255 
##  7     254. -0.00113  
##  8     254.  0.000933 
##  9     255.  0.000392 
## 10     255.  0.000498 
## # ℹ 1,429 more rows

We can now plot the result from the OceanView files.

autoplot(ov_files.spct, unit.out = "photon")

The default smoothing method, with strength set 0.4 instead of the default of 1, removes some of the noise in the low signal regions without affecting the regions with higher signal.

autoplot(smooth_spct(ov_files.spct, strength = 0.4), unit.out = "photon")
## 1092 possibly 'bad' values in smoothed spectral response

From "raw_mspct" objects

descriptor <- 
  which_descriptor(getWhenMeasured(white_LED.raw_mspct$light))
irrad01.spct <- 
  s_irrad_corrected(x = white_LED.raw_mspct,
                    descriptor = descriptor,
                    correction.method = MAYP11278_ylianttila.mthd)
irrad01.spct
## Object: source_spct [1,421 x 2]
## Wavelength range 251.16-898.81 nm, step 0.43-0.48 nm 
## Label: light: Nichia.horticulture.5000K 
## Measured on 2019-06-25 14:03:10.946064 UTC 
## Variables:
##  w.length: Wavelength [nm]
##  s.e.irrad: Spectral energy irradiance [W m-2 nm-1] 
## --
## # A tibble: 1,421 × 2
##    w.length s.e.irrad
##       <dbl>     <dbl>
##  1     251.  0.00101 
##  2     252.  0.00181 
##  3     252.  0.000342
##  4     253.  0.00102 
##  5     253.  0.000757
##  6     254.  0.000777
##  7     254   0.000750
##  8     254.  0.000939
##  9     255.  0.00117 
## 10     255.  0.00198 
## # ℹ 1,411 more rows
autoplot(irrad01.spct, unit.out = "photon")

getWhenMeasured(irrad01.spct)
## [1] "2019-06-25 14:03:10 UTC"
getWhereMeasured(irrad01.spct)
## # A tibble: 1 × 3
##     lon   lat address
##   <dbl> <dbl> <chr>  
## 1    NA    NA <NA>
getWhatMeasured(irrad01.spct)
## [1] "light: Nichia.horticulture.5000K"
cat(comment(irrad01.spct))
getInstrDesc(irrad01.spct)
## Data acquired with 'MayaPro2000' s.n. MAYP11278
## grating 'HC1', slit '010s'
## diffuser 'unknown'
getInstrSettings(irrad01.spct)
## integ. time (s): 0.247, 2.47
## total time (s): 5.19, 7.42
## counts @ peak (% of max): 92.5

Measurement of spectral transmittance

As for irradiance, transmittance estimates are affected by dark electrical noise, dynamic range and stray light. Random electrical and thermal noise can be averaged out by repeated measurement. Dynamic range depends both on the noise floor and on the resolution of the analogue to digital conversion electronics of the instrument. A way to increase the effective dynamic range is bracketing integration time and splicing/merging the resulting spectra. This is what in digital photography is called HDR or high dynamic range images obtained by merging sets of images obtained at a series of different exposure values. In addition to improved optical design, stray light can be corrected for if it is possible to measure it by itself. This can be achieved by use of an optical filter that transmits radiation of wavelengths causing stray light, but that blocks radiation of wavelengths we are interested in. An additional correction that can be applied is for the slit function of the instrument. The slit function describes how a single wavelength peak (e.g. similar to a laser beam) broadens at the array detector.

Which corrections are needed and which make little difference depends on the type of instrument, the characteristics of the sample being measured and of the light source and the region of the spectrum we are interested in. Consequently, depending on the circumstances we may want to use different measurement protocols. The high-level functions in the package “guess” the protocol from the set of file names or set of spectra passed as argument. For transmittance and reflectance calculations, three named members are recognized as different measurements in a protocol: "sample", "reference", and "dark", each of these can be bracketed for integration time. In the case of file names for any of these three members, a vector of file names, is interpreted as a bracketed measurement. In the case of collections of spectra, raw_spct with multiple counts columns are interpreted as bracketed. Of course, at least "sample" and "reference" members are required as input. If either or both are missing an empty filter_spct object is returned.

Protocol sample spct reference spct filter spct dark spct signal range speed
minimal short short - - < 5e1 ?? 1/1
normal short short - short < 1e2 1/3
nor. brkt short + long short + long - short + long < 1e3 1/33
(corrected) short short short short < 1e3 1/5
(cor. brkt) short + long short + long short + long short < 5e3 1/55

The last two protocols, using a "filter" measurement seem unlikely to be useful in practice, but I intend to do some tests in the future to test this. Function acq_fraction_interactive() currently implements only the nornal protocol with and without bracketing.

Examples

In special cases we may need to set a locale matching the language used in the files for month names is not English. In most cases month names are in English, an time zones and decimal markers will be inferred from the data files themselves, making it unnecessary to explicitly pass a locale definition as argument to functions.

From files

The first example is the one listed as “minimal” in the table above, we use two spectra, one for sample and one for reference, using for the dark correction pixels expected to receive no true excitation. For example, those in the UVC region of the spectrum when the light source is sunlight at ground level.

We set a list of file names to be read, with members named according to their role in the measuring protocol.

From a "raw_mspct" object

A suitable "raw_mspct" can be acquired directly from a spectrometer using functions in this package, or built by reading files obtained using Ocean Optics SpectraSuite software. In the case of spectral objects we can query the measurement date and use it to find an instrument descriptor containing a valid calibration. Although for transmittance calculation the calibration multipliers for irradiance are not used, the calibrated wavelengths are. If the wavelength calibration stored in the spectrometer’s non-volatile memory is valid, it is not necessary to pass a descriptor argument to the function.

descriptor <- 
  which_descriptor(getWhenMeasured(blue_filter.raw_mspct$sample))
tfr01.spct <- 
  s_fraction_corrected(x = blue_filter.raw_mspct,
                       descriptor = descriptor,
                       correction.method = ooacquire::MAYP11278_ylianttila.mthd,
                       dyn.range = 3e2)
tfr01.spct
## Object: filter_spct [2,068 x 2]
## Wavelength range 187.82-1117.14 nm, step 0.41-0.48 nm 
## what label: sample
## user.label label: UQG_Blue 
## Measured on 2016-11-28 14:46:25.335316 UTC 
## Rfr (/1): NA, thickness (mm): NA, attenuation mode: NA.
## Variables:
##  w.length: Wavelength [nm]
##  Tfr: Total spectral transmittance [/1] 
## --
## # A tibble: 2,068 × 2
##    w.length   Tfr
##       <dbl> <dbl>
##  1     188.    NA
##  2     188.    NA
##  3     189.    NA
##  4     189.    NA
##  5     190.    NA
##  6     190.    NA
##  7     191.    NA
##  8     191.    NA
##  9     192.    NA
## 10     192.    NA
## # ℹ 2,058 more rows
autoplot(tfr01.spct)
## Warning in range_check_Tfr(x, strict.range = strict.range): Off-range
## transmittance values [0.000823..1.181888] instead of [0..1]
## Warning in range_check_Tfr(x, strict.range = strict.range): Off-range
## transmittance values [0.000823..1.181888] instead of [0..1]
## Warning in range_check_Tfr(x, strict.range = strict.range): Off-range
## transmittance values [0.000823..1.181888] instead of [0..1]
## Warning in range_check_Tfr(x, strict.range = strict.range): Off-range
## transmittance values [0.000823..1.181888] instead of [0..1]
## Warning in range_check_Tfr(x, strict.range = strict.range): Off-range
## transmittance values [0.000823..1.181888] instead of [0..1]
## Warning in range_check_Tfr(x, strict.range = strict.range): Off-range
## transmittance values [0.000823..1.181888] instead of [0..1]

The light source used does not emit enough light across the whole range of wavelengths that the spectrometer measures. Values for many wavelengths are replaced by NA’s based on the expected noise, but still some out-of-range transmittance value trigger warnings. We can “clip” the spectrum to avoid them.

tfr01.spct <- clip_wl(tfr01.spct, range = c(450, 1100))
autoplot(tfr01.spct, unit.out = "photon", w.band = VIS_bands())

getWhenMeasured(tfr01.spct)
## [1] "2016-11-28 14:46:25 UTC"
getWhereMeasured(tfr01.spct)
## # A tibble: 1 × 3
##     lon   lat address
##   <dbl> <dbl> <chr>  
## 1    NA    NA <NA>
getWhatMeasured(tfr01.spct)
## $what
## [1] "sample"
## 
## $user.label
## [1] "UQG_Blue"
cat(comment(tfr01.spct))
getInstrDesc(tfr01.spct)
## Data acquired with 'MayaPro2000' s.n. MAYP11278
## grating 'HC1', slit '010s'
## diffuser 'unknown'
getInstrSettings(tfr01.spct)
## integ. time (s): 0.488, 4
## total time (s): 20, 20
## counts @ peak (% of max): 75

From a raw_mspct object with a known reference

In contrast to the previous example, we will here assume that the reference is not 100% clear, and that we know its transmittance to be 0.95.

descriptor <- 
  which_descriptor(getWhenMeasured(blue_filter.raw_mspct$sample))
tfr02.spct <- 
  s_fraction_corrected(x = blue_filter.raw_mspct,
                       ref.value = 0.95,
                       descriptor = descriptor,
                       correction.method = MAYP11278_ylianttila.mthd,
                       dyn.range = 3e2)
tfr02.spct <- trim_wl(tfr02.spct, c(460, 1000))
## Warning in range_check_Tfr(x, strict.range = strict.range): Off-range
## transmittance values [0.000823..1.181888] instead of [0..1]
tfr02.spct
## Object: filter_spct [1,213 x 2]
## Wavelength range 460-1000 nm, step 0.13-0.47 nm 
## what label: sample
## user.label label: UQG_Blue 
## Measured on 2016-11-28 14:46:25.335316 UTC 
## Rfr (/1): NA, thickness (mm): NA, attenuation mode: NA.
## Variables:
##  w.length: Wavelength [nm]
##  Tfr: Total spectral transmittance [/1] 
## --
## # A tibble: 1,213 × 2
##    w.length   Tfr
##       <dbl> <dbl>
##  1     460  0.908
##  2     460. 0.908
##  3     461. 0.901
##  4     461. 0.888
##  5     462. 0.884
##  6     462. 0.880
##  7     462. 0.865
##  8     463. 0.862
##  9     463. 0.852
## 10     464. 0.840
## # ℹ 1,203 more rows
autoplot(tfr02.spct)

In the next example, we use a spectrum to describe the reference’s spectral properties.

In contrast to the previous example where we calculated transmittance, we will here calculate reflectance assuming that the reference is not 100% white, and that we know its spectral reflectance.

descriptor <- 
  which_descriptor(getWhenMeasured(blue_filter.raw_mspct$sample))
rfr01.spct <- 
  s_fraction_corrected(x = blue_filter.raw_mspct,
                       ref.value = as.reflector_spct(white_body.spct) * 0.97,
                       descriptor = descriptor,
                       correction.method = MAYP11278_ylianttila.mthd,
                       dyn.range = 3e2,
                       qty.out = "Rfr",
                       type = "total")
rfr01.spct <- trim_wl(rfr01.spct, c(460, 1000))
## Warning in range_check(x, strict.range = strict.range): Off-range reflectance
## values [0.000823..1.181888] instead of [0..1]
rfr01.spct
## Object: reflector_spct [1,213 x 2]
## Wavelength range 460-1000 nm, step 0.13-0.47 nm 
## what label: sample
## user.label label: UQG_Blue 
## Measured on 2016-11-28 14:46:25.335316 UTC 
## Variables:
##  w.length: Wavelength [nm]
##  Rfr: Total spectral reflectance [/1] 
## --
## # A tibble: 1,213 × 2
##    w.length   Rfr
##       <dbl> <dbl>
##  1     460  0.908
##  2     460. 0.908
##  3     461. 0.901
##  4     461. 0.888
##  5     462. 0.884
##  6     462. 0.880
##  7     462. 0.865
##  8     463. 0.862
##  9     463. 0.852
## 10     464. 0.840
## # ℹ 1,203 more rows
autoplot(rfr01.spct)

autoplot(tfr02.spct)

Assembling spectral data for an object

Once we have total spectral reflectance and total spectral transmittance, we can compute internal spectral transmittance and spectral absorptance. These are straightforward calculations when the spectra have been measured at the same wavelengths, which is not the case when using a two-channel array spectrometer. In either case, to create an object_spct object we can use function merge2object_spct() from package ‘photobiology’, which maps the spectra to the same wavelength values if needed.

Irradiance calibration from Ocean Optics

Data provide by Ocean Optics in calibration files is expressed independently of the entrance optics. The true calibration multipliers to apply to linearised counts per second data need to be calculated based on the light collecting area of the cosine diffuser used. First step is to read the file with function read_oo_caldata() to obtain an object of class generic_spct and the second step is to convert with function oo_calib2irrad_mult() this object into an object of class calibration_spct by converting the calibration values into multipliers expressed in the correct unnits and matched to the cosine diffuser used.

Calculating calibration constants

Applying calibration constants

High level functions

Defining new protocols

You can compose from simple functions different measurement protocols and data processing sequences. This is the reason why they have been designed as a set of small functions, each one doing a single operation.

Please see the example scripts installed with the package.