--- title: "User Guide" subtitle: "`ooacquire` `r packageVersion('ooacquire')`" author: "Pedro J. Aphalo" date: "`r Sys.Date()`" output: rmarkdown::html_vignette: toc: yes vignette: > %\VignetteIndexEntry{User Guide} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ## Introduction In this package we define higher-level functions using packages 'rOmniDriver' and the 'r4photobiology' suite as bases. These functions conform a sort of "grammar for spectral data acquisition". We provide a separate function for each conceptual step in the process leading from raw counts to final spectral data---e.g. spectral irradiance. Some higher level functions are included as well, serving also as examples of how to implement specific protocols. The package includes functions for direct data acquisition and near real-time processing of spectra, plus functions that read raw-detector-counts and instrument settings from text files output by programs and instruments from Ocean Optics. After acquisition or file reading the processing of the raw spectral data is the same. For data acquisition, the initial steps are opening a connection to an instrument and retrieving a descriptor including model type, optical configuration and possibly calibration data from its non-volatile memory. These first steps in most cases are needed only at the start the R session where data will be acquired. The drivers from Ocean Optics support multichannel instruments and simultaneous connection to multiple spectrometers. This is also supported by this package. The steps for the acquisition of one raw spectrum are to 1) define the settings to be used, 2) optionally adjust or tune some of these settings automatically, 3) acquire the spectrum, 4) convert the raw counts into linearised counts per second. In most cases we need acquire more than one raw spectrum, frequently following a certain protocol. There are two possibilities: a) they are conceptually a single observation, such as multiple scans to be averaged to reduce noise or acquired with different integration times (bracketing) and to be merged to increase the dynamic range, and b) sets of spectra in which each spectrum represents a different observation, such as a 'measurement' spectrum and a 'dark' spectrum to be used to correct it. The distinction may be not that easy to grasp, but in case a) we are measuring a single thing, what we are measuring remains unchanged for all spectra acquired in a set---e.e. we are measuring repeatedly, possibly changing the instrument settings. In case b), although the different spectra may contribute to a single processed spectrum, they are based on measuring a different thing, say a light source and a paired dark reading. In case a) all the acquired raw spectra are stored as columns in the same `raw_spct` object, while in case b) the different conceptual spectra are each stored in a separate `raw_spct` object, member of a `raw_mspct` collection of spectral objects. As a) and b) are usually combined, this approach allows and easy distinction of the two situations, and greatly simplifies the coding of later data processing steps. When spectral data are acquired with other software and read from files into R, this same approach is used for raw data storage. The role of the different files is determined by the structure of a named list used as argument. Files do not contain a full descriptor of the instrument and consequently the instrument descriptor need to be supplied as an additional argument. Following the approach of the 'r4photobiology' suite, we store all available metadata as attributes to the same objects where the acquired spectral data themselves are stored. The stored metadata includes a descriptor of the instrument and the instrument settings used. These attributes are an addition to metadata that is normally held in spectral objects of classes defined in package 'photobiology', such as time, location, and label for the measured data. Furthermore, metadata is added at each step, allowing the tracing of the origin and processing of the data. This allows the functions to detect mistakes like an user attempting to linearise raw-counts data that have been already linearised. This safeguard works even for data linearised in the spectrometer software and read from files. In all cases a wavelength calibration is used to map the pixels to wavelengths. Wavelength calibration can be that retrieved from the instruments' non-volatile memory or supplied directly by to user. Different further steps are needed depending on the quantity measured and method. In the case of irradiance, a calibration needs to be applied to the counts-per-second spectral data. Usually calibration data is also stored in the non-volatile memory of the spectrometer, but in some cases it is convenient to apply special algorithms or calibration data that cannot be retrieved from the instrument itself. Functions for both the normal and special cases are also included in the package. Here it is important to remember that a calibration will be valid only if the raw spectral data processing steps are exactly the same for the spectral data used for calibration and the measured data. **This in practice means that calibrations supplied by instrument manufacturers will almost never be usable with all the methods implemented in this pacakge. However, methods are provided also for this case.** When doing a calibration "manually" by simply measuring a standard lamp you will need the reference lamp spectrum and the raw detector-counts data from the calibration event. Doing the calibration manually allows the use of any of the different measuring protocols and correction algorithms implemented. This permits one to obtain valid calibration coefficients for the data-acquisition protocol one adopts for measurements.** On the other hand, this also means that the functions in the present package, as each of them fulfils a simple and well defined step of data processing, are ideal for _in silico_ testing of alternative data processing approaches and measuring protocols. ## Functions | function name | role | | :---------|:----------------------------------------------------------| | **data acquisition** | | | `start_session` | search for connected instruments and obtain a handle to the driver | | `end_session` | close all connections and release handle | | `list_instruments`| list all connected instruments | | `get_oo_descriptor` | get a descriptor for one of the possibly several connected instruments | | `set_descriptor_wl` | replace the wavelength values | | `set_descriptor_bad_pixs` | replace the indexes to bad array detector pixels | | `set_descriptor_nl` | replace the function used to correct for array non-linearity | | `set_descriptor_calib_mult` | replace the vector of irradiance calibration multipliers | | `set_descriptor_integ_time` | replace the vector defining the range of valid integration times | | `get_oo_settings` | query the instrument to obtain its current settings | | `acq_settings` | build a list of setting values | | `tune_acq_settings` | tune the integration time and number of scans to the current radiation conditions | | `acq_raw_spct` | take one measurement (data point) | | `acq_raw_mspct` | take a sequence of spectral measurements (several data points) | | `hs_acq_raw_mspct` | take a sequence of spectral measurements at high speed | | `acq_irrad_interactive` | interactively acquire spectral irradiance or fluence and save data and plots to disk | | `acq_fraction_interactive` | interactively acquire spectral transmittance or reflectance and save data and plots to disk | | `acq_fraction_pulsed_interactive` | as above but using a pulsed light source | | **data procesing** | | | `trim_counts` | replace values from saturated array pixels with NAs | | `bleed_nas` | replace values from pixels neighbouring NAs with NAs | | `linearize_counts` | linearise the raw counts | | `skip_bad_pixs` | replace bad pixels with interpolated raw counts | | `ref_correction`| subtract a reference (e.g. dark signal) from a measured spectrum | | `photobiology::fshift`| can be used to subtract a region of the same scan as an internal "dark" reference | | `raw2cps` | methods to convert raw counts into counts per second. The method specialization for `raw_spct` returns a `cps_spct` object | | `merge_cps` | method to do HDR merge of bracketed scans | | `slit_function_correction` | apply a correction for the 'long tails' of the slit function | | `filter_correction` | apply a correction for stray light based on the use of UV-cut-off or other selective filters | | `no_filter_correction` | apply a correction for stray light not based on the use of UV-cut-off or other selective filters | | `uvb_corrections` | apply corrections for reducing the impact of stray light and slit function on counts-per-second estimates. This function intelligently adjusts the processing method used based on the spectral data item supplied as `x` argument | | `which_descriptor` | select an instrument descriptor from a list based on the dates between which a calibration is valid | | `photobiology::cps2irrad` | methods to convert counts per second into spectral irradiance | | `photobiology::cps2Tfr` | methods to convert counts per second into transmittance | | `photobiology::cps2Rfr` | methods to convert counts per second into reflectance | | **calibration** | | | `compute_irrad_calibration` | compute irradiance calibration multipliers from raw counts data and polynomial coefficients from standard lamp certificates using any of the different measuring protocols and data processing methods implemented in the package | | `oo_calib2irrad_mult` | compute irradiance calibration multipliers from Ocean Optics provided calibration | | **file reading** | | | `read_oo_ssdata` | read data from a 'SpectraSuite' output file with raw-counts data | | `read_oo_ovdata` | read data from a 'OceanView' output file with raw-counts data | | `read_oo_pidata` | read data from a 'XXXXXXXXX' output file with raw-counts data from a Raspberry Pi board | | `read_files2mspct` | read multiple data files and assemble from them into a `raw_maspct` object compatible to that obtained through direct data acquisition | | `read_oo_caldata` | read irradiance calibration data from an Ocean Optics supplied calibration file | | `set_oo_ssdata_settings` | set the instrument settings attributes by decoding the header of text files saved from SpectraSuite | | `set_oo_ssdata_descriptor` | set the instrument descriptor attribute by decoding the header of text files saved from SpectraSuite and optionally merging these incomplete data with that in a _descriptor_ supplied as argument | | `merge_raw_mspct` | Merge `raw_spct` objects members of a single `raw_mspct` object into a `raw_spct` object with multiple columns | | **high level method** | | | `raw2corr_cps`| higher level function that returns **corrected** counts per second useful for calculating spectral reflectance and spectral transmittance.| | `s_irrad_corrected` | high-level method with specializations for files and `raw_mspct` objects returning spectral irradiance in a `source_spct` object. | | `s_fraction_corrected` | high-level method with specializations for files and `raw_mspct` objects returning spectral transmittance in a `filter_spct` object or spectral reflectance in a `reflector_spct` object. | ## Workflow The workflow consists in several steps. First all one needs to start a new connection to the driver by means of `start_session()` and disconnect at the end of the measuring session with `end_session()`. To learn which instruments are connected and which numerical index pints to each one uses `list_instruments()`. The indexes start at zero, so if there is only one instrument connected one can rely on its index being 0. The same applies to channels, for multichannel instruments. Once we know which instrument we want to address and its index, we must create an instrument descriptor with function `get_oo_descriptor()`. From this point onwards we will use this _descriptor_ to address the instrument. The descriptor contains information about the instrument obtained initially by querying it. Instruments may have some pieces of information unavailable (e.g. not stored in there non-volatile memory), or the user may want to override the available information with that from a more recent or better calibration. `set_descriptor_nl()`, `set_descriptor_wl()`, `set_descriptor_bad_pixs()` and `set_descripto_integ_time()` can be used to set or override the function used for correcting for the non-linearity of the sensors, the wavelength values corresponding to each pixel, a vector of indexes (starting at one) corresponding to bad pixels in the array (pixel known to return bad data) and the range of integration times in ms that are valid input for the instrument. __It is always good to check against the specifications of the instrument whether the values stored in its As there are several settings needed for the acquisition of a spectrum, we will store sets of settings into objects that can reuse and modify. To create these objects we use function `acq_settings()`. The _settings_ include both actual parameters that need to be set in the instrument by sending commands and parameters that our software may use to calculate suitable values for such settings. This function is quite flexible and allows the use of bracketing. In other words a single observation can consist in more than one data acquisition, each of these using different setting, with the aim of merging them into a single spectrum. Some settings need to be adjusted according to the current irradiance level so as to make full use of the dynamic range of the instrument. For this we use function `tune_acq_settings()`, which updates the settings stored in a list as created by `acq_settings()` or as returned by a previous call to `tune_acq_settings()`. Function `get_oo_settings()` can be used to query the instrument for the settings currently in use. To acquire spectral data we can use one of two functions, `acq_raw_spct()` to acquire a single spectrum, or `acq_raw_mspct()` to acquire a collection of spectra according to a user supplied protocol. In cases when we use HDR (bracketing) or _overexpose_ some part of the spectrum the first data-processing step is to replace the data from the saturated pixels with `NA`'s using function `trim_counts()`---as the settings used in the acquisition and descriptor of the instrument are stored in the spectral object, this function needs only to be supplied a `raw_spct` object, unless the user wants to discard more pixels. As pixels neighbouring _overexposed_ pixels are disturbed by the charge _overflowing_ from the saturated wells, these pixels need also to be set as NAs by means of function `bleed_nas`. If the instrument array contains bad pixels, it is best to replace the data produced by them at this early stage. Function `skip_bad_pixs()` is useful here and it replaces the bad data with the average of the readings from the neighbouring pixels. The third step after data acquisition, unless the linearisation is done by the instrument, is to linearise the acquired _valid_ raw counts by means of method `linearize_counts()`. The non-linearity of the detector is related to how "full" are its electron wells, irrespective of whether the electrons originate from measured photons or any other source, consequently the linearisation function should be applied to raw detector counts at this point. CCD arrays, especially if not cooled, have a significant dark signal. The best approach to remove this _background_ signal is to measure it---to take a spectral reading with the array not exposed to light. This _dark spectrum_ is them subtracted from the one measured in the light. An alternative, with advantages and disadvantages, is to use the signal from pixels corresponding to wavelengths **known** _a priori_ to be absent in the light source being measured. This calculation is done with the **linearized** raw counts by means of method `photobiology::fshift()` with a pair of suitable wavelength values for `range`. In the case of measuring transmittance (and absorbance) and reflectance we need to also measure a _clear_ and _white_ reference respectively. The raw counts after linearisation followed by subtraction of the dark signal are still expressed in number of counts per integration period. As the duration of the integration time is variable, we need to re-express the raw counts as counts-per-second (_cps_). Once more, the integration time used for the acquisition is stored in the `cps_spct` object and the conversion can be done by method `raw2cps()` with just this object as input. The values returned, expressed as counts per second, are linearly proportional to the rate at which photons imping each detector pixel. Be aware that as detector quantum yield depends on wavelength, a different multiplier is needed for each pixel to be able to convert the counts-per-second data to units of photons-per-unit-time. If bracketing for HDR has been used during acquisition, at this point, the values of the _cps_ for each of the bracketed spectra are expressed on the same scale. This means that they can simply be spliced together. The simplest approach possible, used by method `merge_cps()` is to replace the clipped data from saturated pixels (and possibly their neighbours) in the longer integration with data from the same pixels obtained with shorter integration. As we used `trim_counts()` and `bleed_nas()` on the raw counts, those saturated pixels have already been replaced with `NA`'s making the operation easy to implement. After the preceding steps, if the _cps_ or merged _cps_ values are from an irradiance measurement the spectral data are now ready to be multiplied by the calibration constants corresponding to each individual pixel. If the values are to be used for calculation of transmittance or reflectance, all what is left to do is simple arithmetic. ## Additional corrections ### Apply slit function correction The slit function describes how monochromatic light "reaches" adjacent pixels which should have been in total darkness had the monochromator been perfect. This is a subtle correction, which will affect only regions of a spectrum where the change in signal strength with wavelength is very steep, such as the UV region of a solar spectrum or a very narrow absorption peak. The slit function of a spectrometer can be estimated by measuring laser beams at different wavelengths and fitting suitable functions to the observed data. Here we explain how to apply such a function. The slit function is the result of an optical effect, and consequently independent of integration time and other detector settings. Consequently, can be applied to either the merged _cps_ values or after application of a calibration. **Obviously, the stage at which the slit correction is applied needs to match the stage of processing of the calibration data used for estimating the slit function.** The correction to apply to a given pixel depends on the readings of nearby pixels. Some authors recommend to apply this correction recursively, however, we follow the simple approach of applying it only once as an approximation. ```{r "setup", include=FALSE, cache=FALSE} library("knitr") opts_knit$set(cache = FALSE, root.dir = system.file("extdata", package = "ooacquire")) sr.online <- FALSE ``` ## Code examples All examples in vignettes use data and files included in the package. Data files in formats _foreign_ to R, are stored in the `"extdata"` folder, as is the norm for R packages. _To reproduce any of the examples which use files as data input, it is best to make a local copy of the whole `"extdata"` folder._ The files used in examples are organized into sub-folders, but all example code assumes that its in being run with `"extdata"` or a copy of it as the current _working_ folder. The chunk below sets the working directory to point to the exdata folder in the installed package. ```{r, eval=TRUE} folderpath <- system.file("extdata", package="ooacquire") ``` ## Measurement of spectral irradiance Irradiance estimates are affected by calibration errors, dark electrical noise, dynamic range and stray light. In this section we assume that the available calibration is valid. Random electrical and thermal noise can be averaged out by repeated measurement. Dynamic range depends both on the noise floor and on the resolution of the analogue to digital conversion electronics of the instrument. A way to increase the effective dynamic range is be bracketing integration time and splicing/merging the resulting spectra. This is what in digital photography is called HDR or high dynamic range images obtained by merging sets of images obtained at a series of different exposure values. In addition to improved optical design, stray light can be corrected for if it is possible to measure it by itself. This can be achieved by using an optical filter that transmits radiation of wavelengths causing stray light, but that blocks radiation of wavelengths we are interested in. An additional correction that can be applied is for the slit function of the instrument. The slit function describes how a single wavelength peak (e.g.\ similar to a laser beam) broadens at the array detector. Which corrections are needed and which make little difference depends on the type of instrument, the characteristics of the light source and the region of the spectrum we are interested in. Consequently, depending on the circumstances we may want to use different measurement protocols. The high-level functions in the package "guess" the protocol from the set of file names or set of spectra passed as argument. For irradiance calculations three named members are recognized as different measurements in a protocol: `"light"`, `"filter"` and `"dark"`, each of these can be bracketed for integration time. In the case of file names for any of these three members, a vector of file names, is interpreted as a bracketed measurement. In the case of collections of spectra, `raw_spct` with multiple `counts` columns are interpreted as bracketed. Of course, at least a `"light"` member is required as input. If it is missing an empty `source_spct` object is returned. | Protocol | `light` spct | `filter` spct | `dark` spct | signal level | signal range | speed | |:----------|--------------|---------------|-------------|--------------|--------------|-------| | minimal | short | - | - | high | < 5e1 ?? | 1/1 | | normal | short | - | short | high | < 1e2 | 1/2 | | corrected | short | short | short | medium | < 1e3 | 1/3 | | cor. brkt | short + long | short + long | short + long | low | < 1e4 | 1/33 | ### Examples We first load the packages to be used, and set the working directory to the location of the example data files included in the package. ```{r} library(photobiology) library(photobiologyWavebands) library(ggspectra) library(ooacquire) ``` #### From files