Instructions for better data treatment
Dear colleagues,
If you have already had the opportunity of getting data from EMIR, you may have noticed excessive noise in the preprocessed data frames, together with jumps in the signal throughout the data series, which in some cases are severely hampering the overall performance of the instrument. This is especially important when observing faint sources in spectroscopic mode.
Just for clarification, preprocessed files are those output by the EMIR Control System (ECS) derived from the raw data frames. In the frame list, these are the files without the suffix 'raw' in the name. For CDS read mode, they are calculated by subtracting the second read from the first in each sequence, while for the RAMP mode, they contain the fitted slope to the raw data in each sequence multiplied by the integration time of the sequence. These files are the starting point of the subsequent data reduction and are the ones ingested by the EMIR DRP, both online and offline.
From the very beginning we in the EMIR team have been concerned with the noise and lack of stability in the detector data frames and have tried several methods to alleviate these problems. In the course of this work, we recently discovered that the procedure implemented in the ECS to remove the Dark Current and fix the instabilities in the detector data series has not been properly applied. The purpose of this note is to instruct users on how to recover the correction in their EMIR data.
First of all, it is important to note that the flaw is only present in the data taken in RAMP readout mode. Data taken with the CDS mode are unaffected by this. Please check in the corresponding keyword of your fits file, READMODE, to get the read mode of the data before applying the correction method. Second, the gain will be more noticeable in dark areas of the detector, registering only low signal (in between sky lines, outside the target spectra, faint spectrum signal, low background, etc.), so if the data contain bright sources, this correction may not be needed.
Fortunately, the calibration method has been applied properly to the raw frames, which are safely stored on disk, while to derive the slope of each ramp, ECS has incorrectly used the original raw frames, without calibration. The correction consists in reprocessing the raw data frame series to derive the new preprocessed files, from which the reduction should start. To this end, a simple python script (fitramp_ecs.py) is available that performs the method. Needless to say, the raw data set has to be at hand before going ahead with this method. If this is not the case, the user must get in touch with the GTC team to get it.
In the figure below, the result of applying the method is shown in comparison with the same measurements before correction. The plot shows the signal in the J band in three areas of the detector frame, with background only, throughout 20 ramps in a ABBA series before and after the correction. The color code is the same in both cases. Note the reduction in the absolute value and the gain in uniformity along the series.
