Subject: Fwd: Spectral extraction from JEM-X. Algorithm description ----- Forwarded message from nl@spacecenter.dk ----- Subject: Spectral extraction from JEM-X. Algorithm description Hello all: Working with the extraction of flux values I have gradually realized that the problem is quite hairy, and it is not always obvious where in the sequence of steps the various effects should be handled. I have therefore written the attached note which I hope you will read with a critical eye and point out to me where things may be improved. There are also some data which will be needed and which I dont think we have in the OSA system at this moment: 1) A table describing the 'curtain' gain correction done on-board (Carl, Soren ?) 2) A table describing the 'curtain' variation of the position rms in X and Y. (Carl ?) 3) A function describing the photoelectron range contribution to position uncertainty at energies above 10 keV (Carl ?) 4) The values of the 'upper pha-cutoff' as function of mission time (Soren). We need all the above for both JEM-X1 and JEM-X2. with best regards Niels ******************************************************************** Note on JEM-X spectral extraction using pif-maps Niels Lund. 20060708 The issue is how to calculate the effective area (number of pixels) used for extracting a source flux at a particular energy in the case where the detector gain is varying in time. We also want to take into account the spatial gain variations across the detector. The gain affects the position resolution. Therefore the pif-map evaluated for a low gain situation will be more smeared than the map evaluated for a high gain situation. The spatial gain variations is described by a gain map which is constant in time, at least within a specific science window. It can therefore in principle be included in the standard pif-map evaluation. We shall assume that the temporal changes in the detector gain through the observation is small enough that we can generate a time average of the pif-maps corresponding to the different gain situations and use this average for selecting fixed pixel sets to be used for signal and background extraction throughout this observation. We shall treat the gain as a discrete variable with logarithmic steps of a few percent. I have divided into a number of steps the algorithm for the generation of a gain compensated pif-map for a specific source position and for derivation of a flux value for this source at the specific keV energy interval (I expect that it will be significantly easier to write the algorithms based on a linear keV scale rather than based on the PI-values. We can convert back to the PI-scale at the end): 1) Set up the sequence of time dependent gain values relevant for this observation: TG_0, TG_1, .. , TG_n. TG_0 is at the start of the observation, TG_n at the end. If there are no grey filter changes during this observation the intermediate values can be spaced uniformly in time at roughly 5 minute intervals (TBC). However, if there are strong (TBD) grey filter changes we must introduce time interval boundaries where these changes take place. Set up an array of effective live times, LiveT_i, for each TG-interval. 2) For each of the above TG-values set up an extended sequence of gain values, g_0, g_1, .. , g_m, extending to lower and higher gain values to cover also the spatial gain variations below and above the values in the TG-sequence. These two steps are independent of the source position and can be done prior to going through a source loop. 3) Build a sequence of 'comet-maps' describing the two dimensional scatter of probable photon detection positions around the position where the photon actually enters the detector at the window level. Each map corresponds to rms uncertainties in detector-X and -Y corresponding to a gain value between g_0 and g_m, and to the energy at which this pif-map is constructed. The comet maps are of course also specific for each source position. --- rms values We have a relation giving the rms in detector-X and detector-Y as function of the pha value. (Until now we have not considered the difference between a keV- and a pha-scale, but this is just the issue we now want to tackle). The rms values depend on the pha-values which will depend on the local (spatial) gain, but in addition there is a variation in localization accuracy which depend on the position and is correlated (or maybe anti- correlated ?) with the 'curtain variation' of the gain. We must find this variation and include it in the rms evaluation. In the low energy regime the rms values may hopefully be described as: Xrms_k = Xrms[X] / sqrt(pha_k); Yrms_k = Yrms[Y] / sqrt(pha_k); where k is the pixel index, and Xrms and Yrms are tabulated as functions of detector X and Y. At the high energy end we may need to add (in quadrature) a term describing the position uncertainty arising from the finite range of the photoelectrons. This term, of course depend on E (keV) and not on pha. 4) Determine the minimum and maximum pha value corresponding to each element in the TG-sequence. The action in the next step depends on these pha values: 5a) If (pha_min < pha_nom10keV (10 keV TBC)) build a pif-map for the full detector area for the given energy and source position*). Add this pif-map weighted by the live time spent at this gain value to a sum map. If either the delta map or the sum map contains -1 in a pixel position set the corresponding output pixel to -1 in both maps. Save the delta map for later. 5b) If ((pha_min >= pha_nom10keV) && (pha_max <= pha_cutoff)) use the nominal pif-map evaluated without regard to the spatial or time dependent gain values for the current gain interval. Add this pif-map weighted by the live time spent at this gain value, to the sum pif_map. If either the delta or the sum map contains -1 in a pixel position set the corresponding output pixel to -1 in both maps. Save the delta map for later. 5a) If (pha_max > pha_cutoff) use the nominal pif-map evaluated without regard to the spatial or time dependent gain values for the current gain interval. Add this pif-map weighted by the live time spent at this gain value to the sum pif_map. If either the delta or the sum map contains -1 in a pixel position set the corresponding output pixel to -1 in both maps. Additionally, if the pha value at the current energy and local gain value is larger than pha_cutoff, set the correspon- ding pixel to -1 in both maps. Save the delta map for later. *) in the old lib_pif subroutine, sh_xy, the same values for the position uncertainties (Xrms and Yrms) were used for all pixels and throughout the full observation duration. Now, sh_xy should use values for Xrms and Yrms which are matched to the gain of each individual pixel and to the overall detector gain value at the moment of the photon arrival - the gain being quantized in 3 % (TBC) steps. When step 5 is completed for all time intervals and gain values we have a weighted sum pif-map for the source which takes into account both time and the spatial gain variations of this observation. 6) Call the j_pif_limits subroutine. This subroutine must determine the optimal limit for the pif value above which the the pixel should be used for signal extraction, and another pif value, below which the pixels should be used for background determination. These limits are applied to the weighted sum map to select the pixel sets which will be used for source flux extraction. The same, fixed pixel sets will be used for source and background extraction throughout the observation, however, the average electronic efficiency and the average geometric source extraction efficiency will be evaluated at the instants corresponding to the TG-series and applied in the calculation of the average source flux for the entire observation (for spectral extraction) and linearly interpolated in time and applied to each time bin if a light curve is requested. j_pif_limit returns the pif limit values, the pixel maps for source and background selection, and arrays with the values for the electronic efficiency and for the mean value of the source illumination value for both the source and the background pixel set. --- Electronic efficiency: The electronic efficiency, EC_k, for pixel k is calculated based on the raw pha value which is derived from the PI-energy value and the effective gain, G_k, for the pixel. G_k is the product of TG_i (time dependent gain), S_k (spatial gain) and C_k (curtain function gain). The mean electronic efficiency, mean_EC, is just the simple mean over the pixel set: mean_EC = SUM{EC_k} / K; K is the total number of pixels in the set. The delta pif-maps from step 5 above are used by j_pif_limit in the evaluation of the mean illumination values. --- Mean illumination values: Rather than just being a simple mean of the illumination values, I_k, we will use a weighted mean, where we use the electronic efficiency, EC_k, as a weight factor: mean_Illum = SUM{I_k * E_k * C_k} / SUM{E_k * C_k}; The mean_EC and mean_Illum values are placed in arrays with a value for every interval border in the gain time series, G_i. 7) Source flux calculation: (Recall that this flux calculation refers to a specific energy interval, and the whole procedure above is repeated for all energy intervals). 7a) Let C_s be the total number of counts in this energy interval collected from the 'signal' set of pixels. Let P_s be the total number of pixels in 'signal' set of pixels. Let C_b be the total number of counts in this energy interval collected from the 'background' set of pixels. Let P_b be the total number of pixels in 'background' set of pixels. Then the excess signal counts from the signal pixels for this time interval can be written as: Excess_counts/s = (C_s - C_b * P_s / P_b) / LiveT_i; with a variance: Variance_on_counts/s = (C_s + C_b * (P_s / P_b)**2) / LiveT_i**2; 7b) The calculated excess is smaller than its ideal value due to two effects: electronic (in-)efficiency and geometric (in-)efficiency. The geometric efficiency is the difference between the mean illumination of the signal pixels (which is less than unity) and the mean illumination of the background set (which is larger than zero): mean_Geo_Eff = mean_Illum_signal - mean_Illum_background; We need to compensate for these inefficiencies to get the real excess: Real_excess_counts/s = 'Excess_counts/s' / (mean_Geff * mean_EC); with variance: Real_excess_variance_on_counts/s = 'Variance_on_counts/s' / (mean_Geff * mean_EC)**2; Note that the mean_EC is not so much a mean over time as a mean over the pixel set. The time intervals are chosen such that the gain, and therefore the electronic efficiency does not change much within each time interval. 7c) The effective area of the detector which have collected this real excess is taken to be the number, N_geo, of pixels within the signal pixel set which is indicated to be illuminated in the geometric pif-map (with only zeroes and ones for the illumination values): Flux_per_pixel = 'Real_excess_counts/s' / N_geo; Variance_per_pixel = 'Real_excess_variance_on_counts/s' / N_geo**2; That's all! ****************************** --- Uncertainties The above algorithm contains contains two phases: a) The modelling phase - all steps except step 5. These steps only depend on the modelling, they are independent of the photon count statistics. These steps primarily will contribute to the systematic errors in the flux determination (because the illumination model is not complete and correct in all details), but there could also be situations where statistical errors could appear, in cases where only a very small number of pixels can be used for the signal or for the background. b) The 'count' phase. Step 5 - this is where we are dependent on physical counts. In this step we need to take care of situations where the user tries to apply the algorithm with too narrow bins in time and energy such that the background cannot be determined (zero background counts). But also for more normal situations it will be important that we makes clear to ourselves what data we should carry along to allow combination of data from narrow energy bands into wider ones, and from small time intervals to the full science window f.i. If we follow Nicola Petrous reasoning it should be sufficient to have the flux per pixel and the variance per pixel for each energy/time slot, but I feel uneasy about adding a large number of very noisy values and believing that a well determined mean result will emerge. ----- End forwarded message ----- ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. ================================================================================================ This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is prohibited. If you received this message in error, please delete it from your system and notify the sender. E-mails can be altered and their integrity cannot be guaranteed. ESA shall not be liable for any e-mail if modified. =================================================================================================