Discussion of systematic errors in spectral fitting
From: Silvia MartÃnez Núñez
To: Stefan Larsson
Subject: Systematic Spectral Errors -- comparing old and new RMF and software
Date: Wed, 16 Jun 2004 17:49:13 +0200
Cc: Peter Kretschmar , njw@dsri.dk, Carol Anne
Oxborrow , Jérôme Chenevez , Sami Maisila
Dear All --
I've been testing the new spectral extraction software deliveried by Stefan
yesterday.
I've estimated the systematic errors for two different periods:
1. PV data (calibration period 4 - RMF versions 0004 and 0014).
2. Calibration period 5 - RMF 0009 and 0015.
using the old software 3.5.1 (rmf 0004 and 0009) and new one 4.1.0 (rmf 0014
and 0015). The results are attached in sys.txt. The systematics for the PV
data have decreased significantly with the new software. In the case of the
second calibration period the errors are more or less the same in both cases.
On the other hand regarding the issue of fitting the Crab spectrum to an
absorption model plus a power law I attach the obtained results in the file
crab_fit.txt.
Best wishes,
Silvia. results results
From: Peter Kretschmar
Subject: Re: Systematic Spectral Errors -- comparing old and new RMF and software
Thanks for these efforts Silvia -
question to Stefan and Niels Joergen: can we take the systematic
uncertainties derived by Silvia and recommend using them in the SVR
and the user manual?
Cheers,
Peter
From: Stefan Larsson
Subject: Re: Systematic Spectral Errors -- comparing old and new RMF and software
Very good Silvia.
This is the type of testing we need.
How did you make the estimate of the "Fractional systematic errors"?
Stefan
From: Silvia Martínez Núñez
To: Stefan Larsson , peter.kretschmar@obs.unige.ch
Subject: Re: Systematic Spectral Errors -- comparing old and new RMF and software
Date: Thu, 17 Jun 2004 10:15:36 +0200
Hej Stefan --
> This is the type of testing we need.
> How did you make the estimate of the "Fractional systematic errors"?
What I have done to estimate the fractional systematic errors is the
following:
1. I have fitted the Crab spectrum with wabs*pow fixing the Nh to 0.2 and the
gamma to 2.1 and living free the normalization. Then I have obtained the
ratio between the model and the data.
2. This ratio allowed me to define the fractional systematic errors as the
absolute value of the difference between the ratio and 1 (ABS(1-ABS (ratio)))
for each channel. Finally I have obtained an average value for each 10
channels.
Best wishes,
Silvia.
From: Silvia MartÃnez Núñez
To: Stefan Larsson , peter.kretschmar@obs.unige.ch
Subject: To trust or not the spectral fitting parameters values ? Comparing 3.5.1 with 4.1.0.
Date: Wed, 23 Jun 2004 12:58:45 +0200
Dear Stefan and Peter (and the others) --
I am still fighting with the scientific analysis of my Thesis, uff ...
Last week I decided to compare Cygnus X-3 spectra using the old spectral
extraction software 3.5.1 with the rmf 0004 and the new software 4.1.0 with
the new rmf. To do so, I've created an average spectrum for each software
version over the 38 ScWs I have and I've defined systematic errors as I told
you. I have fitted both spectra (raw spectra can be seen in
raw_x3_spectra.ps.gz where the green dots correspond to the new software)
in the energy range 5 to 25 keV with a gaussian+compst model fixing the sigma
of the line to 0.45, i.e, spectral resolution at 6.5 keV, and then
surprisingly the obtained parameters are a "bit" different.
The obtained parameters are (I attach x3_sum_comp_osa3_osa4.ps.gz to see the
residuals of the fitting. The fitting is good even the chi is very low due to
the systematic errors.):
KTc Tau chi (dof=120)
v.3.5.1 5.2 [-0.5 +0.6] 8.7 +- 1.0 0.18
v 4.1.0 7.7 [-1.3 +2.7] 6.2 +- [1.0,1.2] 0.26
Taking a look directly to the raw data we can see that the obtained count rate
has increased with the new software below channel 100 ( ~ 7.5 keV), if I
constraint the fitting of the spectrum to the energy range 10 to 25 keV and
then I fit only the compst model the obtained parameters are similar in both
cases but with bigger uncertainties than before. The obtained values are:
KTc Tau chi (dof=64)
v.3.5.1 4.6 [-0.6 +1.0] 11 +- [3,6] 0.15
v 4.1.0 4.9 [-0.7 +1.1] 11 +- [3,4] 0.17
Therefore, could we trust the absolute values of the spectral fitting or not ?
Best wishes,
Silvia. figures figures
P.D. Regarding the systematic errors, Stefan did you receive my e-mail with
the description how I estimate them ?
To Peter: how can I estimate the quality of a fitting when I define systematic
errors ?
***********************************************************
Silvia MartÃnez Núñez
From: Peter Kretschmar
Subject: Re: To trust or not the spectral fitting parameters values ? Comparing 3.5.1 with 4.1.0.
To: Silvia Martínez Núñez
Dear Silvia,
> The obtained parameters are (...)
> The fitting is good even the chi is very low due to
> the systematic errors.):
>
> KTc Tau chi (dof=120)
> v.3.5.1 5.2 [-0.5 +0.6] 8.7 +- 1.0 0.18
> v 4.1.0 7.7 [-1.3 +2.7] 6.2 +- [1.0,1.2] 0.26
>
First of all, a reduced chi^2 << 1 means you are overestimating the
errors in your data. We have to look again how you obtain these.
But it is true that the values differ now - they should as before
there was some systematic deviation - or am I missing something.
> Taking a look directly to the raw data we can see that the obtained
> count rate has increased with the new software below channel 100
> ( ~ 7.5 keV), if I constraint the fitting to the energy range
> 10 to 25 keV and then I fit only the compst model the obtained
> parameters are similar in both cases but with bigger uncertainties
> than before. The obtained values are:
>
> KTc Tau chi (dof=64)
> v.3.5.1 4.6 [-0.6 +1.0] 11 +- [3,6] 0.15
> v 4.1.0 4.9 [-0.7 +1.1] 11 +- [3,4] 0.17
>
> Therefore, could we trust the absolute values of the spectral
> fitting or not ?
>
How do you mean this? Are you worried about the difference in
parameters if you just use 10-25 keV compared to the full range?
That's possible and I cannot tell from this if it is significantly
better.
>
> To Peter: how can I estimate the quality of a fitting when I
> define systematic errors ?
>
There is no real solid way of estimating the absolute quality
of a fit in this game. You can only see (e.g. with the F-Test)
if one model seems to fit significantly better than another.
As indicated above, recuced chi^2 should be close to 1 for a good
fit with correct error estimates.
Cheers,
Peter
Date: Thu, 24 Jun 2004 13:21:17 +0200 (MEST)
From: Stefan Larsson
Subject: Re: Systematic Spectral Errors -- comparing old and new RMF and software
To: silvia.martinez@uv.es, peter.kretschmar@obs.unige.ch
Hi Silvia,
Yes I got your mail, just a bit slow to respond :-)
It looks reasonable to me, but I have one more question.
I presume you averaged the ratio over some number of spectra.
Are these with the same or different pointing directions?
Stefan
From: Silvia Martínez Núñez
To: Stefan Larsson , peter.kretschmar@obs.unige.ch
Subject: Re: Systematic Spectral Errors -- comparing old and new RMF and software
Date: Thu, 24 Jun 2004 13:34:43 +0200
Cc: njw@dsri.dk, oxborrow@dsri.dk, jerome@dsri.dk, smaisala@astro.helsinki.fi, stefan@astro.su.se
Hej Stefan --
I have not averaged the ratio over some number of spectra, the ratio was
obtained from only one spectrum with the source in the center of the FOV
(0.13 deg off-axis). Therefore the systematics I obtained are valid for
on-axis sources but many not be valid for off-axis positions. I think that we
need to study the variations on systematic errors against off-axis source
position.
Best wishes,
Silvia.
Date: Thu, 24 Jun 2004 15:32:41 +0200 (MEST)
From: Stefan Larsson
Subject: Re: Systematic Spectral Errors -- comparing old and new RMF and software
To: silvia.martinez@uv.es, peter.kretschmar@obs.unige.ch
Hi again Silvia,
So it means that your estimate will contain also the statistical
noise? If you use both this estimate of the systematic noise and
the statistical noise estimate that come with the spectra you
will overestimate the errors and get too low chi-square (as
Peter pointed out). Unless you make some correction, or ignore
the errors in the file?
Stefan
From: Silvia Martínez Núñez
To: Stefan Larsson
Subject: Re: Systematic Spectral Errors -- comparing old and new RMF and software
Date: Thu, 24 Jun 2004 19:51:12 +0200
Hi Stefan !
> So it means that your estimate will contain also the statistical
> noise?
I do not think so. In my opinion estimating the systematic errors as the
ABSOLUTE VALUE OF (1- ABSOLUTE VALUE of the RATIO) give us
how much for each channel the spectrum deviates from the model, and those
deviations are independent of the statistical error, ¿ or not ?
Best wishes,
Silvia.
Date: Thu, 24 Jun 2004 20:12:46 +0200
From: Peter Kretschmar
Subject: Re: Systematic Spectral Errors -- comparing old and new RMF and software
To: Silvia Martínez Núñez
Buenas noches Silvia -
>
> I do not think so. In my opinion estimating the systematic errors as the
> ABSOLUTE VALUE OF (1- ABSOLUTE VALUE of the RATIO) give us
> how much for each channel the spectrum deviates from the model, and those
> deviations are independent of the statistical error, ¿ or not ?
>
This would be true if your statistical errors were zero.
But if you take a single measurement your datapoint may be
randomly shifted a bit up and down and if I understand what
you did correctly, you take that as systematic error.
Somebody correct me if I say something stupid, but to
estimate systematic errors from this method you better have
first good statistics and then need to do an estimate on
how much the data points deviate from the expected value
beyond what can be explained by statistics alone. Usually
people just use rough estimates by eye, applied to larger
energy intervals.
I'm sorry because I think I told you more or less to do
what you did but without sufficient explanation. I think
it is becoming urgent to set up some reasonable values for
these errors between all of us (maybe sans Carol Anne,
except if she wishes to contribute :-) ). The kind of fits
I've done recently on Crab data should be a good starting
point.
Regards,
Peter