SDAST CHAT MEETING #116: Wednesday 20th September 2006

Participants: Carol Anne Oxborrow, Niels Joergen Westergaard, Stephane Paltani,
	Peter Kretschmar, Silvia Martinez-Nunez, Jerome Chenevez
Summary
1) Date for next SDAST meeting
	Thursday 26th October 9.am to Friday 27th October 15.00 (times approx.)
2) Status for j_cor_gain, j_cor_position, j_dead_time_calc (CAO)
	All three components have been redelivered, and apparently integrated
	without problems. The improved hotspot removal for strong hotspots is
	the only remaining outstanding SPR/SCREW. IC gain history files (OCL
	tables) will be delivered by end of next week.
3) Status for j_ima_iros and j_src_properties (NJW)
	j_ima_iros has been delivered but it gives a segmentation error due to
	a divide-by-zero error. j_src_properties will not be part of the OSA 6.0
	release and should show some progress in a month or so.
4) Status for mosaicking (JC)
	Redelivered j_ima_mosaic to include latest changes to j_ima_iros output
5) Status for scripts (SMN)
	j_correction and j_scripts are separate entities and should be treated
	as such. j_ima_iros is giving problems because of a divide-by-zero 
	error. SPR-5422 on default hotspot removal(rowSelect='&&STATUS<256')
	needs to be updated to include j_ima_shadowgram, j_ima_iros to ensure
	that they can function if users forget the '&&'. Scripts ready to
	deliver since j_correction functions with the test OCL tables.
6) Status for integration (SP)
	SP and ISDC still waiting for the following items: OCL files (CAO);
	JEM-X BTIs (SB); ARFs (NJW). Needless to say these are needed ASAP.
7) ISDC data analysis workshop. no news. SP had to leave the meeting.
8) AI list - Several AI's updated as `ongoing', 'closed' and `OBE'
9) AOB - Soeren and Jerome collaborating on an ATEL about the Galactic Bulge
	observations with Erik Kuulkers.
	
	
Action Items

AI_200906_1:   SB   22/09/2006	Provide ISDC with a list of SCWs where the
				instrument configuration is such that the data
				isn't good for science purposes (BTIs)
				
AI_200906_2	CAO 29/09/2006	Deliver verified OCL files to ISDC

AI_200906_3	NJW 29/09/2006	Deliver new ARFs to ISDC			


**** Logging Started : Wed Sep 20 10:56:19 MEST 2006
*** njw (~njw@130.226.216.2) has joined channel #jemxadr
> Hi Niels Joergen? How are you? How's your awful cold? END
<njw> Hello Carol Anne, I'm fine, my cold is slowly receding END
> Agenda:

<njw> I don't think you so much coughing any more (?) END
*** silvia (~silvia@130.226.216.2) has joined channel #jemxadr
<silvia> Good morgen Carol Anne eg Niels Joergen !
*** [stephane (~paltani@isdcsf3.unige.ch) has joined channel #jemxadr
<[stephane> Hello Everybody!
> Hello dear SDASTers! Is every one well? END
*** peter (~pkretsch@crab.unige.ch) has joined channel #jemxadr
<peter> Hola! Made it from my previous meeting ... END
> Hi Peter! Nice to see you so prompt!END
<njw> Hello Stephane, Peter, and Silvia END
> Jerome is here today but he's not well, so I think we can start without
> him. First point on the agenda is SDAST meeting. NJW can you just summarize
> the dates and times for anyone who hasn't read your email yet? END
<njw> I've sent an email that our meeting #36 will be October 26 and 27 2006.
*** jerome (~jerome@130.226.216.3) has joined channel #jemxadr
<njw> Niels L has agreed to those days as well as Peter, Stéphane, Carol Anne,
<peter> Yep. Lars doesn't like the idea that I'm gone on what maybe the few days 
we can still see each other at work, but we will find an arrangement. In the 
worst case, I will have to fly back Thursday night already. END
<jerome> Hello everyone! Sorry for the delay, I was talking to Søren... END
<njw> Silvia and myself END
<silvia> I've already booked my flights. I'll arrive on Wed. 25th and return on 
Friday 27th at 17:20 p.m. END
<jerome> Btw I may have forgot to say that 26 and 27 Sept are fine for me too. 
END
<njw> Sounds good, Silvia END
<njw> Very good, Jérôme END
> Please let me know how many of you will be in town on the evening of Wednesday
> 25th - you're all invited to supper Chez Oxborrow. END
<jerome> Sounds good... END
<[stephane> I will, but I land at 10pm, so I better go directly to my hotel, but 
thanks anyway. END
> Sorry, I'll have to retract that invitation - it turns out that Wednesday
> is a terrible night for us. END
> So any questions about when and where for the SDAST meeting, or shall we
> move on? END
<njw> N Y
<silvia> Don't worry Carol Anne ! Just relaz and I try to go all together to 
FIASCO on Thursday ! END
<silvia> N Y
> Okay, next on the agenda:
> 2) Status for j_cor_gain, j_cor_position, j_dead_time_calc (CAO)

> Well, basically, all these three components have been redelivered, and as
> far as I know there's no integration problems - leastways I havent' heard
> anything. j_cor_gain 7.2 can handle input IC gain history tables and now
> produces a SCW data structure JMXi-GAIN-SCP that gives the detector gain at
> 1 minute intervals during the SCW along with the gain calculated from the
> individual calibration shources. The program can also handle revolutions where
> one or more anode segments are switched off to protect the telemetry 
allocation
> in the cas of very strong sources. This style of operating the instrument
> with some anode segments switched off will be tested during the next Crab
> calibration. j_cor_position no outputs event positions randomized within the
>  pixels instead of events being given the position of the centre of their
> pixel. j_cor_gain also produces event energies randomized within their PI
> bin as well as the PI value. Both these features I'm told will be needed for
> the forthcoming j_src_properties. I didn't have time to improve the hotspot
> flagging facilities of j_cor_position to cover the case of very strong
> hotspots that produce stray events along entire anodes, but that will be done
> by the time j_src_properties is delivered. j_dead_time_calc was tweaked a
> little to cut out multiple HK lines produced by having On Request HK mixed
> in with ordinary automatic CSSW HK. There will be  SPRs forthcoming about this
> practice of mixing OR and CSSW HK by PreProcessing, but for the time being
> j_dead_time_calc can handle it. And now I'm working on making, verifying and
> delivering the IC gain history tables to ISDC in time for the next release.
> Any questions? END
<peter> No
<njw> N
<[stephane> These components should have been integrated in NRT and CONS by now.
<[stephane> No news is good news.
<[stephane> Also, I'm not not sure what you mean by:
<[stephane> "There will be  SPRs forthcoming about this practice of mixing
<[stephane> OR and CSSW HK by PreProcessing"
<silvia> N
<[stephane> This is normal, and will continue END
> This may be normal practice for you, but we can at least show our concern
> by fighting it a little! END
> You'll probably be hearing from Soeren END
<[stephane> It's normal for INTEGRAL, and for Jem-X as well END
> It is a flawed procedure that isn't even necessary in most cases - 
PreProcessing
> doesn't even check if there's a valid value of DFEE_STATUS in the OR package
> before filling the row with 350+ NULL values - this at the very least should
> be changed. END
<[stephane> Tell those who designed the INTEGRAL spacecraft, but I won't expand 
here. END
> I think this really needs to be discussed with the PreProcessing guru. END
> Any more questions? END
<njw> N
<silvia> N
<jerome> N
<peter> N
> Okay: next on the agenda:
> 3) Status for j_ima_iros and j_src_properties (NJW)

> Fire away NJW END
<njw> j_ima_iros-1.6.3 has been delivered with some help from
<njw> Mark Gaber, accepted (numerical differences considered acceptable).
<njw> All SCREWs covered and many SPRs but there are still three
<njw> left. There are changes in the value of the IMATYPE in order
<njw> to make that keyword characterize the images completely, i.e.
<njw> no longer need for IMADESCR, but it is kept for the time being.
<njw> The image correction (called 'neighbor correction' because
<njw> it is based on an estimate of how many counts are lost to
<njw> neighbor pixels considered closed) is formally in place but
<njw> thorough testing is still lacking (Stephane and Jerome are
<njw> working on it). The idea is that the ISDC component
<njw> 'image_spec' will have some images where the flux (and spectra)
<njw> are of reasonable quality.
<njw> Any questions on j_ima_iros ? END
<peter> N
<silvia> No now
<[stephane> it's "mosaic_spec" END
<njw> Sorry, I'll remember that, but it should also be able to
> No questions about j_ima_iros. What about j_src_properties? END
<njw> make spectra from images that are not mosaics, right ? END
<[stephane> Yes END
<njw> j_src_properties works formally, but the spectra it produces are bad.
<njw> I'm suspection that the PIF calculation is not accurate enough
<njw> but we are still fighting to find the cause. END
<njw> suspection = suspecting END
> So any ETA for j_src_properties? END
<njw> What is an ETA ? 
<silvia> Questions no  ! ETA no idea END
> Estimated Time of Arrival (airport jargon) END
<silvia> Good, I was just thinking about the Spanish Terrorrist Organitation ?? 
END
<njw> Since we are stuck we cannot say how long time it will take
<njw> but we hope for progress in a month or so END
<peter> Is there anything someone outside DNSC could do to help you? END
> Yes - they could get the Space Shuttle to cath the satellite and rip the
<njw> I can't see it for the moment but we will keep it in mind END
> JEM-X Collimators off! END
> How does that tally with ISDC's release dates/plans, Stephane? END
<[stephane> No impact, since j_src_properties won't be part of OSA 6 END
> Okay any other questions about j_src_properties? END
<silvia> N
<peter> n
> Okay next on the agenda:
> 4) Status for mosaicking (JC)

> Your turn Jerome END
<jerome> j_ima_mosaic 6.0.1 and 6.0.2 have been delivered last week. The latter 
in a
<jerome> hurry to be compatible with the last changes in j_ima_iros as described 
in _SCREW_ 1918.
<jerome> There is not much to tell, unless there any questions? END
> None from me. END
<njw> N
<silvia> N
<jerome> I have one to Silvia: have you started to look at the new parameters? 
Is it OK for you?
<jerome> END
<silvia> I think so. I've been able to test yet your new softare becasue the 
pipeline is stuck in j_ima_iros. More in a few seconds. END
<silvia> sorry I've NOT been agle ... END 
> Don't worry, we know what you mean Silvia. END
<jerome> Y
> So that brings us neatly to the next agenda point, if there are no more
> questions:
> 5) Status for scripts (SMN)

<jerome> I guees you'll let me know about your progress Silvia with 
j_ima_mosaic. END
<silvia> j_correction, that it is an independent scripts sicne March 2004, will 
be accepted along today.
<silvia> Of course JC.
> Any you've got the mechanism for finding the right IC gainhistory table
> working okay? END
<silvia> The new version of j_correction handles the changes performed by Carol 
Anne in j_cor_gain and j_cor_position. 
<silvia> And of course it finds the appropriate gain history for the given data. 
> Great! So the new functionality will be used. END
<silvia> It is important you keep on mind that this script is independent 
j_scripts, to include it in the SPRs and SCREWs related instead of j_scripts. 
> Thanks for pointing that out - I wasn't aware of the distinction. Also have
> you made rowSelect="&&STATUS<256" a default of all the scripts - this is
<silvia> Regarding j_scripts, they are also neraly ready for delivery. I've 
implemented several SPRs and SCREWs, and pipeline it is running nicely until IMA 
level, where it crashes due to a Segmentation Fault in j_ima_iros.NJW, have yu 
make progress in this point ? 
> essential for automatic hotspot removal. END
<njw> No, I've been tied up with something else, but I'm looking forward to your 
test
<njw> with rev_2 data END
<[stephane> Carol Anne, is there an SPR for this rowselect=...?
<silvia> Unfortunately, I could not run the test in rev_2 due to other problems. 
I'll try latter and let you know about it. I'll 
<silvia> appreciate if you take a close look to the parameter settings, maybe, 
there we have the reason of the problem.
> Yes, I'm pretty sure I wrote one because I was tired of ISDC folk complaining
> about horrible noise in the images when there was a hotspot. END
<njw> Silvia, I'll do that END
<silvia> To CAO. Yes, thre is a SPR on the rowselect. And I was alaready talking 
with NJW about it. 
<[stephane> SPR 4522; it's not ISDC; it is all users END
<peter> And in the faraway past we agreed that ging rowSelect with nothing to 
the executables should make them do something "sensible" ... END
<silvia> NJW, have you worked in SPR 4522 ? END
<peter> Regarding the j_ima_iros crash wouldn't someone with a debugger (at 
ISDC?) be able to repeat this given the parameters which we can get from the 
logfile?
<peter> END
> That's good Silvia, just so long as SPR 4522 hasn't got lost somewhere. END
<silvia> I've already sent a debugger log to NJW, and there is a divison by 
zero, reasonable reason to get a the segmentation fault. END
<silvia> But, NJW is a bit surprised about this zero since it si the first time 
it is happening ! END
<njw> 1) I have not worked in  SPR 4522 because j_ima_iros or j_ima_shadowgram 
is not
> Divisions by zero are rather insidiuous because out system here can handle
> them and gives the answer `INF', and even knows that multiplying `INF' by
<njw> mentioned as affected items. Perhaps they should. I've understood that the 
script
> 0 gives 1.00 again. Then you run it at ISDC and the whole thing goes belly
<njw> could take care of that
> up because their system can't handle inf. END
<njw> 2) I'll try to rerun Silvia's problem giving SWID and see if I can 
reproduce the error END
<peter> Many systems can't, and any well written numerical code (not mine often) 
checks before dividing. END
<[stephane> Carol Anne, you should throw away your computer immediately if it 
behaves like this END
> Happily, it's not mine. It's DNSC's. END
<silvia> To CAO: thanks for the hint. Now I understand why NJW has not have the 
same problem that me. END
<njw> To Peter: There are A LOT of checking for zero divisors, but unfortunately 
not in
<njw> this particular case END
<[stephane> Note: I'm not sure what is the relationship between division by 0 
and SPR 4522 (hotspot removal) END
<silvia> Regarding SPR-4522, if we all agree the scripts can handle setting the 
parameter to "STATUS>256" for BIN_I and IMA levels. END
> SPR4522 has nothing to do with j_ima_iros. It's a scripting problem. If
<jerome> Yes. Are you talking about two totally different things in the same 
time? END
<silvia> To Stephane: not relationship at all. Just that we are discussing two 
different things at the same time. END
> the rowSelect is set to "&&STATUS<256" the hotspot events and other bad
> events will be filtered out by the components after COR level. We know this
> works, but it must be made the default. END
<silvia> To CAO: It's not true that it's just a scripting problem. Take into 
account that in the spectral and lc levels the default setting (=0) is 
intrepreted by the software as the appropriate value for the selection of the 
events.  
> That's because Stefan has very strict event selection (status==0) built
> into all his components. You have to be a very good event to get used by
> him. But these components will soon be replaced by j_src_properties and so
> we have to ensure that medium-severe filtering is forced on them if they don't
> do it themselves. END
> Any more questions about the scripts anyone? END
<njw> N
<silvia> OK. Then, are we all agrre in that the event selection has to be 
defined by the scripts ? If yes,
<silvia> I'll change the parameter setting of BIN_I_rowSelect to "STATUS<256". 
END
<njw> Please remember the && in front END
<silvia> Thanks NJW ! END
<silvia> Peter, what about your BIN_I tools event selection ?
<njw> It is an additional selection string since it is also used for the radius 
selection END
<peter> To Silvia: I don't remember right now, but I think they work similar to 
Stefan's code.
<peter> To NJW: It is not safe to leave it to scripts/users to provide you with 
completely correctly formulated rowSelection strings. The '&&' should rather be 
insde your code.
<peter> _But_ for OSA 6 we should not add a complication now. 
<peter> Still in the long run, nver forget to make the software rather _not_ 
rely on user's getting everything right.
<peter> END
<silvia> For OSA 6 we will need to explain to he users why the default setting 
is this selection string. END
<njw> Will you write and SPR on this ? END
<silvia> NJW, are you refering to the future change in the string ? END
<[stephane> I don't think a new SPR is needed END
> How about I update the existing SPR right now to include j_ima_iros that
<njw> Yes, I meant the string change. Alright I'll just do it at some point but 
the
> should be able to a `&&' to the rowSelect string if one isn't provided by
> the user or scripts? END
<njw> word needs to be spread around for the users that already have scripts 
with the
* njw && formula END
> Okay, any more script related questions? END
<njw> It is actually j_ima_shadowgram that has this parameter END
<njw> N
<silvia> N
<jerome> N
<[stephane> I'd like to remind that we are already extremely late, and that
<[stephane> any further delay will have to be compensated by a decrease of
<[stephane> the testing, which is very bad for all of us.
<[stephane> END
<jerome> Is OSA (6 or whatever has been decided) not soon to be frozen? (just 
curious) END
<[stephane> It is frozen, in theory... END
<jerome> I see... END
<peter> My idea was that for OSA 6 we do as said, i.e. pass the '&& ...' 
parameter by the scripts but in the future the executable gets improved to deal 
with all possible inputs sensibly. END
<njw> I didn't think of redelivering j_ima_shadowgram for OSA6 but for OSA6.1 or 
.. END
<[stephane> I'm sorry; I can't stay on-line much longer. We still need to 
discuss
<[stephane> IC files. Some IC files are missing for OSA6:
<[stephane> - OCL --> Carol Anne
<[stephane> - ARFs --> Niels Jørgen
<[stephane> - BTIs --> Me.
<[stephane> To prepare the BTIs, I urgently need Søren's list of ScWs with
<[stephane> non-standard instrument settings. Could one of the you put a little
<[stephane> bit of pressure on him?
<[stephane> Carol Anne and Niels Jørgen, how far are you in the process? END
<silvia> Stephan has been update of the IC_TREE  ( 
/isdc/ic_tree/osa_ic-20060703) this early morning that 
> Okay the SPR's updated. `COULD WE PUT SOME PRESSURE ON HIM'?? I've tried.
> I'll try again. END
<[stephane> Thanks! END
<silvia> already include the OCL and the ARFs, as well as the latest IMOD. END
> I should be ready with my IC files by the end of next week at the latest. END
<[stephane> These are test OCL; not the real ones END
<[stephane> Which week is 'next week' ? END
> Yes, real OCLs are being made and verified now. I'd probably have them done
<silvia> Uff ... then,  do I have to wait for the real ones to delivery the 
scripts ? END
> this week except there's a personnel conference on Friday that we all have
> to attend. END
<njw> I hope to have the new ARFs by the end of the week END
> I don't think silvia needs to wait for the real OCLs to deliver her scripts:
> if they work with the test files they'll work with the real ones. END
<[stephane> Yes, IC files in the test IC tree are OK for the dleivery END
<silvia> Good  END
<[stephane> I need to leave now. I'm looking forward to your deliveries! I'll 
read the end of the chat later BYE BYE!
> Next week begins on 25th September - and I'm at the Danish Natural Sciences
> Festival all monday afternoon. END
*** Signoff: [stephane (Disconnecting)
> Okay - any more on scripts? END
<njw> N
<jerome> N
<silvia> Once we will solve the iros problem, I'll deliver them including the 
SPR-04522 that has 
<silvia> already be implemented. END
<jerome> I think the rest of the aganda most concerns Stéphane... END
> Yes, can anyone add anything on either of these points?
> 6) Status for integration (SP)
7) ISDC data analysis workshop

<silvia> N
<njw> N
<peter> n
<jerome> N
> Me neither, so let's hop ahead to AI list. Last time you all moaned and
> groaned, but after I went through it I found there was actually lots of
> obsolete stuff. Please have a look at the list RIGHT NOW. Here's the
> URL: 
> http://spacecenter.dk/~oxborrow/sdast/AIList.html
<peter> Nothing for me :-)
<njw> AI27.7 cancel or put 19 July 2007, I'll have no time to do it
> Actually, there's quite a lot of things that should be done by Soeren and
<njw> AI30_6 Overtaken by events - new corrections in j_ima_iros so obsolete
> Carl - so I'll try to APPLY A BIT OF PRESSURE. What an idea! Has anyone
> ever succeeded in applying pressure to Soeren - all hints gratefully received!
> END
<njw> AI35_4 Obsolete now Simona has solved the question
<peter> Regarding Soeren: have you tried calling him twice daily? ;-)
<njw> AI35_5 is onging with j_src_properties
<silvia> Or even 3 ? :-)
<silvia> To NJW: I've run the scripts test in rev_2 and I get the same error 
with the same debugger message. Let's take a close look to the parameter 
settings. However, after CAO comments on zero divison at DNSC, I'm afraid the 
problem is more serious. END
> Any more updates to the AI list? END
> Also be aware that division by zero or arithmetic exception without division
> (both of which can cause a core dump) can be caused just by trying to READ
<silvia> N
> (not process) data in an uninitialised array. END
> So any other business? END
<silvia> Just one tiny  thing ! How do you say in Danish "You're wellcome" 
(after thanks) ? I cannot remember. END
> `Det var saa lidt' seems to cover the situation. END
<silvia> Mange tak ! END
> or 'Velbekomme' if you're giving food. END
<peter> AOB from Erik: Jerome (and Soeren) should urgently read their emails and 
respond to him regardin Galactic Bulge and an ATel
<peter> END
> Shall I APPLY A BIT OF PRESSURE (the very idea cracks me up!)? END
<peter> We should apparently get you a nice black-leather outfit and some 
suitable paraphernalia :-)
<silvia> Go Caro Anne, go ... END
> Okay, if that's it for AOB perhaps we'd all better see to our lunches.
<peter> AOOB? (Any other other business?) END
<peter> Good idea, just about time at ESAC.
<silvia> No ... 
<silvia> Too early for me !! END
> Good talking with you all - the next SDAST chat will be on Wednesday 11th
<silvia> Hej, hej !!! END
<silvia> Perfect CAO ! END
<njw> NAOBBL : No any other business than lunch
<peter> You know aftreliving in Spain it really cracked us up to read "Lunch 
10:30-13:00" at Swedish
> October. Be there or be rectangular! END
<peter> restaurant doors. END
<peter> OK, see you then. END
<silvia> Bye .. Adios ... END
> Take care till we chat next! END