SDAST Chat Meeting # 111: Wednesday 8th March 2006 Participants: Peter Kretschmar, Carol Anne Oxborrow, Silvia Martinez-Nunez, Niels-Joergen Westergaard, Jerome Chenevez Subjects discussed: ADD 10.0 - Will be released to ISDC today when PK's affiliation has been changed to ESAC News from ISDC - Obsolete s/w in NRT pipeline upgraded to OSA5.1 - Sometimes no images of SCO X-1 because no GTIs - Software trigger over limits automatically causes BTI - Limits table needs updating - NJW - Users can provide their own GTIs to image SCO X-1 (IF they know how to do this). Users must remember to do and og_clean to re-analyse their data from COR level if no images are available with the automatic processing - User manual should contain something on special processing for strong sources (PK and SMN) - Bad JEM-X images in the barn are just ignored. Suggest in future operators send email about all unusual things that occur with JEM-X - Lack of participation in ISDC representative rotation - Needs to be discussed at SDAST meeting and list of representative duties should be drawn up - Andrii sees about 10% drop in intensity at beginning of EO data for the very lowest energy bin only, that has to do with gain change at start of each revolution. -This is probably due to the electronic efficiency problem, but could also be due to gain changes all over the plate that can't be corrected for. - Ideally planners should wait till instruments have stabilized - NL/NJW looking into whether JEM-X can be switched on sooner after radiation belt exit. - A lady, Aga, from MPA claims rev 33 has gain calibration problems - CAO looking into this Plans for OSA 6.0: improve hotspot extraction; adding together weak calibration spectra; using supplied gain history tables; new source extraction program; source-finding for mosaics; new spectral tools; improved binning tools for burst detection; mosaiced images with real sigmas; removing distortion in mosaics at large angles; making mosaics with overlapping energy bands. SDAST Meeting: CAO not available in late afernoon/evening; Jerome not available on Friday and Peter not coming at all. Next chat: Wednesday 29th March. Stephane should be joing us for that one Action Items AI060308_1 NJW, PK Update limit table used to determine GTIs, especially OPEN hardware and software trigger limits, so that very strong sources can be viewed. AI060308_2 PK, SMN Write a section in the ISSW users manual on setting OPEN GTIs for very strong sources AI060308_3 CAO Send email to ISDA operators suggesting they report all OPEN unusual activity to us. AI060308_4 PK, SMN Ensure that problem of instrument stabilization is OPEN mentioned in the user manual AI060308_5 ALL Send Jerome dates when they are available to be instrument OPEN representative at ISDC AI060308_6 ALL Create a list of instrument representative responsibilities OPEN and activities **** Logging Started : Wed Mar 08 11:00:39 MET 2006 *** jerome (~jerome@130.226.216.3) has joined channel #jemxadr > Hi Jerome! If no one else turns up, we'll have the meeting in your office *** silvia (~silvia@130.226.216.2) has joined channel #jemxadr > and you can provide some good biscuits! END Good morgen Carol Anne and Jerome ! > Hi Silvia! How are you? It's ages since I heard from you. END *** njw (~njw@130.226.216.2) has joined channel #jemxadr Buenas dias Silvia! Fine ! I'm Madrid with Peter. how are you and your family ? END > Thanks for all the survey results Jerome: I have them hidden on the gain Hello everyone! > archive page, to be revealed when the data's made public. END *** peter (~pkretsch@isdcsf5.unige.ch) has joined channel #jemxadr > In Madrid, Silvia? Is that warm? We've just had more snow here. END Hi Peter! It's nice ! The spring is already starting in Spain ! END > Hi Peter! How's life in Madrid treating you? END > Stephane can't be with us this week, but he promises to join us for the first > time next week. I assume Stefan won't be participating either, so Shall we > get started? END Y Y Y > First on the agenda is the release of the ADD. Have you anything to tell us > Jerome? END Well, as I was ready to send it to ISDC, Niels came and showed me that I forgot to change Peter's affiliation: it stands ISDC several places. Silvia is partly taking Peter's tool, but I need to ask Peter: are you still responsible for j_bin_***_spectra and j_bin_***_lc? If yes, should ISDC be removed and replaced by ESAC? END Yes, I am but evidently this means that you should not expect huge progress ... END OK then you will appear as ESAC. Otherwise there is nothing to add if I have no more feedback from you, I am going to send the document officially to ISabelle at ISDC this afternoon. OK? END > Great! And so we move forward, both documentationally and agenda-lly. > Next is news from ISDC. As you all know I was there for a week a couple of > weeks ago. I think the big problem with using obsolete software in NRT > pipeline is more or less cleared up now. Jake and Bruce claim that they've > upgraded the pipeline to OSA5.1 modules, and this will become effective in > mid March. > Another problem was that there were no images during the SCO X-1 observation. > j_ima_shadowgram shut down because there were no GTIs. I've since discovered > that the software trigger rate was above its limit for these observations. > Is there anyone out there who can confirm that this condition will cause a > complete lack of GTIs??? END Well, we have a set of limits and if they are exceeded the time is considered "bad", it's as simple as that. > Okay, then we really must update the limits table in IMOD. Can you do this You may remember that I offered twice to update those limits but to no avail. > NJW? Does the same apply in OSA? Can people choose to look at the images even > if a parameter is out of bounds? We don't want all our strong sources thrown > away like this. END Naturally the same applies. I'm just looking a the script parameters - there is a parameter "GTI_limitTable" which would need to be set to a table with higher limits if you want to analyze Sco X-1. _Or_ , alternatively you could pass a specific setting of "GTI_gtiJemxNames" to override the default GTIs that are used in the GTI step during OSA. Keep in mind that you _must_ og_clean a rev_2 Obs. Group back to level PRP (ignoring all the warnings about non-deletable files) if you want to have the GTI's redone. IS not it enough to run from COR? END On the risk of sounding cranky I can only emphasize that more of you should really study these scripts and try to understand all implications. END Jeroe, you are right - I tend to redo COR as well ... END So, no need to og_clean before? END > I think we need an improved table so that the operators and SCODYs in the > barn, who are certainly never going to look at our scripts and understand Maybe Silvia and I could take an action to write a paragraph on how to cope with bad GTI's for the analysis manual, the ISDC and Georg Weidenspointner (PI of the "Sco X-1" obs). END > them, can automatically get pictures. Otherwise, they tend just to ignore Sure, but there always may be something beyond our latest table and a new IC table can also take weeks to be installed. END * oxborrow JEM-X and assume that its just actiing up again. END > ....or months to install...END So if someone tells me what limits I should put in, I could redeliver a table within a day or so. END ... which may be used first in 6 month... END OK, we should pass this request on to Søren and Niels END Back to JErome: You _MUST_ og_clean, at least up to COR (if you want to keep that) or the GTI step will fall ovber. END Ok. Thanks. END > I think it's worth making an effort on the table, and hope it gets used. > I think most IDSC peole understand my displeasure over the NRT cock-up. > NJW, can you determine the new limits, appropriate for SCO-X1 and send them > to Peter? END I presume you did a hint more than tell them you were "not amused" :-) END Yes, I'll do that END > Yes, I told several people that it was a scandal what they'd released > publicly in the rev_2 packages prior to 402, within hearing of several more > people. END > So, are we agreed how to solve the GTIs problem? END Y > Another problem uncovered is the lack of participation in the the ISDC > rotation system. I find so many little problems and complaints when I go > there, that it's clear that we need someone every month there. Everyone seems > to agree with this basic statement, yet only Jerome and I seem to actually > visit the place - something's got to give! END Yes. We have no representant for this month at ISDC. I plan to take my turn at the end of April. Or we still folowing the plan I sent at the end of last year or should I make a new one? END Please give me your possible date for disponibility... END date -> dates! I'm sorry I cannot go this month. Since Jerome is going at the end of April, may be we can plan the rest of stays during our SDAST meeting. What do you think ? END It is mostly an internal DNSC problem since Silvia is the only one from > Yes, our guys are very sluggish at offering to be mauled around by SABENA outside this house (Peter has probably too many other obligations), so we'll > for the good of the project. END continue to plan this and it is a good idea to confirm at the SDAST meeting END Right. But I agree we should mention that issue at the next SDAST meeting. END Yes, the truth is that I would not be able to serve in this function. END > Continuing with ISDC problems. The barn ops software had only recently been > updated to OSA 5.1, since when they'd had awful images because they were > looking at the non-flatfielded images. Unfortunately there's no one there > who can point out problems like this to them. I hope NJW's email explaining > the different sort of images has straightened out this problem. END > Andrri has been working on the EO data, all of which was taken very close But who actually reads these emails at ISDC? END > to insturment switch on. He appears to see a 10% drop in intensity due to > some effect related to gain changes at the beginning of the revolution. > This probably has to do with the fact that the electronic efficiency is not > taken into account yet. This will be done in OSA6.0. I sent you all an email > about this on Monday, and Peter was seriously concerned. I'm not sure that > serious concern is called for, since related energy changes seem to be > about 2%. Basically, however, if you want really good data it's best to > wait till the instruments have stabilized, just incase the effect is some > gain-change thing we can't correct for. I don't think it's unreasonable to > ask planners to wait till the instruments have stabilized when this takes all > of a couple of hours. NJW is looking into us having switch on a little sooner > after radiation belt exit, perhaps changing the order of the switch ons with > other instruments. Comments? END I am just conrened that WE do know about these issues, but else does...? END I think this issue should be mentioned at least in the user manual of the ISDC. END I fully agree. END Right, we need to update that one END A comment w.r.t. barn - one should insist that if things look strange a mail is sent to the JEM-X team. Even with increased presence, which I fully support (quickly changing to my GS Coordination Hat :-) ) > Yes, that's a good suggestion because they tend just to note it in the log Yes. It seems like they just wait for seeing us when we come there. END strange instrument results can happen when no representative is there. END > book and then accept that the images are appalling. END I'll instruct Nami and Simon and ? for this action END > Okay, I'll get on to Nami and Simon and Philippe about anything strange at > all being reported to us. This will mean a lot of odd emails however. END All right, fine if you do it END > Andrii was also having problems getting JEM-X2 data to process, but it turned > out he'd saved a bad parameter file somewhere with horrible typos in it. > Finally, there's a ladying at MPA who's looking at revolution 33 and says > there's a problem with the gain correction. I'm still looking into this - > I can't even remember revolution 33. So that's all the news from ISDC. > Any comments? END No further, except thanks for noticing and acting on these problems! END Just one ! During the SDAST meeting will be good to define more or less a set of responsibilities during our visit to the ISDC END Right, it would be good to do a brush up on that one END > That's a very good idea Silvia. There was a very old list on theold forum, Yes. We actually did that a long time ago and it would be good to update it. END Good ! END > but we definitely need an upgrade. END > Okay shall we move on to the next agenda point? END Y Y Y > Next on the agenda: plans for OSA 6.0. I'll start the ball rolling, then > each of you should let us all know your planned improvements. > j_calib_gain_fitting is currently being improved to add to gether calibration > spectra in cases where the spectra are too weak to be fitted singly. This is > necessitated by the decreasing source strengths of the calibration sources. > j_cor_position will have improved hotspot removal > j_cor_gain has to check whether there are pre-smoothed offline-corrected > gain history tables available in an ic file, and use these instead of > automatically created gain histories. > I think that's all. What about you NJW? END You all know about j_src_properties which is the process I'm digging in Stefan's code to re-use as much as possible and to shape into a format that will work with the PIF making library from j_ima_iros. There is still a lot to do. As a side line I'm working on a source finding tool particularly for mosaic images but it can also be used for standard j_ima_iros images. The prototype is almost finished, but there remains some work if it should be included in the jemx_science_analysis script. More discussin at the SDAST meeting. But those two are foreseen for OSA6 END > Thanks, what about you Silvia? What will be new with scripts? END I'll have mainly to modify the scripts to handle all your changes. Then, please keep me inform about them :-) ! And some SPRs probably but nothing serious. END The main change will be to include the new spectral tools ! END > Okay, Peter, will you be doing anything new or improved for OSA 6.0, or is > this sort of work no longer your responsibility? END I should improve the j_bin_*** tools as we now see people using them a bit more for burst detection. There is a problem for the first/last bins in a ScW that I never resolved in the past. Also Andrii used them, as far as I know. So some corrections there but nothing more. END Yes that would be great (a user comment :) END > What about you Jerome, M.User, do you have some improvements for OSA6.0? END If you have some good examples of things going wrong that might help me detect the source of these issues? END I may have... let you know. END That was for Peter. About j_ima_mosaic: The mosaic significance map will be improved to give real sigma numbers and to help the source finding tool Also the distorsion at large angles from the center of the mosaic image due to too simple geometric transformation will be solved, the current version cannot make mosaics of overlapping E-bands in one time, I shall also solve that one. At last the list of input ScWs will be available in the HEADER of the mosaic FITS files. I guess that is all for the moment. END > That sounds like we're going to be very busy........Now on to the AI. > Has anyone done some AIs? I don't think there are any outstanding right now N > except for Soeren. Most are due in the summer, presumably as part of the n > OSA development. END > Okay, last on the list AOB. Under this heading I'd like to mention that > during the week when we have the SDAST meeting my husband will be away at > the annual meeting of the EGU, his first trip in over a year, so my participation > at the meeting will be a little limited. Anyone else with AOB? END When we are "speaking" about No that: as I said to NJW, I will not be able to join the 2nd day (the kindergarden is closed that friday!)... END Peter is also not coming and if Jerome is only half and Carol Anne limited, why do we no change the dates ? END Aha... I mean: that is to start another long exchange of emails... END The only chance is then to have it at a later time, which is also very difficult, but I think that we should have the meeting well before > I'll be there for 90% of the meeting, but not for supper or other evening > things. END the OSA6 deliveries and it is really HARD to find a time that suits everyone so I'll need very strong arguments to restart the data finding process. Stéphane would also like that we don't delay the meeting too much. END Thatøs right: there will always be someone having problems to come... END Ok you are right NJW is a bit crazy to look for a new date ! It was just that it seems we will be just few of us ! END > Just like the constortium meeting! END > Any other AOB? END Lunch! No I'm counting on some ping-pong with Peter before and after the meeting now that we will miss him END > I hope that's no AOB and not no to lunch, Silvia. END NOB Well, in ESAC lunch is CET ! What means in some minutes ! Bye everyone and see you soon ! END Look, Peter is already gone to lunch! No, I'm still here :-) > We chat again in 3 weeks, with Stephane present I hope. Have a good time > till then! END Just joking :) Yeah, yeah - always these prejudices about ESA staff :-) END Bye and have a nice day with lunch END *** njw has left channel #jemxadr : (njw) *** silvia has left channel #jemxadr : (silvia)