Weekly SDAST Chat # 70: Thursday 22 January 2003 Participants: Peter Kretschmar, Sami Maisala, Jerome Chenevez, Stefan Larsson, Silvia Martinez-Nunez, Carol Anne Oxborrow, Niels Jørgen Westergaard Subjects discussed: Status of development and delivery of all pipeline and scripts software New ways to check for dead anodes New ECAL-MOD tables: ECAL alerts Making background models `Flower' background in images Strange time ranges in binning and source extraction results BKG level in pipeline/scripts When to update ADD jemx.h: update and delivery Science Results: EXO2030 appearances and disappearances Cyg X-1 spectra Action Items: AI030124.1: CAO Feb 2003 Write SCREW and submit templates to extent OPEN j_prp_verify so it can produce dead-anode data by binning shadowgrams by rawX values **** Logging Started : Thu Jan 23 14:58:02 MET 2003 Hej Carol Anne ! > Hello everyone, how are we today? END Perfect, it is spring day in th "V" city ! END > It's NOT spring here! END To Siliva: We are still discussing how much our presence at ISDC is required Perhaps we will continue after March but with a smaller coverage. END *** larsson (~larsson@tenma.dsri.dk) has joined channel #jemxadr Thanks for the info Niels Joergen ! END > Hello Stefan and Sami - it's good to have you both here, now we're just waiting hhhiiiiii there! END Hi Stefan and Sami ! END > for Jerome - isn't he in your office peter? END Carol Anne, Peter is not here, yet ! END Hi all! END Niels Joergen: have you read my e-amil regardind EXO 2030 ? END > I wonder where Peter and Jerome have got to - not like them to be late. END Yes I have, but unfortunately I've been very tied up with the imaging thing. Hope to get to science very soon END Niels Joergen: do not worry ! It is just that I am very exciting with the science ! END *** peter (~pkretsch@isdcul9.unige.ch) has joined channel #jemxadr Hello, sorry for the delay - seems I'm running from meeting to meeting today ... END *** jerome (~jerome@uhuru.dsri.dk) has joined channel #jemxadr Hello, Hej, bonjour... > Hi guys! It's good to `see' you all again. Let's get on with the meeting! END > Any more agenda points people want to add at this time? END You think we don;t have enough ;-) ? no No . END N N > Okay, I'll kick off with a quick summary of where I've got to and what I'm > doing. First data preparation: > j_prp_status_table was redelivered last week and can handle most of what's > thrown at it, but Tess sent me an email to say that she's had one set of > SVT-F data where it missed a grey value somewhare - it sounds like a very > minor problem that shouldn't be time consuming, and isn't high priority. > j_prp_verify has a couple of problems - too many `bad OBT' alerts when there are > data gaps and too many dead-anode alerts. We're thinking of scrapping the > dead-anode checking function in favour of an offline tool looking at all the > slew data for a revolution instead, with some human checking thereafter. > This will be a little time yet. > j_calib_gain_fitting was redelivered 2 weeks agon > but apparently isn't been used in operations, so the same old `can't fit these > spectra' rubbish is currently being churned out. This is for ISDC to solve. Unfortunately I have to confirm that j_calib_gain_fitting 6.2 seems still in use. A glitch here (the software was supposed to be updated this perigee passage). It will be solved ASAP - which might mean next perigee. END > Other than non-useage, I think j_calib_gain_fitting is all correct now and > currently needs no work. > j_calib_adc is working fine in its current version. The alerts from this > morning are only due to using an obsolete -DEAD-MOD table, so I'm working > on a new one. In fact the table I sent NJW before Christmas would not have Tess just installed j_calib_gain_fitting-7.0, so most of Rev 34 should be fine. END > produced these spurious alerts, however I'm making a table now with much > wider tolerances. I'm also going to look into the time-variation in our > ECAL results: I think we've got a steady drift of the offsets upwards, so I > think the long-term behaviour should be looked at a little. Other than that > no current work to be done on J_calib_adc. However, the instrument team are > planning a change in the ECAL procedure which I BELIEVE can be handled smoothly > by the software, but only time and experience will tell. > For data correction: j-cor_position's just fine. I'm currently working on > j_cor_gain to implement the `decay' model of the gain time variation, and > also the 16bit STATUS flag (see below). This will be several more days yet: > I want it well tested before delivering which will add a day or so. > Finally j_dead_time_calc I hope to start major re-working after delivering > j_cor_gain: it's not clear how we're going to calculate deadtime now that > Soren and Niels have found that we're getting quite a few double triggers > in the hardware trigger counter. They're thinking about how to deal with > this while I finish j_cor_gain. > Basically there's a fair amount of work to do, but I feel I have control > over it. Any questions? END N N N Yes, 1, but I am waiting for my turn to speak about dead anodes. END > Okay, next up to bat: Sami, on background handling automatic and otherwise. END I'm still working on bkg models. The main problem is to separate instrument and diffuse bkg There is quite lot of variations of spectral shapes in different parts of detector and I'm trying to find solution to select the 'best' spectras to fill the spectrum map. I'm going next week to ISDC and I hope after this very productive week we will have the first bkg models and software to test. END > Might I suggest that you make sure all the energy/spectral data you use for Niels Lund is also trying to do that separation - in a different framework, though - > this analysis come from J_calib_gain_fitting 7.0, else you're just looking at > rubbish much of the time, which might explain large differences in spectral so at some stage in the near future I think it will be a good idea to make a comparison END > shape from one anode segment to another. END CAO I'm using the version 7.0 and blank field observations from rev0024. END > That's good to hear Sami - I'd hate to find all your work wasted by that > appalling version 6.x! END > Is that all from you Sami? Any questions? END The idea of comparision of the results could be really nice! I think that's all now. END > Yes, I agree two independant appraisals would be very valuable. END > Okay next at the plate: NJW and imaging END I have now a running version of j_ima_refined (I have not checked the results yet, that's what I'm doing at the moment besides talking to you). j_ima_refined is the Niels Lund version of the imaging (IBAS reworked) with IROS i ncluded. I hope to be able to deliver a version early next week. END Sorry ! I have not helped you with that ! As I said, in our last SDAST Meeting ! Could I still help in someway ? END Thanks Silvia, but I fear that it is not so easy, but I'll keep the offer in mind END > Any more on imaging NJW? Any questions ? END You are welcome ! END Not for the moment END Niels joergen,do you think that the flower structure is indeed the mask geometry ? END It seems IMA is not removing all the pattern ! END The three-fold symmetry is certainly from the mask. Perhaps these flowers will be absent in the new imaging END That is great ! END OK, but what is the run time of the new imaging? Comparable or much longer? > I'll miss the flowers......END Not me Carol Anne, not me ! END We need an imaging tool here that works for QLA & SA and does not hide weak sources in very systematic structure. Do I gather correctly that currently no one really knows where it comes from? END I have invested my efforts in developing the new executable in stead of digging into the old one. That might become necessary if there are problems with execution times. Else yes, no one knows for sure END So do you have numbers for the execution times? Not yet END Are you saying we should foresee the possibility of ditching j_ima_basic_recon and running j_ima_refined_recon instead? Do I assume correctly that their outputs are compatible so source finding still works? END To #1: yes Sorry for being so pushy but acording to plan we would want to give software to users in <2 months! END To #2: There is still some work needed to produce a proper JMXi_SRCL_RES END > But off course we know where the flowers come from: if you rotate the results Another question: when Carl made his iamges they looked much smoother. what did he do differently? END > (images) from JEM-X1 you get an identical flower pattern to JEM-X2 - at worst > we can remove all the systematic structure that's the same in the two > units once rotated by 180 degrees. END Fine, but: (a) now we usually have just one :-) > I'd forgotten that! END (b) one can in theory take an (appropriately scaled) "empty field" image and subtract that, but then one has to be very careful to subtract it correctly. In the sky frame different images have the structure rotated. One could imagine subtracting such a noise map from the RSTI images before storing them and going to the sky images. Have you considered that Niels Jørgen? END I've considered subtracting an empty field shadowgram before doing the image reconstruction - because of the linearity of the process this should be equivalent to subtracting from the RSTI. END Niels Joergen: Peter and I we did that last Christmas time (to substract shadowgram from an empty field), and we are not sure is good way ! I will send you some results ! END I'm not convinced as according to one test of mine even an almost perfectly smooth shadowgram returns a flower after deconvolution. END Well, thinking a little bit more, that is perhaps not quite true, since the We do not see real differences ! END systematic pattern is imposed by the reconstruction, so it is probably a better idea to do the RSTI subtraction after all END Though it would be a clutch, admittedly ... END > When do you expect to have you program finished Niels Joergen? END Some time next week I will have a version that can run and show the promises of this method. It will take a little longer to get the image in celestial coordinates and a source list in the correct format * njw "a little" is probably one week more END > Any more on imaging? Any questions? END N No N N n So is it my turn now? END Taking silence as affirmation ... The spectrum and lightcurve _binning_ (as I do not really extract anything) tools seem to work fine so far, except that this morning when Jérôme and I analyzed a possible GRB event cutting times with a user GTI the output contained a larger timerange than that covered by the data. I'll look into this. Stefan - by the way for j_src_lc it was even worse, it seems to set up the output at full length and then put the small selected time output at the beginning. And SPR will be written, once I get around to check it out once more.END Another problem is that at one point I need to validate the application of correction factors. To be done when I get a breath. END Next orator please ... END OK Peter, I'll look at it when I have the details. END Unless there are questions to Peter...? END N > N > Looks like it's your turn Stefan. END Status is: j_src_lc was delivered one and a half week ago. j_src_spectra should also be delivered to clean off the same SPR/SCREWs as _lc. The most important part which is missing right now is the capability of processing REST mode data. Except for bug fixes that is my next task. Last few days I have learnt how to use fitsio so I can read the lightcurve files directly into my timing analysis program. Good for science but also good for all the testing. You know that you could link DAL independently from the rest of our libraries? Questions? I've done so for PIL once for a tool which had nothing to do with ISDC. END OK Peter, I suspected I could but it turned out quite easy to use Fine. the fitsio too so I went for that. END (Thanks for the hint anyway.)END > Just a quick note chaps: I've just had an exchange of emails with Tess, and > j_calib_gain_fitting 7.0 is now running in the pipeline with the new version of > the gain history table, and every seems to be okay. So we should be getting > good gain history out automatically now, if not actual good energy values, yet. > The current j_cor_gain uses the new gain history okay, but it doesn't ignore > gain dips or crazy values at the beginning of the revolution. END A note from my side there: I've started creating gain history tables for the PV phase with j_calib_gain_fitting 7.0 and will make them available to everyone. END Carole Anne: your turn on offline tools, I guess. END > Okay, well, it doesn't look like we'll be needing j_ical_frss any more: all > the data's there in the gain history table, so you can easily compare time > intervals of your choice and gain between different anodes. What we do need > is a tool to sum up the events for given X values to check for dead anodes, > and this could be incorporated into the tool for finding the collimator footprint > by summing up shadowgrams over all the slew data in a given revolution. > I envisage two outputs therefore. It's possible that the summed shadowgrams > for the checking the position corrections (droopy, warped collimator footprint) > will also show up the dead anodes. These things are several weeks from being > coded - pipeline stuff naturally takes priority. END I'd like to come back to the question I was preparing when you said something about the offline toollooking at all the slew data The question is: how have you thought to get the necessary data? Are they not private? After speeking with Peter I suggested to get them in the same way as we get engineering data, or something like that. Was it what you had in mind? One could use GPS and GCDE data plus, naturally, the calibration observations. END This would give you an anode check about every 12 days in normal operations. Another option with a different strategy is to create a tool that runs in the pipeline and dumps out binned data not usable for normal scientific analysis. > What about slew data between observations: who do those belong to? END Also to the owner! END The coming Crab calibration should give a hint if 12 days of interval > Can we add a new pipeline tool, say in the revolution file pipeline? The single > SCW approach leaves alot to be desired. END for anode checking is appropriate. Else the extra tool option should be implemented END The revolution _file_ 'pipeline' has its actions triggered by the arrival of certain files in the raw data. It cannot access science data in the ScW pipeline => no chance there. END But nothing keeps us from providing, e.g. one histogram over RAWX per ScW (or slew) and to combine these histograms in an offline tool. I'd help in the implementation but need some clear ideas of what is needed. END > In that case I can tweek j_prp_verify, so that is makes all these files and > an offline tool to compare them. END This sounds like a good idea. So we should soon: - define what we want - setup templates - write SCREW(s) accordingly and then go and implement those changes. > Okay, I'll put this on my to-do list END > Next: Sami how's offline tools working for you? END I really haven't got a time to put my attention to offline tools. I have made earlier one tool to examine bkg database and modify it but it's still really early beta version. END > How's performance monitoring going - does it tell us the things we need to > know? END I do not see any specific performance monitoring done by anyone. This would be a task of the Instrument Representatives, but I guess one would first need to decide what should be checked beyond what can be done by OSM and looking at allowed sciencedata. END I delivered the performane montor collection tool a long time ago. Peter send us an email where he said the table is readable with hk osm display tool. After that I haven't tested that. END sorry,performane=performance. END Soeren is analyzing the IPF files for this on an almost daily basis END > Okay, any more on Sami's tools? If not let's move on to Silvia and the scripts. > END Peter, do you want to talk about this ? ENN He delievered a new version yesterday . END delievered = delivered ! Sorry END The only change this time is that the scripts now have a hidden parameter "instMod" to point directly to the IMOD file one wants, > Are there any changes we should know about? END thus avoiding having to mess with the IC data to test a new Instrument Model. END Anymor changes ! Next week, I will start to work again on the scripts to perform minor changes. I hope to have a new version in a couple of weeks ! END My worry is to introducce BKG step in the pipeline, we have not done yet ! END Don't worry, i often forget to skip it and it runs fine. Just that the results make no real sense yet until Sami has real-world models. END Sorry for skipping back to a previous item: That nice to know that the executables are working! END Since last SDAST meeting, we have delivered two versions, one on 12th of DEc. and the one fom yesterday. In the firts one, the main change was to introducce the new GUI interface ( I suppose you have seen already) and some minor changes ( parameter checking and nearly all) END Thanks Peter ! END > Is that all for scripts? END I think so ! END I think yes. We plan to improve various points (like actually having matching .txt files ...) But in general things work fine. END Yes, I can confirm that! END Any questions ? END > Now ADD updating: do we have a date set? I can't find one on my calender: > is there anyone out there who knows when we're supposed to have it updated? END Sorry for being so chaotic in the scripts comments. Tomorrow the Science Minister is coming, and Victor wants him in our desk. The clen woman just came,complaint because she has to clean our office ! I am very sorry ! END clen= clean END I have a faint notion that it should be done early February END I thougth it was January 20, 2003... END > Well you can wipe that idea from your mind.....it simply ain't getting done > then! I think March is a more realistic timeframe, because it always takes > me at least a week to do all the cross checking with data structures and > parameters and so on to remove all the obsolete ones. I think we should > leave it open then. END Ok. I agree with that, because I am still waiting for some inputs about fluxes (NJW, SL, PK :-) END I agree with Carol Anne. I have not yet improved the scripts help files ! I will do it soon Peter, I will do it ! END > Okay, next point: jemx.h update. > This file hasn't been updated in ages. I'm pretty sure it's got obsolete > data structure names and stuff in it. Could we, around the time of the > ADD update make an effort each to look through the current version and come > with comments, additions and subtractions? Also, has the idea of delivering > it separately been forgotten. It's currently static state would indicate > it's ready for this kind of treatment. END Not completely forgotten. Maybe we should really get at this. My proposal is: - We define a new software component (jemx_lib) or so - As a start, this component only contains jemx.h, we could add common functions later - Developers have a choice between having their local jemx.h and mentioning jemx_lib as a dependency on delivery. Does everyone agree. END Yes, I do ! I like the idea !END Y Y y So when would aim to have a first delivery (I'd do it) - next week already? END > Yes > I think the file needs a bit of a clean out first. END > but if you'd like to deliver it as it is.......it has been unchanged since > June, which probably speaks best for it. END > Okay, I'll send you the newest version Peter, and you can deliver it just > to get the ball rolling, but I still say we need an update fairly soon. END > Next business: science results. How's it going chaps and chapesses? END Trying to understand why EXO2030 appears and disappears from one pointing to the other ! END Yes, good question. And also from one instrument to the other... END In my analysis of revolution 19, the source is detected in the same SCWs in both instruments ! END A question: has anyone looked into why our Cyg X-1 spectra are so far below that of RXTE? END No. END To Sylvia: Not for me!! But I have also it in Scw 90 and 100 for JEMX2. END To Jerome: That is interesting ! I think we should compare our input parameters, what do you think ? END I have looked at Cyg X-3 in rev 23. It is detected in both JemX's over Yes. I have used 20 and then 10 as detLev. END 7 SCW's but one is REST and one is FULL im. mode. END Unfortunatly I have to leave. There is a small boy waiting for me (his mother will go riding). END > have a good weekend Stefan! END The same! END Bye Stefan! Enjoy the afternoon and the weekend, Stefan ! END > Perhaps the rest of us should call it a day too: we'll pick up with this The afternoon is left... Just a short note from my side on science results: > agenda, starting at `Science Results'. Is that amenable to everyone? END *** Signoff: larsson () YES. END Y Y In one ScW of the GPS of revolution 30, I'm quite convinced we see GX301-2 and halfway convinced that we also pickup 1145.1-5141. So the list of sources seen is slowly growing ... END Otherwise I agree to call it a day. END > Is this normal-people-convinced or astronomer-convinced? END Which one is the more strict? END I'll also leave you now, have a nice evening! END Lets try to finish ! END *** njw has left channel #jemxadr : (njw) I have to discuss with the clean woman , what fun ? END > Have a good weekend everyone - there will be a chat next week - by which On Science Results, Jerome I' d like to know your input paramters for rev. 19 ! And to compare results ! END > time the wonderful new super-smooth, gain-glitch-free j_cor_gain will be Ok, you 'll get a email. END > running! END Thanks Jerome ! END Carol Anne has a nice evening and a nice weekend ! END Sounds good CAO. See you next week in our office. END > Bye everyone! END