Background Problem Chat#2: Friday 29th January 1999 Participating: Peter Kretschmar, Carol Anne Oxborrow, Niel Joergen Westergaard, Allan Hornstrup, Stefan Larsson, Niels Lund, Gerry Skinner(part-time) and Juhani Huovelin Subjects discussed: Agenda for SDAST meeting Febuary 1--2 1999, the principal subject of the meeting to be background handling problem. Countrate data Background handling discussion document Action Items: None. *** IRC session log started at Fri Jan 29 09:27:01 1999. #jemxadr> Hi, Peter! It's REALLY cold here! END *** njw <~njw@heao1.dsri.dk> joined channel #jemxadr #jemxadr> Hi NJW - good job on the background problem: even I can almost understand what's going on END Hi Carol Anne, around here we had quite a bit of snow even down to the lake. REALLy cold means how much, btw? END thanks Carol Anne - actually most of the discussion comes from Helsinki END #jemxadr> Enough to make your bottom cold for a couple of hours after cycling to work! END #jemxadr> No snow yet END H njw #jemxadr 0 Niels Joergen Westergaard ~njw@heao1.dsri.dk H oxborrow #jemxadr 0 Carol Anne Oxborrow ~oxborrow@130.226.216.181 H Peter@ #jemxadr 0 KRETSCHMAR Peter ~pkretsch@isdcul10.unige.ch 315 apollo.dsri.dk * End of /WHO list. *** huovelin <~huovelin@taurus.astro.helsinki.fi> joined channel #jemxadr #jemxadr> Hi Juhani - good job on the gackground discussion document. Also, thanks for the list of refe +rences, I got most of them. END *** allan <~allan@apollo.dsri.dk> joined channel #jemxadr #jemxadr> Hi Allan! END Hi #jemxadr> I hope Stefan will be with us soon. He just sent me an email to say he has an allergy to pres +ervatives, so we'll all be eating very healthy, home-cooked chemical-free food on monday evening. Be w +arned! If any of you have a miniumu daily req #jemxadr> uirement for additives in your food, I can always throw in a few bowls of potatoe chips! END #jemxadr> We'll wait one more minute and then get on with discussion of the agenda for Monday's meeting + END H allan #jemxadr 0 Allan Hornstrup ~allan@apollo.dsri.dk H huovelin #jemxadr 0 *real name unknown* ~huovelin@us.astro.helsinki.fi H njw #jemxadr 0 Niels Joergen Westergaard ~njw@heao1.dsri.dk H oxborrow #jemxadr 0 Carol Anne Oxborrow ~oxborrow@130.226.216.181 H Peter@ #jemxadr 0 KRETSCHMAR Peter ~pkretsch@isdcul10.unige.ch 315 apollo.dsri.dk * End of /WHO list. I have nothing to add to the agenda. I believe the finishing time tuesday is dictated by + departure times ? END #jemxadr> Okay, count down to agenda discussion...5...4...3..2..1..Now END *** nl <~nl@heao1.dsri.dk> joined channel #jemxadr sorry for the head start END #jemxadr> Hi Niels, good to have you with us. We're just discussing the agenda for the meeting. Anythin +g you'd like to add? END No, today I am just the fly on the wall I have included a small discussion of the link to ISDC - requirements from the SDAST to the ISDC. THis is an AI on me from ISDC , that I need to get out of the door. It will happen around noon monday. END #jemxadr> Any other additions? END By the way, is it very cold there in Copenhagen ? Here we have cryogenic weather EN +D Currently around -5 C END No, I just hope we get this annoying raw data formats discussion finished. END #jemxadr> I'm scheduled to discuss DD, but I really don't have a lot to say: we're doing it. I haven't +heard of any particular progress from others. Perhaps during that spot we could each just give a VERY +brief account of where we are on DD. END CAO: That was my intention, your name was on there just to control the discussion... :-) END Fine with me, except that there's not much to tell. END #jemxadr> I have things to tell about my work, but not other peoples'. END Peter: Do you think we should extend the telemetry-format discussion? END #jemxadr> Is it even worth discussing data structures for the next month or so, until ISDC know what th +ey want? END We need to know in which direction (or lack of?) ISDC is heading END *** larsson <~larsson@tenma.dsri.dk> joined channel #jemxadr #jemxadr> Hi Stefan, we're discussing the agenda for the meeting, and specifically data structures and +telemetry formats END Good morning. My train was stuck for 1,5 hrs this morning (its cold!) END It is worth discussing data structures. Currently I plan to present two clear options +and hope to have an _informed_ decision between these two. The whole current problem essentially arose because SP and me had an idea which _almos +t_ worked. END #jemxadr> If there are two possiblities that require our input, then of course, a thorough discussion i +s required. END Stefan : About the agenda, we would like to know if you have found issues missing. END Nothing on my mind now. END 1~I am trying to clear up the uncertainties in the Count Rate Format with Grzegorz, Soren + and Ib. Hopefully we have it sorted out by monday. END That would be nice END #jemxadr> Even if it's not complete sorted out by then, a clarification of how things stand would still + be valuable. END Niels: Tuesday could do - :-) END Tuesday it will then -what about wednesday? No. We need to finish the issue before we leave (that is before lunch Tuesday END #jemxadr> So there's nothing except allan's ISDC AI-based issue to add to the agenda? #jemxadr> and we all know that we have to give a summary of our current DD status? #jemxadr> And NL and the other instrument `boys' will regale us with tales of countrate data. Is that a +ll? END Seems so END #jemxadr> Okay, on to the discussion of the background handling problem. #jemxadr> I hope everyone has seen the lovely document sent round by NJW. It sets out the whys and wher +efores pretty succinctly I think. END Could we please hold this discussion for 5 minutes or so? END I have invited Gerry to participate from 10 (Danish time) and unfortunally, I have forgotten to include his name in thte list of persons receivingthe instrucitons about how to IRC. He has received them now (I hope), but should be given a few minutes to get the system running (if possible) END. #jemxadr> Perhaps the most fundamental question, as I a beginner and outsider see it is on page 4, sect +ion 3.1: what will happen if we don't do anything with the background? Is there ANY way of knowing thi +s before it's too late? END #jemxadr> We'll wait five minutes for Jean and Gerry. Any one want to compare low temperatures? END I should add, that JEan has already announced not to be able to join the Chat, too busy with SAX proposals,... END #jemxadr> Could you get on the phone, Allan, and see if Gerry's actually going to join us? END He is trying ... END While waiting ffor Jerry I can report that the INTEGRAL project has agreed to implement t +he hardware to eliminate the cross-talk between the two JEM-X units. END Good. END very good, Niels END vertically or horisontally? END #jemxadr> Good: any further news on how long the launch might be delayed? End They will implement the horizontal solution - two shelves END No news on launch delay END Except that a delay 'will happen' ! END #jemxadr> I just got an email from Gerry - he's trying to connect even as we chat, so be patient. END #jemxadr> How's Oersted today, Niels? Or is that a sore point you'd rather not discuss? END The ORSTED team is relieved that the Delta-II did not go up in flames after the abortewd +launch attempt yesterday. Next launch attempt will come in about 10 days. END #jemxadr> Look on the bright side: everyone's really going to remember this satellite: if it had gone u +p first time they'd only have remembered it till the next royal faux pas END Question to njw: why did you expect the background counts to have a peak in the center of + the detector? #jemxadr> Are there any non-background matters we should be discussing: how's the coding and DD going? +END (Gerry is not answering the phone. I have even sent instructions about What about the Monthly report ? We may not have much to report END how to log on here and do irc. he's probably busy installing the SW on a machine not in his office?). END Coding/DD: I've just started on my first executable, no real DD yet, but some coding w +ith PIL, RIL, ... END #jemxadr> About monthly reports: I'll send a new report template to you all next wednesday or so. Numbe +rs for percent complete on various phases, are apparently expected by ISDC at the very least. END We are also tstruggling with PIL, RIL and DAL, but just only managed to start succeeding in using them END #jemxadr> This month you should all at least be able to report so progress with DD/prototyping/coding: +let me know how much on the template. END Reply to NL: because the edge of the detector could be more effectively shielded by mask edge and collimator END And we will have to comment on the data structures in the m.r. :-) END #jemxadr> I've started from the bottom up: I'm leaving all the ISDC interface stuff till the very last +bit of coding, so that everyone else can discover the problmes with the ISDC libraries: I don't want t +o be wasting too much time on these things #jemxadr> while there's so much other coding, actuall processing, to be done. END To Juhani and Sami: there are some examples of the use of DAL and PIL in our developer's + Forum END #jemxadr> We'll worry about the actual contents of January's monthly report after the meeting. END You now, for my first tasks, actual processing is almost zero, so it's natural to star +t with these interface problems. Also, I'm just learning C en passant. Whoever thought up string handl +ing in C deserves several eons in unpleasant afterlife ... END Hm - we should be able to write it today - but never mind END BTW Sami, how is the simulation going ? END We have already utilized NJW's examples, and also the simulation seems to work END #jemxadr> Please, if anyone has grappled with the complexities of the DAL and found some actual working + code, sent it to me for inclusion on the forum webpage. END ABout Gerry: he claims in his email, that he is trying to install the SW, but I called + his office, and they claim he is at another meeting ? I think we should start the BG discussion when + the previous ends, and have a link on the WWW to CA's logging for Gerry to catch up, should he succeed to join END. Fine with me, let's start. END About section 3.1 (How serious is the problem?) can I report that 3 (not very experience +d) students will start working on this question and put GEANT/TIMM results in the #jemxadr> Allan's just sending a message to Gerry with the URL of the chat log, so he can catch up, if +he can reach us. END simulation END Where do you hidethose results (spatial distribution) ? END The students will actually be present at the BKG discussion part of our meeting monday/t +uesday END #jemxadr> That's good, about the students. I'm interested to hear the result of their endeavours. END To Juhani: I don't have them at the moment but Marco Feroci has the result of at least o +ne run of GEANT. END #jemxadr> Also, is it possible to produce end products, with and without background corrections, for th +ose astronomers who simply must do everything for themselves. END It must be possible for the observer to redo the background corrections. END We intend not to do the part of processing automatically , but leavethe applying of + background correction to the user END Is there an onvious way for this facility, given our current architecture? END #jemxadr> Or leave them out entirely. END Let's get this sorted out for me: normally image reconstruction includes bakcground su +btraction in our current scheme - right? I thought not (to Peter) END If this is true, how do you precisely foresee the user to chose a different or no back +ground? I have no doubts it is possible, but I'd like a precise description of how and where. END In the image reconstruction by correlation the background is removed 'automatically' under the assumption that it is uniform, which it is not. I think, therefore, But I think they mean the instrument (uneven) background (to NJW) END that at least a simple version of the background model should be applied END #jemxadr> The astronomers I know seem very wary of any data they considered to be `messed about' too mu +ch, so there must be options to process the data as little as possible, if that's what the end-users w +ant. END As long as we supply corrected data, not background corrected, and access to My impression was that we should provide weights (or probabilities) for each event +to a background event, and then someone could just apply that information in his/her analysis END the background libraries, it should be fine for guest observers END #jemxadr> Yes, the background library access seems like the right way to go. ENd To Juhanis comment: Those weights will depend on backgr. model END Yes, but one of the problems is, that cross-correlation will always remove some backgr +ound and it might be wiser to try and remove 'all' when deriving those images and source data in autom +ated analysis - or even interactive, as long as the user choses to run a 'standard' pipeline. END to NJW: That maybe be too dangerous, since the user (guest observer) may not have e +nough experience on how to apply different ways of determining the background) END I think we may include some simple background terms (simple radial varying terms or unifo +rmly tilted planes in the correlation analysis - this may be good enough to allow source identificatio +n - detailed analysis must be left to the user. E #jemxadr> Rolling the background corrections in with image reconstruction seems like a one-way process +which cannot be removed. END To JH: Whatever we think of the GO's knowledge, the possibility must be there too redo + all. END Yes. That's right. I agree also with CAO END To CAO: I don't agree, image reconstruction and source finding can always be repeated EN +D The "serious" user will probably always start from pretty raw data END He/she will, but we do have obligations to provide something in the SA (and QLA). END Yes, but there could be different levels of "correctly" reduced data END To NJW: my memory is slightly hazy, but wasn't there a refinement of cross-correlation + that did not lead to average, smooth part = 0? END #jemxadr> Yes, you must expect the observer to have a certain amouunt of skill, commensurate with the a +mbitiousness of their project. If they want to do a `standard' analysis, directed by experts (us) then + they'll get one sort of data. If they want t #jemxadr> to be more independant, then they're going to have to have a little sophisticated knowledge a +bout backgrounds etc. It's a reasonable expectation from our side, I think. END I don' t think they can go ahead and publish SA data , I wouldnt dare, at least END To Peter: there are several options with cross correlation. One is to have the smooth di +stribution disappear END I agree to Allan's statement: the SA results are just guidelines END If we talk about fitting the data rather than cross-correlating the issue of handling the + background becomes more explicit - if not easier. END I am sorry to interrupt a nice discussion, but could we back up and do a minor agenda +for what we need to discuss today and what we rather should do Monday? END I suggest to have now (since Gerry is not here) questions directly related to the interpretation of NJW's docuiemnt. END #jemxadr> I think Monday should be devoted to the actual dynamics of background determination, while to +day we talk about more general issues: how much BKG etc. #jemxadr> How will we use background in our syste? What do users expect background-wise? END - if we need to go into these details, for how long will we extend this meeting END Until we have answered all questions we should on Monday/Tuesday (to Allan) END I'm not sure we can deal with all the topics CAO raises today. #jemxadr> As for the document itself, it seems very complete in setting out the problems and the list o +f things to be answered - and a good place to start an involved discussion. END #jemxadr> Strictly from the document point of view, I don't thinks there's anything to discuss today. E +ND I agree . Maybe we should stick to a short meeting today END My impression too END A lot of this seems to be rather fundamental and linked with the various mental pictur +es of how the software works and will be used. END #jemxadr> Since Gerry isn't with us, shall we all call it a day? END In that case I'll call for specific comments to the document - is there any ?? END One moment please, I have one more point. END *** guest <~guest@tenma.dsri.dk> joined channel #jemxadr #jemxadr> Hi `guest', please identify yourself. END NJW, SL and me should summarize how background is supposed to be taken into account wi +thin our tasks of the ISSW, i.e. ImageGeneration, SourceDataExtraction and DataBinning. This summary s +hould point out who this treatment may differ between QLA, SA and offline analysis. END msg While the ADD says a lot, it seems to me this information isn't fully "on the table" f +or everyone involved. END I guess our guest is Gerry Skinner, right? names END msg help help msg Gerry i f you hit /who you'll see who's on. Further, just type ahead, and we all see what you write. END (We end sentences with END ) END #jemxadr> Hello Gerry, it's good to have you with us. If it is you. END Guest is Gerry Skinner , but I have no idea how this system works It works fine, as you see END sorry, I have to leave you now - see you on monday Welcome ! END As long as you just type text, everybody can read it. END #jemxadr> I'm logging the chat session, so you can catch up with us if you want at the address allan se +nt. END *** Signoff: nl () OK, I genin to get the idea END The URL is sent as email, did you get that? END To Peters comment: I fully agree, we obviously need to clarify this END NJ/Peter: Agree. END Clarify what ? END #jemxadr> Gerry, could you give us any comments you have on NJW's discussion document, since we've deci +ded not to discuss the technicalities of background determination itself. END Having made contact, I think it is probablyt best if I just wait to review the transcr +ipt after wathing any further transactions - unless there are specific issues/questions for me. END #jemxadr> Specfic issue: what do you think of the discussion document: anything missing? Anything that +can be discarded immediately? etc. END My comments on the discussion document are that it is a very good start, and the most +of what I could add is very detailed. Perhaps the main thought I had was that filtering in the detecto +r plane is largely equivalent to filtering the image END #jemxadr> I think we should definitely leave the details to monday. But we're all satified with the doc +ument: many thanks to Juhani, Niels Joergen and Stefan. END OK. It is much easier to be around the same table and ask the questions then END #jemxadr> Any comments on detector plane vs. image filtering? Does this affect the document now in any +way? END I think that's a kind of "detail", which cN BE LEFT TO MONDAY END #jemxadr> We maybe don't have to consider these as two separate issues? END #jemxadr> Well, then, now that the agenda is agreed, and the document approved for detailed discussion, + I think we should finish for today. It was just that the document seemed to assume that filtering was done first, before i +magein END #jemxadr> apologies to Gerry, who only just joined us. In future we will host somewhat more regular bac +kground chats, and I hope that the knowledge and setting up you've done today will be of help to you i +n the future. END Wait a second, CAO. Gerry, could you elaborate a bit on the term filtering: does it mean smoothing on a spec +ific scale ? END To Gerry: the instrument on-board hard- and software will filter out a large fraction +of the particle bkg, at least this is expected. Do you refer to this? END Yes, I use the term to refer to correcting slowly varying levels, either by filterimg +out low frequencies or by subtracting a polynomial or spline or whatever END Sorry, the YES was to the comment about smoothing on a particular scale. I did not mea +n YES to Peters conclusion that I was talking about background rejection such as done on board. END to Gerry: But wouldnt it be different in the two frames of reference, since the ins +trument background is not dominated by mask coding END No problem, it was very clear from the rest of the sentence. END Are we then dressed to continue Monday? END #jemxadr> I think that it's important that we all begin on monday with the same terms of reference. #jemxadr> Therefore, could I suggest that we start the discussion with Juhani presenting a little on th +e background problem and THEN Gerry do the same, so that we can found out the different ways we each d +escribe the same things. eND good suggestion END Gerry, If you have any additional points or questions to us regarding these issues you could maybe send us a mail during the day (will give us something OK. I'l try to tell you, what's bugging me in the background END to think about during the weekend)? END #jemxadr> We don't want to find out after a couple hour's discussiont that we've been talking at crosse +d purposes. END I am logging off now. I will look at the transcript and if there is anything else we can exvhange e-mail or leave it to MondayBYE #jemxadr> Allan, could you puton the more detailed agenda 15 mins each fro JH and GS to present their c +ases? END #jemxadr> Bye Gerry, thanks for joining us. See you on Monday. END I'll do that. Agenda comes out right after this END See you Monday, have a safe trip! END See you Monday - have a nice day and then week-end BYE *** Signoff: njw () See you all on Monday - looking forward to it! END #jemxadr> See you all on Monday. Have a good weekend. END Bye bye, see you monday END *** Signoff: huovelin () quit See you all on Monday. And to CA: Mineral water is great! END #jemxadr> Bye! *** Error: Closing Link: oxborrow[~oxborrow@130.226.216.181] () *** IRC session log ended at Fri Jan 29 11:18:48 1999.