Interactive background handling chat #1: 2000/11/09 Participants: Peter Kretschmar, Niels Joergen Westergaard, Sami Maisala, Carol Anne Oxborrow, Stefan Larsson, Allan Hornstrup, Juhani Huovelin Agenda: 1) What quantities must a user be able to see plotted out? 2) What quantities would be nice to see but are not essential? 3) What quantities would the user be able to change? 4) What software can the user run once quantities in number 3. have been changed with the aid of 1. (and maybe 2.)? Coarse image reconstruction? Refined image reconstruction? Simulation of a desired sky background? 5) How will the user know that the result is `better' or `worse'? 6) How detailed with the user manual be for this tool? 7) What data do we need from other missions to help us decide on all of the above? 8) What are user expectations of this tool? Action Items: IBH1.1 SL, SM 10/11/00 Circulate emails with a summary of the design OPEN requirements for their executables that were decided during the chat meeting. IBH1.2 ALL 15/12/00 Read and comment SL and SM's emails as OPEN preparation for more detailed AD figures IBH1.3 SL, SM 5/1/01 Make and cirulate figures showing proposed data flow OPEN (data structures) for their respective IBH tools/executables as preparation for meeting 9/1/01 **** Logging Started : Thu Nov 09 09:27:39 MET 2000 > Good morning NJW? Ready to party....I mean chat ? END *** Peter (~pkretsch@isdcul10.unige.ch) has joined channel #jemxadr Sure, but there's a lot to read just before chatting Good morning, Peter. END > Hi Peter - thanks for the email END Good morning. Sorry for the belated email, yesterday I just did not get around to it. END *** larsson (~larsson@tenma.dsri.dk) has joined channel #jemxadr Good morning, Stefan END Morner down there.. END *** smaisala (^smaisala@mizar.astro.helsinki.fi) has joined channel #jemxadr Hmmm should be Morning (I am a little tired maybe..) END Good morning, Sami. END > Hi Stefan and Sami - thanks for the emails. I'll just go and see if Allan Good morning END > is with us or not. If not we'l start right away. END > Allan's not in the institute today, so perhaps NJW can get the ball rolling: *** huovelin (~huovelin@taurus.astro.helsinki.fi) has joined channel #jemxadr > could you perhaps summarise the purpose and extent of the interactive > background handling? END *** allan (~allan@uhuru.dsri.dk) has joined channel #jemxadr Good morning, Juhani and Allan END > Hi Juhani - it's good that you could make it too! Hi Allan - I though you'd > forsaken us! How could I think such a thing? END > NJW's just going to start us off with a summary of the common ground END Alright, it's great that we're all here - shows the importance of the subject. Sami's mail describes nicely what the IBH (Interactive Background Handling) Hello all. I try to sneak in the subject, being away from it for a (too) long while END - may I introduce such an acronym although it is a bad habit, but for low-speed typers it can be practical. We are to discuss the ambition level for this new tool that we are planning, what it must be able to do, and perhaps also where we can benefit from other, already existing, software, such as ISDCROOT, as Peter mentions, maybe can be used to perform tasks already in the ISSW for a specific purpose in the IBH. For the rest of the discussion I think that we can follow the agenda outlined by Carol Anne so: What would you like to be able to plot ? (BTW, CAO do you save the chat for the minutes ? ) END > Yes, I'm loggin the whole thing: so let's start with point one: > What, and how should we plot the available information. Stefan, you have > some very good ideas in this area: are there ways that Stefan's ideas can > be extended to encompass all the things that need to be visualised? END Stefan's proposal for a hardness-color plot is a good one, but it relies on an effective 'removal of point source contribution'. We will face two situations: one where the shadowgram is dominated by strong sources and one where the point source contribution (seen from a JEMX point of view) is small. END > Does anyone have any ideas how these two situations might be handled? END For week sources the hardness shadowgram might be usefull without source subtraction. In case of strong sources they should be subtracted first. END Maybe we should start with an implementation of the weak source case which is where we need a good background subtraction most, and in a second step add the source contribution removal. END Is the conclusion then that we need a tool for the removal of known sources ? END Sounds like a reasonable strategy. But I think we will need the source > Yes, couldn't such a thing be used iteratively - remove the sources with > the current background estimate - improve the background estimate - remove removal also for other things (calibration eg). END > the sources again - improve the background again etc. etc. until grant money > runs out END To Carol Anne: in principle yes, but one would have to run image deconvolution inbetween. END So I think we agree that we want the abiliity to interactively remove Not necessarily, Peter, I think that Stefan's software will be able to a source contribution from a shadowgram. END extract the source properties without a deconvolution. Only the search for even more sources in the FOV will require a deconvolution. END Agree, Stefan ? END Thats right NJ. END OK, but I understood Carol Anne to the effect that she wanted to find ever more sources and thsi would need image deconvolution. And one could argue that deriving source properties is some > I think that depends on what you expect from the IBH: to clean up the sort of deconvolution - the classical IROS is nothing else. END > already existing image and sources, or remove background to find fainter > sources? END I had in mind that more accurate source properties would lead to a better subtraction of the known sources. END One thing I don't quite understand right now: Stefan's tools (which I'd like to get delivered, btw :-) ), work on event lists, _not_ on shadowgrams, as far as I remember. So how do you model the contribution of a source to a shadowgram with this? END (That make two of us Peter, END) (that comment was regarding delivery) Let me think how to state this. Stefan, don't you assign a probability for each event to be coming from the background and from each of the sources in the FOV ? END Thats essentially right NJ. Actually I calculate how different areas of the shadowgram contributes to a particular sources. > So it should be fairly easy to remove source photons, even if re-deconvolution > and other processing thereafter can be expected to be quite slow. END It means that i do calculate all information needed to remove source contributions in the shadowgram. I just have to go throu all events and take away a fraction corresponding to the source. END From what do you take away a fraction? END In principle I can take away a "fraction of an event" but in practice > Surely you can't just remove a random fraction in a given pixel - we can I will bin it up. END > expect the source and the background to have different spectral properties. > Could the spectral modelling of the background allow us to determine WHICH > fraction fo the events get removed? END It is correct in a statistical sence. I cant say that exactly that fraction is source. END So the subtraction of source photons (fractions) can be done in energy bins ? END For spectra I will do this for each energy range. END So a source contribution removal tool can be based on Stefans software and if we agree that such a tool is required we must decide who is going to be responsible for it END Stefan I'd say ;-) END Have to agree with there Peter..... END > I agree: Stefan's our sources expert, just as Juhani and Sami are our background > guys. END Ahhh, my morning cup of expresso was just delivered to my desk!! END > I also agree that we could do with this tool, of course END Alright, that's good. We will not forget the simulation aspect, perhaps as verification in special cases END The question remains if this source removal "tool" is an executable (called interactively via ISDCTask), a function or osme sort of ROOT application. END executable. END This was a useful decision, but we should perhaps return to the agenda: > Yes, with SWG input: shadowgram, source list etc. It should be able to run * njw what kind of plots do we need ? END > by itself, and iteratively even. END Anybody want to talk about plots? END We need to plot a shadowgram with specified energy and time selections > Yes, very much. What needs plotting and how informative can we make the > plots. I like the idea of colour coding detector maps/shadowgrams etc. to > give maximum user information. END and we need to plot spectra from a specified section of the detector END Let's get clear here. It seems to me that in many cases you need a tool to read-in and build the information rather than a simple plot tool. As a start we have events, let's say Full Imaging for example. We have an existing tool to build shadowgrams from corrected data (j_ima_shadowgram) In additin some generic isdcroot tools will be able to just read in and sum up all raw data and plot them. With ROOT's classes there is no intrinsic problem to plot any shadowgram or spectrum _that__someone__has__built_. END So what do you want: plotting capabilities, another tool to build shadowgrams? A list of tools to manipulate shadowgrams? END Case of shadowgrams: at least the j_ima_shadowgram needs to be modified because presently it only produces a shadowgram in the 'skew' format. But with quite small changes one can have more natural representations. What is the situation for spectra ? END Do you mean source spectra peter? (NJ that is , Sorry END) I think we are talking about detector spectra here. No, I mean spectra of events selected from a time interval and a section Good! of the detector END j_bin_evts_spectra could be extended to also take a position selection Correction: it would do the job already as it has a parameter rowSelect that is passed on to DAL3JEMX and which could contain position selection. END can it select circular rings Peter? END So from the selection of data to plot we seem to have basic tools, what Only if you can formulate that in a "DETX>=... DETY<=..." logic, I fear not. END about selecting a specific background model ? END If the instrument background changes with radial distance such a capability would be very usefull. END I agree to Stefan's comment END I could add this capability. END Sami, do you have some means to present the data from a background model so that it can be used by the plotting functions of ISDCROOT that Peter mentioned ? END I think that is not problem if we are using root. We can use all DAL3 functions and we can read all wanted information from database and plot that information out. > This sounds very promising indeed END Question is: what kind of selection/userface we like to have? END I have one principal point regarding the plots. I think that it is important to use the extracted information (plots) to obtain quantitative information. from the radial plot of spectra, or hardness ratio one should fit a function or background model and not just look the plot. Sami, is your question for the IBH in general or for the bkg model spedifically ? END END We should try to think in terms of what QUANTITATIVE info we can get out. END I can fully support Stefan's wish for numbers. When the selection has been done, then it should be possible to plot the result _and_ make some fits and comparisons. END One possibility in shadowgram is 3D plot. ISDCROOT gives possibility to rotate image in 3D. END Let's get focused: Stefan (rightly) wants some fits and numbers. The question is what do we do with them, exactly? In our current scheme, our background model corresponds of E.g. select a background model . Or build one. END - a sort of shadowgram of background intensity We were thinking about a parameterized fit, which gave rise to a new BG being fed back into the imagereconstruction tools. - a map of index numbers of spectra - a list of these spectra Not just scaling of models, which already has taken place (right?) END Sorry, this is all too diffuse for me to understand how it is all supposed to work together. 1. Look at the BG plot. If it eg is skew to one end, fit a parabola or a line Is this some new data structure or are you replacing the existing model? END through. Define a new BG model and feed it back in. END What does "Define a new BG model" mean exactly? A replacement of the intensity normalized shadowgram? END We must be careful about understanding the interfaces. Replacement of the bg model used in the image. I am uncertain exactly how. END This question goes to all: What do you envisage as output of IBH? Basically the same data structures, just filled with different information? Or something completely different? If the former - where is the connection between the spatial fits and these data structures? END Sorry, spatial fits were just an example ... END *** Signoff: allan () > I think we could do to define separate IBH data structures, so that we can *** allan (~allan@uhuru.dsri.dk) has joined channel #jemxadr > always refer and compare with the SA results. the IBH data structures: > sky images, background models, source lists, source LCRs/spectra could all > be overwritten many times without worrying which version we're keeping END I dont think we should create new data structures for every case where the user might like to test or select different options I agree, Stefan END We should be very careful in introducing new data structures. I agree with Stefan. END instead the user must be able to store differnet sets of analysis My opinion is that outputs are the same data structures - but user can define own models and save those models somewhere and after testing select the 'best' one which replaces the originals. END results. A user might like to do lots of alternative analysisisisi. END Some notes about Offline Scientific Analysis: We do foresee to support multiple alternative analysises At the moment, this is thought to be organized by having many 'observations' sharing some data and files while having their separate results for other stuff. There are surely many detailed problems to be resolved, but there is no need to multiply data structures to keep different results apart. END So most of us seem to agree that the out of IBH is suppose to be the same data structures we have already defined. END Can we sketch out the ways to arrive there today, or do we need more time to think? END Btw, Sami still hasn't seen too much responses to his questions about user interface. END > I'd prefer identical data structures to those from the SA, but with slightly > different names (IBH in there somewhere) to indicate that the data is not Yes, the IBH will take over from SA at the point where the background enters > arrived at by some well-defined set of automatic rules that can easily be the scene END > r ecreated, but by a good deal of playing around and heaven knows what. END To Carol Anne: I'd rather use keywords for that END Carol Anne: if the user wants to have the automatic background, she just runs j_bkg_model. If she wants her own best guess, she runs IBH. Both produce the same data structures. The idea of flagging the origin in keywords is OK. The SA results will be in a different directory tree under a different OBSID than your OSA results. END *** Signoff: allan () To return to Sami's question on interface: you mean for IBH in general? END > The use of keywords to flag the processing origin of the data sounds good END > The ideas bounced around so far sound very good. Just to firm things up could > we put an action item on Stefan to make a figure with proposed data flow and > processing indicated, so that we can really see where all the information > comes from for this tool and what we do with it? END For the overall flow I would suggest sami. The IBH if after all his task? END May I suggest that Sami makes a small note, including a figure, on his > That's two AIs then: one on Stefan to produce a data flow/processing figure > for the source removal tool and one on Sami for IBH in general, including > where he proposes to add the source removal executable. END ideas on IBH - this chat has produced some good information - and distributes it ? END I have to go for lunch. I think we would need to discuss this in a physical meeting. Please consider it. END *** Signoff: huovelin () Is there a need for a physical meeting before January? END > Okay, let's have the two AIs as preparation for the physical meeting. > I think the meeting should be in January with the general SDAST/QLA meeting. > END * njw Are there any deadlines before January, besides the ADD5.1 ? END > Not that I'm aware of. END Neither me. END Then I can support CAO's point. END Fine with me. END Me too. END fine with me END But Sami & Stefan should still circulate what they made out of this discussion soon, while we still remember. We should have a good gorundwork for the physical meeting. END > I agree. Two AIs EACH - a quick mail to sumarize what was decided at this Yes, better sooner even sketchy ideas will be worthwhile to keep the kettle boiling END > meeting within a day or so, and the figures and maybe AD descriptions ready > just before the meeting. Is that Okay? END Yes. END Yes. END > First AI has due date by tomorrow(Friday), and second AI is due 5th Jan - > agreed? END Y YYYY. END > Perhaps there should be an Ai on the rest of us to read and comment on the > emails, before they get started on the figures and more detailed suggestions END OK OK > Shall we have one more chat dedicated to this subject before the physical > meeting? Afterall, we didn't get so far with the agenda today....END Lets wait with that decision for a week or so and let this sink in. END I agree with Stefan. END But my guess would be that we will have more topical chat meetings. END > Okay, let's call it a day then. I'll circulate the updated AI list ASAP. END While talking of chat meetings: both next Tuesdays are bad for me. In any case, I think that we need to speculate about the interface. Can we have our next chats on Wednesday? END Sami never got an answer, probably because we don't have any firm ideas. But we should have thme soon. END > Sure, Wednesday is fine for me, what about the rest of you? END Wednesday is OK. END > 9.30 as usual? END Yes. END Next wednesday is fine. The 21 nov I have another meeting in the morning. END The time is OK. END Regarding user interface, I will try to get information flowing between Sami and Reiner Rohlfs here, who has already nice tools to plot spectra and shadowgrams. END > We'll settle on a date and time for the chat two weeks from now next week. END > Chat with you all next wednesday. AOB? END OK. bye for today. END Good bye! END Bye, bye END > Bye everyone! END bye END *** Signoff: larsson () *** njw has left channel #jemxadr : (njw) *** Signoff: smaisala (njw) *** Signoff: Peter (Leaving)