Extreme integration times and stacking [Deep Sky] Processing techniques · Timothy Martin · ... · 63 · 3253 · 11

This topic contains a poll.
How stupid is the OP?
Very
Largely
Somewhat
Who cares?
AccidentalAstronomers 18.64
Topic starter
...
· 
·  1 like
·  Share link
But with my latest projects I used the manual mode in WBPP of Local Normalization, with this you can use the best 30 or 20 images to be stacked as reference. I also figured out, that you can apply decent gradient correction to those reference files before normalization starts (just open a new instance, correct and save before you select the ref frame).


Another thing to try during a rainy week. Thanks, Philip!
Like
kevinkiller 2.11
...
· 
·  Share link
but John Murphy discusses why he thinks NSG is ostensibly superior to LNC


If you could point out the place where he discussed this I would be very grateful.
Like
RideTheLiger 0.90
...
· 
·  Share link
Ani Shastry:
I wonder if you are somehow running into fp32 floating point precision issues when stacking the large dataset based on the underlying algorithmic implementation in either APP or PI; essentially where a full “multiply-add” across the gigantic number of subs before normalization (divide) is overflowing something that isn’t happening with a smaller set of subs.

Could also be potentially SIMD optimization related issues in the underlying implementation, albeit APP uses Java straight up from what I understand while PI uses a mix of Java and C++. Have you tried another program like Siril to see if it is any different?

Ani

Interesting thought, in PI you can do a 64-bit stack, but don’t know whether it is possible to turn it on in WBPP.

Probably manually creating a master for local normalization would also help. But now we are back to M(anual)WBPP
Like
AccidentalAstronomers 18.64
Topic starter
...
· 
·  Share link
John Stone:
If you could point out the place where he discussed this I would be very grateful.


It's not one particular place in his tutorial series, but the initial discussion is near the beginning of this one:

https://www.youtube.com/watch?v=1drKc0mEUgg
Like
jwillson 3.66
...
· 
·  2 likes
·  Share link
John Stone:
I'd like to point out the PixInsight now has a Local Normalization process built into WBPP that takes the place of the older NSG script.

You could save a lot manual labor if tried that instead...   just load up the lights, darks, flats into WBPP and press go.

I don't understand how or why a stack of all the images is producing less SNR than one made up of sub-stacks unless there's something funky going on with the weighting scheme as reported by @Jared Willson just above.



I re-ran my stack from scratch. I tried integration using PSF Signal Weight and got much more typical weights--running from 20% to 100% depending on light pollution and phase of the moon (and, to a lesser extent, seeing conditions, wind, and focus). Only 3 frames dropped below  my 15% threshold this time. Just for comparison, I also ran the stack using PSF SNR, SNR, and a weighting based on star count (which I had stored in a FITS keyword). I got very similar results with all of the above, but marginally better results with PSF SNR vs. all the rest. This was based on an SNR measurement (however PI performs that) as well as my own visual inspection when images were stretched to show similar details. 

So, my supposition that something was wrong with my weighting was not correct. I had likely made some error in how I setup WBPP. I am now getting expected results. All typical methods for weighting produce similar results, and combining all stacks at once or one stack for each day work about the same. The error rejection is much better if I run all stacks at once, though. That is to be expected.

- Jared
Like
Wjdrijfhout 6.78
...
· 
·  2 likes
·  Share link
Just a couple of thoughts come to mind here, @Timothy Martin :

- If you want to automate more, perhaps you could add a custom keyword 'Batch' to label your fits files into batches of say 200 files using BatchFitsKeywordEdit. In WBPP you can group your images by keyword, so all your batches would be individually processed and stacked. It would mean using Local Normalisation instead of NSG, but that's a very similar process.

- If you're stacking so many images, perhaps it would help to tighten the clipping values for pixel rejection and be more aggressive with image weighting? You could experiment with different settings by batch cropping your files to some very small fragments and run various integrations on them. If you have that many files, it becomes almost a kind of lucky imaging, where you're less interested in lowering SNR, but mostly interested in just the very best images in the batch.

- The concept of batch stacking is very common in extreme macrophotography. There you apply focus-stacking to create any meaningful depth of field. For high magnification macro, the distance moved between photos is in the microns and you may have hundreds or thousands of photos. Batch stacking often creates better results than all-in-one stacking.

Amazing project though, can't wait to see the results being released.
Like
AccidentalAstronomers 18.64
Topic starter
...
· 
·  1 like
·  Share link
Willem Jan Drijfhout:
Amazing project though, can't wait to see the results being released.


Here it is: https://astrob.in/yfy2hp/0/

Thanks so much for your input. There are a ton of things to try with this data. I could probably spend the rest of my life working on this data set!
Like
Habib_Sekha 1.51
...
· 
·  1 like
·  Share link
@Timothy Martin it looks very dusty…

WOW! Amazing! You sure know how to set gold standards! Thank you for your hard work and for sharing it.
Like
AccidentalAstronomers 18.64
Topic starter
...
· 
·  Share link
Okay, @Kevin Morefield -- once again you have identified one of my issues and proposed a solution whether I like it or not. I ran the test you suggested on all three scopes and the results were telling. In each case, I subtracted the 180-degree master flat from the 0-degree master flat and vice versa. Here are the results.

FSQ106
image.png

TOA130
image.png

CDK12
image.png

So from now on, I'll be taking flats for each unique rotation angle. I'm not at all thrilled about it, but it has to be done. Fortunately, I can automate that in NINA with some generic code in my main template. So that's not a hassle at all. But managing an additional 210 404MB files (84GB) for each rotation angle will be no fun. I'll have to work out a system for that.
Like
morefield 12.31
...
· 
·  Share link
Timothy Martin:
Okay, @Kevin Morefield -- once again you have identified one of my issues and proposed a solution whether I like it or not. I ran the test you suggested on all three scopes and the results were telling. In each case, I subtracted the 180-degree master flat from the 0-degree master flat and vice versa. Here are the results.

FSQ106
image.png

TOA130
image.png

CDK12
image.png

So from now on, I'll be taking flats for each unique rotation angle. I'm not at all thrilled about it, but it has to be done. Fortunately, I can automate that in NINA with some generic code in my main template. So that's not a hassle at all. But managing an additional 210 404MB files (84GB) for each rotation angle will be no fun. I'll have to work out a system for that.

Sorry Timothy - are those flat panel flats?
Like
AccidentalAstronomers 18.64
Topic starter
...
· 
·  Share link
Kevin Morefield:
Sorry Timothy - are those flat panel flats?


Yes. All three PrimaLuce Giottos. I was surprised at how bad this issue is--especially on the TOA, which I always thought was giving me pristine results.
Edited ...
Like
AccidentalAstronomers 18.64
Topic starter
...
· 
·  Share link
Kevin Morefield:
Sorry Timothy - are those flat panel flats?


And no need to be sorry. I'm the one who should apologize for spreading disinformation on here. This is the sixth or seventh time you've helped me improve the quality of my results, so I really appreciate that!
Like
morefield 12.31
...
· 
·  2 likes
·  Share link
Timothy Martin:
Kevin Morefield:
Sorry Timothy - are those flat panel flats?


And no need to be sorry. I'm the one who should apologize for spreading disinformation on here. This is the sixth or seventh time you've helped me improve the quality of my results, so I really appreciate that!

Glad to be of any help at all!  Did you check the magnitude of the differences?  Maybe take the brightest spot you see on these difference and divide by (65535/2) to get the magnitude of the error that might occur.
Like
whwang 15.16
...
· 
·  1 like
·  Share link
Timothy Martin:
I used LNC in the full WBPP stacks. Maybe he's off base, or maybe he's been overtaken by events, but John Murphy discusses why he thinks NSG is ostensibly superior to LNC. I found that to be the case this time around, but still not close to the result I got by splitting things up.

Then I have to say something in the PI stacking process is not optimized.  It can be something very deep down so end users can't easily touch.
Like
 
Register or login to create to post a reply.