"L" flats problem.. Pleiades Astrophoto PixInsight · AstroRBA · ... · 114 · 1980 · 24

I just read through this quickly so apologies if I missed something obvious but why is a master bias being used when we have a master flat and master dark already?

OK... the world does make sense, floating point math still makes sense.

Somehow, in image calibration using WBPP, I suspect the bias is being subtracted multiple times causing therefore, a division error. Here, I simply subtracted the bias from the flat, DID NOT SUBTRACT THIS BIAS FROM THE 300s DARK, and simply calibrated the lights using the raw 300 second dark, and the flat from which the bias was subtracted. Perfectly calibrated frames, no linear fit needed, no need to worry about sensor non linearity.  In thinking about Andrea's work, which motivated my effort here, what I realized is that his Linear Fit process is in effect either adding or subtracting a bias:




It think it really helps to understand how calibration works, rather than treating WBPP as a black box. For the longest time, I avoided WBPP for just this reason. Even now, there are parts of calibration I will do manually before feeding into WBPP. 

Cal_Light= (Light - Master Dark)/Calibrated Flat

Where Master Dark does not have bias subtracted from it, calibrated flat has either bias or flat dark subtracted.

There are other forms of this equation, but understanding the fundamental math and science behind calibration avoids a lot of grief.

And I wonder whether all the other similar issues seen in the recent days might not be related to the use of WBPP…

The Optec Secondary mirror focuser should eliminate this I hope?  (whereas I'll then lock the mirror with the mirror locks - although I've heard some say that the Celestron mirror locks are not perfect?)

The Optec should really be the *preferred* way to focus a C14, aside from your current issues with flats. And, yes, I have heard they aren't perfect. Still, way better than NOT having any. I friend of mine modded his C14 with 3 spring-loaded locking prongs but maybe this is a bit stretching the envelope...

Arun H:
OK... the world does make sense, floating point math still makes sense.

Somehow, in image calibration using WBPP, I suspect the bias is being subtracted multiple times causing therefore, a division error. Here, I simply subtracted the bias from the flat, DID NOT SUBTRACT THIS BIAS FROM THE 300s DARK, and simply calibrated the lights using the raw 300 second dark, and the flat from which the bias was subtracted. Perfectly calibrated frames, no linear fit needed, no need to worry about sensor non linearity.  In thinking about Andrea's work, which motivated my effort here, what I realized is that his Linear Fit process is in effect either adding or subtracting a bias:




It think it really helps to understand how calibration works, rather than treating WBPP as a black box. For the longest time, I avoided WBPP for just this reason. Even now, there are parts of calibration I will do manually before feeding into WBPP. 

Cal_Light= (Light - Master Dark)/Calibrated Flat

Where Master Dark does not have bias subtracted from it, calibrated flat has either bias or flat dark subtracted.

There are other forms of this equation, but understanding the fundamental math and science behind calibration avoids a lot of grief.

Aye, calibration done improperly can lead to all manner of problems. This is where my trust of WBPP, since it is a bulk processing tool, fades. I have come across far too many threads across the web with people having problems with calibration when using WBPP.

In general, I think the following apply:

BIAS: Integrate simple average with large-sigma outlier rejection. No scaling/normalization. No weighting.

DARKS: Integrate simple average with large-sigma outlier rejection. No scaling/normalization. No weighting.

FLATS: Calibrate only with one master...either master bias, or master flat dark (if you have glows, long flat exposures, etc.) No output pedestal.
FLATS: Integrate with simple average, percentile outlier rejection. Multiplicative scaling. No weighting.

LIGHTS: Calibrate with a master dark. Don't optimize!!! Calibrate with master flat. Don't bother with a master bias, as it shouldn't be needed, the master dark will remove both bias and dark, master flat should already be calibrated.
LIGHTS: Integration can be done many ways. Generally you want to average, use additive scaling, weight according to preference. You can reject pixels according to a variety of viable pixel rejection algorithms, its worth experimenting to figure which works best for your data and approach. You can use local normalization (PI changed how this worked, so I'm no longer familiar with how to do it, but in the past it could be quite helpful if your frames had inconsistent background levels, i.e. thin patchy cloudcover or the like.) 

BIAS and DARK should be very simple. FLATS should only be calibrated with either a master bias or master dark. The only time you would ever need to use both a master bias and master dark, and calibrate the dark, is when you intend to do dark scaling (dark optimization in PI.) Its been demonstrated pretty well over the years that dark optimization can be very problematic and can INTRODUCE a lot of problems (one of the reasons I'm not a fan of WBPP, it seems to enable dark optimization by default!! Probably because it tries to use a single master dark to calibrate everything, which is simple, but most likely incorrect and ineffective in most cases.) In fact, whenever you use ImageCalibration, you should make sure you don't have the "Calibrate" checkboxes set, as that will look for a master bias, and subtract it from whatever master you are calibrating. That would generally be incorrect for the master dark unless you are optimizing (which you should generally never do), and should always be incorrect for the master flat, as flats are usually integrate with multiplicative scaling, which would result in funky results if the bias (or dark with flat darks) was not removed first. To calibrate the master flat again, would again subtract the master bias, which would be incorrect.

The impacts of these things can often be subtle...sometimes they can be quite noticeable. It depends, but this is often the root cause of many calibration issues, which seem to arise frequently with WBPP. 

I think you are on the right track now, though. Stick with the manual approach (its not really that hard, only a bit more time consuming), and i think you'll have better results in general as you go forward.

I just read through this quickly so apologies if I missed something obvious but why is a master bias being used when we have a master flat and master dark already?

See my reply above. If it is used, and the Calibrate check boxes are checked, but you are not doing Dark Optimization, then it would be a mistake. I think one of the earlier screenshots of ImageCalibration in this thread did show a master bias, as well as calibrate being checked for both the master dark and flat...and that was likely a problematic configuration. Normally, you would want a full master dark (not bias subtracted, calibrate not checked), and then a properly calibrated and integrated master flat, calibrate not checked, when using ImageCalibration on your lights.

Arun H:

Jon Rista:
If it is used, and the Calibrate check boxes are checked, but you are not doing Dark Optimization

Dark Optimization is, I think, one of the most ill thought out "features" made available in Calibration. They really should get rid of it completely. It is an example of the authors of PixInsight being too clever by half. The idea is to multiply your master dark by some factor such that the global noise is reduced. The problem is that you can reduce global noise, but that does not mean local, non random noise is reduced. It causes bad issues when you have patterns like amp glow or banding.

Totally agree! Its one of my most-hated features.