flats overcorrecting in siril? Siril Team Siril · klm · ... · 27 · 727 · 20

klm 0.00
...
· 
·  Share link
=MsoNormal

I shoot with a Canon DSLR/Tamron zoom, and collect lights, darks, flats, and flat darks each session.  I shoot from a somewhat light-polluted suburban location (Bortle 7?).  Here is an example sub of CrB using the Histogram autostretch view in Siril (with and without false color) before any processing:=MsoNormal

light sub=MsoNormal

light sub (false color)=MsoNormal

 =MsoNormal

As a novice, I still use Siril preloaded OSC_Preprocessing.  My stacked results always have the same defect, a substantial circular gradient from center (dark) and radially outward (full white).=MsoNormal

calibrated stack=MsoNormal

calibrated stack (false color)=MsoNormal

 =MsoNormal

I have always assumed it was a complex mix of light pollution, amp glow, my cheapie equipment, etc.  I can substantially flatten the image using background extraction (BE), but to get the corners and center close, BE introduces many rolling hills and valleys which end up as multiple orange and blue patches in the background.  Extensive playing with smoothing, polynomial v. RBF, sample placement, subtraction v. division, and drizzle does not make much of an improvement.=MsoNormal

 =MsoNormal

I am wonder whether the real culprit is an incorrect calibration with the flats, causing inverse vignetting?  I can see the vignetting in the flats, but it is minimal compared to the what shows up in the stacked image.=MsoNormal

flat sub=MsoNormal

flat sub (false color)=MsoNormal

 =MsoNormal

Has anyone else encountered this problem?=MsoNormal

Is there a solution, such as adjusting Siril preprocessing values to decrease how strongly the flats are divided during calibration, etc.?=MsoNormal

 =MsoNormal

Many thanks for any help or advice.
Like
sgthebert 2.81
...
· 
·  1 like
·  Share link
Have you checked if you had any light leak?
Like
AlexFilothodoros 0.00
...
· 
·  Share link
Did you use the same ISO setting?
Like
lock042 16.66
...
· 
·  Share link
Hello, you should take a look here: https://siril.readthedocs.io/en/stable/preprocessing/calibration.html#understanding-how-the-flats-correct-the-lights
Like
tikevin83 0.90
...
· 
·  1 like
·  Share link
I would suspect an issue with the way the flats are being taken, I had a similar issue until I tried doing sky flats and it started working much better.
Like
macnenia 5.87
...
· 
·  Share link
Be careful that you don't chose an ADU for your Flats which is too high when using a DSLR. You must ensure that you are on the linear part of the response curve. Find out what the saturation ADU is and use a value which is half that. I once used values for my DSLR which were too high because I thought the saturation ADU was 65,536 (16-bit), but it was much lower and they Flats were too bright and on the non-linear part of the sensor response curve which was different for red, green and blue and so I had the same sort of issue you show here. 
Niall
Like
andreatax 9.89
...
· 
·  1 like
·  Share link
Niall MacNeill:
Be careful that you don't chose an ADU for your Flats which is too high when using a DSLR. You must ensure that you are on the linear part of the response curve. Find out what the saturation ADU is and use a value which is half that. I once used values for my DSLR which were too high because I thought the saturation ADU was 65,536 (16-bit), but it was much lower and they Flats were too bright and on the non-linear part of the sensor response curve which was different for red, green and blue and so I had the same sort of issue you show here. 
Niall

*It's rather unclear whether you mean the gain factor (e-/counts) or you mean the dynamic range of an image. I guess it's the latter but you never know...
Like
igorrcavalcante 0.00
...
· 
·  Share link
I have a sv 405cc and was suffering from this sort of over effect in flats.

My first workaround that worked was to stretch the master flat and apply it in the calibration phase. It's a workaround.

I started to work on exposure times/adu and a reliable light source. My flats improved to a point that, with normal calibration, was enough for me.

Remembering that for a correct flat, you will need to take bias frames or a synthetic bias or take dark flats.

My sensor is not very good for long exposure times, so I am using flats about 6 seconds and dark flats. The logic is:

1 subtract the dark frames from the light ones
2 subtract the dark flat frames or biases frames from the flat ones

divide 1/2

I spent months trying to have decent flats, it was very frustrating, but it worked well in the end 

Regards
Like
Gondola 8.11
...
· 
·  1 like
·  Share link
1. Make sure you don't have any light leaks. Cap you aperture and take a dark frame, it should be absolutely black. If you see anything you need to track that down and fix it first.

2. make sure you are taking your flat frames with exactly the same setup that you used for your lights, same exposure, same ISO, same everything.

3. I wouldn't shoot darks with this setup because they will never be accurate with an uncooled camera. Dither enough and you won't need them.

4. I don't know how you are taking your flats but try simple sky flats as that's the most reliable.

There's nothing wrong with how Siril calibrates your data when using a script. If all of the raw frames are as they should be, it just works.
Like
Maqce 0.00
...
· 
·  Share link
Tony Gondola:
1. Make sure you don't have any light leaks. Cap you aperture and take a dark frame, it should be absolutely black. If you see anything you need to track that down and fix it first.

2. make sure you are taking your flat frames with exactly the same setup that you used for your lights, same exposure, same ISO, same everything.

3. I wouldn't shoot darks with this setup because they will never be accurate with an uncooled camera. Dither enough and you won't need them.

4. I don't know how you are taking your flats but try simple sky flats as that's the most reliable.

There's nothing wrong with how Siril calibrates your data when using a script. If all of the raw frames are as they should be, it just works.

You can't take flats wit the same exposure as the lights right? Also I had the same issues with my flats and sky flats worked for me. However how do you manage to have no star in your FOV when taking sky flat? Even when the day light is coming I still have some stars that ruin my flats.
Like
andreatax 9.89
...
· 
·  1 like
·  Share link
Max Gillet:
ou can't take flats wit the same exposure as the lights right? Also I had the same issues with my flats and sky flats worked for me. However how do you manage to have no star in your FOV when taking sky flat? Even when the day light is coming I still have some stars that ruin my flats.


*Then you do it at the wrong time, simple as that. I have been doing for over 20 years and basically never had a star to speak of.
Like
Gondola 8.11
...
· 
·  Share link
Max Gillet:
Tony Gondola:
1. Make sure you don't have any light leaks. Cap you aperture and take a dark frame, it should be absolutely black. If you see anything you need to track that down and fix it first.

2. make sure you are taking your flat frames with exactly the same setup that you used for your lights, same exposure, same ISO, same everything.

3. I wouldn't shoot darks with this setup because they will never be accurate with an uncooled camera. Dither enough and you won't need them.

4. I don't know how you are taking your flats but try simple sky flats as that's the most reliable.

There's nothing wrong with how Siril calibrates your data when using a script. If all of the raw frames are as they should be, it just works.

You can't take flats wit the same exposure as the lights right? Also I had the same issues with my flats and sky flats worked for me. However how do you manage to have no star in your FOV when taking sky flat? Even when the day light is coming I still have some stars that ruin my flats.

Sorry, I mis-typed there, for some reason I had darks in my head. Yes, you are right, the exposure for flats should be based on the histogram. What I meant to say is that the setup should be the same, no physical changes to the light path.
Like
Gondola 8.11
...
· 
·  Share link
Max Gillet:
Tony Gondola:
1. Make sure you don't have any light leaks. Cap you aperture and take a dark frame, it should be absolutely black. If you see anything you need to track that down and fix it first.

2. make sure you are taking your flat frames with exactly the same setup that you used for your lights, same exposure, same ISO, same everything.

3. I wouldn't shoot darks with this setup because they will never be accurate with an uncooled camera. Dither enough and you won't need them.

4. I don't know how you are taking your flats but try simple sky flats as that's the most reliable.

There's nothing wrong with how Siril calibrates your data when using a script. If all of the raw frames are as they should be, it just works.

You can't take flats wit the same exposure as the lights right? Also I had the same issues with my flats and sky flats worked for me. However how do you manage to have no star in your FOV when taking sky flat? Even when the day light is coming I still have some stars that ruin my flats.

Sorry, I mis-typed there, for some reason I had darks in my head. Yes, you are right, the exposure for flats should be based on the histogram. What I meant to say is that the setup should be the same, no physical changes to the light path.
andrea tasselli:
Max Gillet:
ou can't take flats wit the same exposure as the lights right? Also I had the same issues with my flats and sky flats worked for me. However how do you manage to have no star in your FOV when taking sky flat? Even when the day light is coming I still have some stars that ruin my flats.


*Then you do it at the wrong time, simple as that. I have been doing for over 20 years and basically never had a star to speak of.

Exactly, you need to do your flats before or after the point where it's dark enough to see stars in the flat image. I sometimes do them in full daylight with no issues.
Like
Maqce 0.00
...
· 
·  Share link
@andrea tasselli@Tony Gondola oh ok I see, then the problem is that my setup doesn't allow me to take flats with a very bright sky because even at the lowest ISO and shutter speed my flats are overexposed...
Like
Gondola 8.11
...
· 
·  Share link
Max Gillet:
@andrea tasselli@Tony Gondola oh ok I see, then the problem is that my setup doesn't allow me to take flats with a very bright sky because even at the lowest ISO and shutter speed my flats are overexposed...

Ok, heres what I do. After imaging all night while I sleep I get up when dawn breaks, check to make sure that the imaging run completed properly and the scope is parked. Since everything is still connected I'll fire up a short sequence just to get the 40 flats I need. Then I start making test exposures, in my case it could be anything from 0.001 sec. to 0.5 sec. depending on the setup. Once I have a target exposure level, I put that into the sequencer. I keep making test exposures until I see the histogram hitting around 40 percent. Then I fire off the sequence and shut everything down. It sounds like a lot but it's really quick and simple, I can do it in my PJ's while I'm still half asleep. In fact, I usually just go back to bed when I'm done! That's for OSC,  if I was shooting mono it would be a little more complicated but not much, the hard work would be in the sequence.
Like
klm 0.00
Topic starter
...
· 
·  Share link
=MsoNormal

Thank you everyone for the great comments and questions.  I should also point out that I am still using Siril 1.2.6.  All the images included in my below answers to your questions are screen captures of Histogram view in Siril with or without false color, unless indicated, to maximize the defects.=MsoNormal

=MsoNormal

 =MsoNormal

Médéric Hébert & Tony Gondola:  Regarding light leaks, I leave my viewfinder covered during all my shoots.  With my setup, I don’t have any way to dither, so I shoot darks with same settings immediately after lights.  Here is a dark sub (linear, Histogram view, and Histogram view with false color:  =MsoNormal

dark sub (linear)=MsoNormal

dark sub (Histogram view)=MsoNormal

dark sub (Histogram view with false color)=MsoNormal

 I don’t see anything that looks like a light leak, but if there is a better test, please let me know. =MsoNormal

 =MsoNormal

AlexFilothodoros:  Yes, I use the same ISO, also F11, for all lights, darks, flats, and flatdarks,.  According to Canon specs, ISO invariance at 1600; however, the same specs show a substantial drop in dynamic range from 800 to 1600, so I shoot at 800 where the ISO is nearly invariant.  That decision may spark a parallel discussion?=MsoNormal

 =MsoNormal

Cyril Richard:  Thanks for the link to flat correction explanation.  I had read that section of the manual, and in fact, it was those ADU graphs that made me wonder if my concentric circle “gradient” was really inverse vignetting.  In any case, I did not see anything that would allow me to adjust how the flats are used in OSC_Preprocessing.  Did I miss something?=MsoNormal

 =MsoNormal

Travis McGeehan:  I take my flats using an ipad set on very low brightness to avoid refresh rate issues.  Using the AV setting and same F11 and ISO as all my other shots, resulting flats typically are ½” exposure.  I then use M (manual) setting with that exposure and same F11 and ISO to make darkflats.  I have previously tried the T-shirt and overlapping paper (and T-shirt+paper) methods, but even at infinity focus, there is a visible pattern (T-shirt weave, paper inhomogeneity) in my flats.  The ipad appears to give a nice homogeneous image.=MsoNormal

 I may be misunderstanding what you meant by sky flats.=MsoNormal

 =MsoNormal

Niall MacNeill:  My flats have a histogram peak usually around 30-40%:=MsoNormal

 =MsoNormal

=MsoNormal

 =MsoNormal

I don’t know where to find the actual ADU value for the flats or what is the saturation value.  The stat [main] command for a sample flat gives:  =MsoNormal

        B&W layer: Mean: 2805.638588, Median: 2795.0, Sigma: 250.700353, Min: 2223.0, Max: 15303.0, bgnoise: 316.180612=MsoNormal

In any case, I also tried using flats (and corresponding flatdarks) with 2x exposure (e.g. 1”) and half-exposure (e.g. ¼”) at same F11 and ISO, but the Siril stack was virtually the same, leading me to assume that Siril normalizes the flats before calibrating the lights?  =MsoNormal

However, I need to do this again to make sure that I did not change anything else.  Assuming that I reproduce my prior results, is there a way to manually input a weighing factor to decrease the flat correction?

Thanks again!!=MsoNormal
Like
sgthebert 2.81
...
· 
·  3 likes
·  Share link
Regarding light leaks, I leave my viewfinder covered during all my shoots.  With my setup, I don’t have any way to dither, so I shoot darks with same settings immediately after lights.


Best way to look for light leak is to take darks during the day in sunlight.

Those don't look like they have leaks. But then again, you need light to leak in to see the leaks

Since you are using a DSLR, one possible way for light leak is via the viewfinder.
Edited ...
Like
jsastro 0.00
...
· 
·  Share link
When I take flats, I automatically use Nina. At the end of the sequence (end of the night), I wait for sunrise and then slew to the zenith. Nina then starts the sky flats with the optimal exposure time — those work best for me. Sometimes I also set the timer to just before sunrise so that the exposures aren’t too short. I’ve noticed that if the exposure is under one second, the flats sometimes don’t turn out well.



By the time I get up, the flats are usually already done without me having to do anything.
Like
Gondola 8.11
...
· 
·  Share link
Médéric Hébert:
Regarding light leaks, I leave my viewfinder covered during all my shoots.  With my setup, I don’t have any way to dither, so I shoot darks with same settings immediately after lights.


Best way to look for light leak is to take darks during the day in sunlight.

Those don't look like they have leaks. But then again, you need light to leak in to see the leaks

Since you are using a DSLR, one possible way for light leak is via the viewfinder.

+1
Like
carastro 8.21
...
· 
·  Share link
Max Gillet:

You can't take flats wit the same exposure as the lights right? Also I had the same issues with my flats and sky flats worked for me. However how do you manage to have no star in your FOV when taking sky flat? Even when the day light is coming I still have some stars that ruin my flats.

Cover the aperture with some white paper or white fabric.  That way you are only capturing the light, not the stars.

Your histogram peak looks OK, personally I would have it slightly lower, which might mean putting extra paper over the aperture.  the trouble though with doing flats before sunset or at Sunrise is that the sky brightness is changing all the time, and unless you are quick, the value will be different between the first and last.

I do my flats during the day time and use more paper.  I also have to cover up my one filterwheel as it is a manual one or I get light leak.  If you have to disassemble you might not have the option of doing them in the day time.
Edited ...
Like
Gondola 8.11
...
· 
·  1 like
·  Share link
Carastro:
Max Gillet:

You can't take flats wit the same exposure as the lights right? Also I had the same issues with my flats and sky flats worked for me. However how do you manage to have no star in your FOV when taking sky flat? Even when the day light is coming I still have some stars that ruin my flats.

Cover the aperture with some white paper or white fabric.  That way you are only capturing the light, not the stars.

You expose flats according to the histogram, not what you used for your lights. Usually you want the peak to be between 33% and 50%. On getting stars, you need to wait until the sky is bright enough so that stars are not visible. I think we have mentioned that to you before.
Like
carastro 8.21
...
· 
·  1 like
·  Share link
Tony Gondola:
You expose flats according to the histogram, not what you used for your lights.


Absolutely.  I was always told with a DSLR to use 1/3 to just under half way across the histogram.  With my mono camera I use around 23,000 ADU which is just over 1/3 of full well of 65,000, I never expose more than that and it works just fine.
Edited ...
Like
Maqce 0.00
...
· 
·  Share link
Carastro:
Max Gillet:

You can't take flats wit the same exposure as the lights right? Also I had the same issues with my flats and sky flats worked for me. However how do you manage to have no star in your FOV when taking sky flat? Even when the day light is coming I still have some stars that ruin my flats.

Cover the aperture with some white paper or white fabric.  That way you are only capturing the light, not the stars.

I already tried to put a piece of white fabric in front of my tube, but everytime it gave me bad results. It seems that the light is not coming into the tube the exact same way with the piece of fabric.
Like
carastro 8.21
...
· 
·  Share link
It will reduce the light slightly, so you need to adjust the exposure.   This is the way to do it, so you will have to experiment.  Maybe typing paper would work better, I use a cardboard type of lens cap, with paper across the aperture.  

The only other alternative is to use a light screen and dim the light right down as much as possible, but this is in my experience much more apt to cause problems, a) Keeping the screen in place while you do the flats, getting the right brightness, findling a screen with even and unflickering light.   I used a lit tracing board for a while that you could dim the light on,  but eventually it became faulty and I found it gave me varying light throughout a session of taking flats, so I stopped using it.
Like
klm 0.00
Topic starter
...
· 
·  Share link
Hi all,
It's been a few days, but I needed some time (and favorable conditions) to test the many helpful suggestions that I received.

The culprit for me is the ipad.  I have tried multiple iterations and combinations (low/high brightness setting, long/short exposures, glass to glass v distance between ipad and lens), and the result is qualitatively the same; the final corrected/calibrated stack always includes an overwhelming concentric reverse vignette effect
image.pngstack with ipad flats (see discussion above)
As I mentioned, the unstretched flats look normal, but either they are not, or siril does not like them, or my camera is doing something strange that is not readily apparent in the images themselves.  the intensity of the concentric pattern is lessened when the ipad is farther away:
image.pngipad flat 2 feet (max distance) from camera
but still strong, and very difficult to remove using background extraction.

So, as suggested by many of you, I switched to sky flats. I tried paper (1-8 sheets with random orientations), t-shirt, and a t-shirt/paper sandwich.  All the flats have a visible pattern showing the inhomogeneity of the paper, t-shirt weave pattern, etc.  Longer or shorter exposures did not make a significant difference:

image.pngeight sheets paper, midday sun
image.png8 sheets paper, bright 5000K bulb

the pattern gets exaggerated in the stacking process, and also a strong circular diffraction pattern shows up:

image.pngstacked image using 8 sheet paper flats (histogram view/false color)

The solution for me was simply to take images directly of the dusk sky before any stars appeared (no paper, no t-shirt), and adjust exposure time to get histogram in decent range (25%). 4 second exposure (f11, ISO1600) worked well, but I had to be quick, because as someone pointed out above, the lighting is changing rapidly even during the 2 minutes taking the flats. The flats are very blue, but that color cast did not appear to affect my stacks
image.pngflat sky dusk 4 sec f11 ISO1600

image.pngfinal stack without flats (histogram view/false color)
image.pngfinal stack with flats (histogram view/false color)
suffice to say, background extraction was much easier. 

After some simple processing here is the result
image.pnghead of Draco
it's not great because I only have 8 lights.  but the flats are working better, and I am already seeing some resolution of nu Dra.

Many thanks to ALL!
Like
 
Register or login to create to post a reply.