What is real and what is not? How to know what to include... [Deep Sky] Processing techniques · Christian Großmann · ... · 12 · 771 · 6

Hi,

as I always stated, image processing isn't my thing. But while forcing myself to progress in this field, I ran into some issues no tutorial seems to care about. Let's explain the problem.

For my latest images I took the time to collect at least 15 hours of data. This is a general rule I created for myself. So I collected a lot of data on less images and I am always excited to see what the master frames look like. Who won't? When I open the masters in Pixinsight, they show a lot of things going on especially in the background. As many tutorials about DBE or similar tools show, there are always gradients of different kinds visible. That's what these tools are designed for and they handle removing them really well. But I wonder, if all the stuff that I try to remove is really trash. There are two main examples:

1)
I did image Markarjans chain the other day and the masters showed a quite foggy background. You might see it in the image I uploaded. The structure seems to be weird for a simple gradient. So I wonder, if this may be real. The color is quite neutral and so it must be on every color channel and the Luminance as well. (I can't look at the masters at the moment) It looks like it belongs there. But how do I know? Are these things real and should be left in the image? If they are real, they should appear with longer exposure times, shouldn't they? We do take a lot of subs to get those details out of the noise, right?



2)
The same exact thing happened when I imaged M81 as well. You can't see it as detailed as on the Markarjans chain image, but it is there. And it is even better visible in the master frames. The background of the uploaded image was darkened a bit too much, but it hides those things. I wonder, if it has to hide those things.



So there are some questions I need some answers or suggestions to:

Are those things real and if not, what causes such strange things? Is there a way to avoid them? There are other subjects I imaged where the background is quite even and has no issues at all.

If the details are real, why then so many people seems to work so hard to remove them? I understand the "beauty" of clean images as art. But from a scientific point of view it might be a bad idea to remove this stuff.

How do YOU decide whether such things are part of the sky angle you image? At least I saw some images of M81 on the web, that showed such foggy regions around the galaxy (distances not taken into account). So I am not the only one who experienced these things.

So what is real (and in my opinion should be included in the final image) and what is not real (and should be thrown out)?

Thank you for sharing your thoughts.

CS
Christian

There's a few different parts to your question, so hopefully I don't jump around too much. 

First off, figuring out what's real and what's not: Personally, if I see stuff in my image that I'm not sure is real, I go right here to Astrobin and looks for images of whatever object I'm working on, and find multiple that seem to have gone really deep on bringing out faint stuff (could scroll through everything, though I usually filter to IOTDs and Top Picks). I'll take a look at the area I'm confused about and see if I can find that same shape/structure in other deep images, and if I find it, I look at another, and another, and another. If I'm finding the same structure in multiple images, I know what is in my image is real. If I've been through 15 other really deep images, and see a lot of faint stuff, but not the "object" I'm confused about, I assume it's not real. 

Second: "if the details are real, why then do many people work so hard to remove them?" Any number of reasons. They didn't know it was real. They didn't have enough integration time or dark enough skies to get the faint detail (so in this case, they didn't remove it, they just never got it in the first place). Or in some cases, it's there and they know it's real, but there's just not enough detail to actually bring it out in a clean way, so they smash it into the background in favor of a cleaner picture. That was the case with an M31 I did. I got enough Ha data that the really faint surrounding Ha was actually in the data - if you overblew the saturation, it showed up and was in the right location and right shape, it was definitely there. But it was just so faint that any attempt to bring it out just made it look like a poorly managed background, so I had to just give it up and go for the cleaner background instead. Had I gotten enough integration time that I could bring it out, I absolutely would have, I just didn't have the integration time needed for stuff that faint. 

As for your images: On Markarian's Chain, I couldn't tell you for sure - I don't know that area very well. I'm not aware of any IFN or other faint background stuff sitting around in that area, so my guess would be noise, but someone else may know better than me. Don't take my word on that one. For M81, there is absolutely IFN around it, and quite a bit of it. You'll find plenty of images that have it if you want to compare yours to see if the dusty spots line up (I don't see any in your attached image, but you mention that it shows up better in the masters, so it may still be there somewhere). People love the IFN around M81/M82 so if you search it up here on Astrobin and you'll find tons of images with it (my image of it on my profile has it if you want an easy one to find).

Essentially, a very long-winded (as I tend to be) way of saying that it just comes down to researching your target. Asking community members if stuff is real is also viable, but I tend to just go find it on my own so I don't have to wait on the answer. 

​

Not sure how much IFN there is around the Virgo cluster, but M81 is very busy with it. It is rather faint though, so easily mistaken as a spotty background or odd gradient due to some data issue or light leak. In my previous attempts at the target i saw a weak something in the background at around 4 hours from Bortle 4, and it only became obvious as IFN at more than 10. In order to make an appreciable attempt at processing it out i ended up going for more than 30 hours.


That 30+h attempt can be seen here : https://www.astrobin.com/llbasd/B/

Oskari Nikkinen:
Not sure how much IFN there is around the Virgo cluster, but M81 is very busy with it. It is rather faint though, so easily mistaken as a spotty background or odd gradient due to some data issue or light leak. In my previous attempts at the target i saw a weak something in the background at around 4 hours from Bortle 4, and it only became obvious as IFN at more than 10. In order to make an appreciable attempt at processing it out i ended up going for more than 30 hours.


That 30+h attempt can be seen here : https://www.astrobin.com/llbasd/B/

There is a TON of IFN here, where most people get lost in processing this area is that their gradient removal process nukes the IFN and causes that orange peel effect.

What i suggest when seeing this orange peel effect is to break out the luminance, try and stretch and optimize and denoise that and then pixel math it back into the RGB master so the orange peel is lost to signal and then mess with curves to try and highlight it with masks as necessary.

or get more data ;) 

I haven't imaged this recently but my first attempt a few years ago i had so much IFN some people called BS, but it's there - and lots of it.

You need to take care of dithering to eliminate walking noise and ensure proper flat calibration before you should worry about LP.  These are absolutely fundamental, otherwise it makes no difference if you do 5, 15 or 50 hours of exposure - it will all be for naught.

Christian Großmann:

Rafał Szwejkowski:
You need to take care of dithering to eliminate walking noise and ensure proper flat calibration before you should worry about LP.  These are absolutely fundamental, otherwise it makes no difference if you do 5, 15 or 50 hours of exposure - it will all be for naught.

But are these effects walking noise? I do dither and I use an adjustable flat panel for my flats. I try to keep the exposure times of the flats long enough to even out any possible flickering that may come from the pwm signal that controls it. I assume, my workflow is not differnt from those of others, is it?

No, doesn't look like walking noise or mis calibration, it looks to me like gradient removal took away some signal and like i said earlier, most people just go and nuke the rest of it and ignore the IFN.  Denoise will cause it to get splotchy like this in a lot of cases. What was your basic workflow for editing?

Christian Großmann:
At first, I'd like to thank you all for taking the time for your very detailed answers.

@Jon Rista
I try a summarized answer to your whole text ;-) I totally get what you are trying to say. Especially in the image of Markarjans chain, there are really small structures in the background, which I never would have thought of as light pollution. But obviously, it is what it is. So that's one other thing to keep in mind in the future. Your explanations of LP is very logical and I will see, what I can do about it. Maybe I start with ensuring, that there are no light leaks in the light path of the equipment. I use ZWO filter wheels and I did cover some unused screw holes with tape. But maybe that's not enough. But this is a problem I can handle.

Sadly, what I can't deal with is my imaging location. I am situated in a really small town and I am forced to use my backyard. It might be possible to go to darker skies. Lightpollutionmap shows even Bortle 2 on some places. My hometown lies in a bortle 3 area and I know, my backyard is of course even worse. But I am lucky enough to have darker skies than so many other people here. So I am glad I live here and make the best out of my situation. But the main reason for staying in my backyard is time. I am very busy with some other communities as well and sometimes I am at the edge of what I can stand. But I am responsible for the others and trying to reduce my work with them usually means the end of these groups. So astro photograpjy is a great solution, because I was able to automate all my stuff and keep 3 scopes set up in my backyard permanently. This way, I can image during the night and still get some sleep. Going out in the fields is my preferred solution, but it means to spend a lot more time which I simply don't have.

Your answers here are not what I have hoped for. But it sadly is what I expected. So the first step is to see, if fixing possible light leaks might improve my data a little bit. I was thinking about blackening my SCT tube with a darker color as well. I'm still not sure if I will. All of the other stuff I have to ive with.

But the result of this conversation is to realize what my limits are. I had to know what is possible and what's not. Accepting these limits might make life much easier and I will not try to catch the impossible.

I wanted to touch on "Bortle" vs. the Light Pollution map I was recommending. Bortle scale is NOT about imaging. IT is very explicitly about what is necessary to VISUALLY observe certain objects of certain brightness levels or magnitudes. The Bortle scale is too heavily used by astrophotographers for finding a "sufficiently dark site" and sadly, it leads many, far too many, astrophotographers to thinking that they would have to drive 100+ miles to find sufficiently dark skies for imaging. Which is actually false a lot more often than its true! 

If you use the VIIRS map I recommended, that is NOT based on the Bortle scale. it is a different scale, based on actual illuminance measured at those locations from space. The VIIRS map will generally help you find a sufficiently dark site much closer to home a lot of the time. Based on what you've said about your location, I suspect (based on the LP I can see in your images) that you might be borderline as far as sky brightness goes...and you may well not have to go very far to find good skies. If you live in a small town, the LP IN town can jump by a magnitude or so, but if the town is relatively isolated, it may well be all you would have to do is get out of town by a few minutes. 

Anyway, just wanted to clear that up. The Bortle scale is actually a poor way to judge sufficient sky darkness for IMAGING purposes, and it is really not necessary to find a truly dark Bortle site to have nearly perfect skies for imaging. There are times where the VIIRS map will show ideal skies (no color in the VIIRS map) when Bortle might show yellow or even brighter. Now, if Bortle is say orange, and VIIRS shows no color, then directly overhead is good. Depending on where the light pollution bubbles are, it might limit what directions you could image in without picking up LP. If Bortle is dark yellow or green, and VIIRS is no color, then its probably a pretty darn good imaging site. If Bortle is blue, gray, black, and VIIRS is no color, its a superb imaging site (which should be a given, since to find Bortle blue, you usually have to drive pretty far from any artificial light pollution centers.)

So with the VIIRS map, most imagers can usually find a viable dark IMAGING site, a LOT closer than a viable dark Bortle (VISUAL) site. 

As for the answer you wanted...sadly, this is why I way LP is the worst thing astrophotographers deal with. It is some truly nasty stuff. Its not just a gradient, its a signal, and it gets blended with other signals, and because all signals have noise, and in fact LP can sometimes have structure (depends on exactly what's in your skies and where, and where that stuff falls in your field), so totally, ideally, and cleanly removing ALL LP is largely a pipe dream. You can remove the larger stale stuff. The stuff that blends into fainter signals like IFN, well, its going to be very difficult to impossible to eliminate those pollutants, and that's what makes it hard to figure out if the structure you are seeing is real or not. Even if you have a reference signal, its tough to know how clean the skies were when that image was acquired, and even if you think you know what the real structure is, there still aren't really any good ways to separate intertwined pollutant and object signals from each other. Noise will hamper that, and noise will usually occur at multiple scales hampering it at multiple levels. 

If you have to image with LP, then yes, its good to know what your limits are, and how far you can push your signals around. LP adds more noise (and it will usually be noise at multiple scales, such as the colored orange peel junk). If you are dedicated enough, then you can put many tens of hours into each object, and that will help you get beyond the limitations imposed by LP to a degree. If you can get 30, 40 hours or more per object, then you can still make some nice images...but, they will lack the depth of a dark site image 1/10th, 1/15th, maybe even 1/20th the integration time. You will have a hard time getting that truly expansive contrast, which IMO is really what sets a dark site image apart from others. Dark sites won't necessarily mean you can pick up very faint signals in "little" time, sometimes you may still need dozens of hours per image, but in contrast, you would need 10-20x as many hours from a light polluted site, and even with that, you can never truly eliminate pollutant signal from everything in your images, and that will show in the end. 



This is Bright Red Zone Bortle (18.2mag/sq") vs. Dark Green Zone Bortle (21.3mag/sq"), a difference of 3.1 stellar magnitudes. A difference of over 17x the sky brightness. These are identical exposures. Adjusting the stretch from both to be identical:



You can se the difference in noise pretty easily. I also worked to try and extract the pollutant signals...and you can also see how well that worked. I could work at that, removing pollutant signals, till the end of time, and it would probably still never be as good as the dark site image. Especially the center bottom area, you can see where the LP intermixes with notable structures around one of the Pleiades. Pretty much everything I did to try and extract that pollutant resulted in the extraction of the nebular signal as well. You can also se areas where I DID remove some of the fainter nebula, between some of the groups of stars, in the polluted image. I could get 17x as much signal from the light polluted site...and that might help me with the extractions. But I've done that, it takes imaging across many, many nights...and the nature of the LP shifts across all those nights, which complicates the nature of the gradients and any remnant pollutant signals, and I've found that sometimes too much polluted signal can still be just as problematic. (In fact, one of my earliest "tens of hours" images WAS the pleiades, from my 18mag/sq" backyard skies, and even after 20 something hours (and eventually around 40 when I tried to get more data), I did get some fainter details and some better contrast...but, you could still tell the image was quite light polluted, and the problem of extracting the pollutants shifted from the (actually relatively bright) inter-Pleiades reflection nebula here, to the fainter outer dust and background structures (there is actually some faint Ha in this region, not to mention a plethora of very dark dusty nebula), and I was never able to properly separate those signals from the pollutant signals in the long run, even with tens of hours. I had a decent image in the end, but, it was NOTHING compared to 2-3 hours of pleiades from my dark site.

Such are the consequences of imaging under light polluted skies. Its not that you can't image with polluted skies. As you said, know the limitations. I think once you know that, then you can focus on making the workable parts of the image look really nice, and then...maybe just let the chips fall as they may on the parts of the image that are not as workable.