The Astrobin All Sky Survey: A proposal for a community resource Other · Brian Boyle · ... · 363 · 13510 · 59

This topic contains a poll.
Would you be interested in contributing towards an AB all sky survey?
No. I wouldn't find such a survey useful.
No. Satisfactory data already exists for me elsewhere.
No. I would find such a survey useful, but I don't have the time, location or equipment to contribute.
Yes, I would be interested in taking part. One or two fields maximum.
Yes, I would be interested in taking part. Prepared to do multiple fields.
profbriannz 17.56
Topic starter
...
· 
·  Share link
@Astrogerdt  I am happy to share with you - or anyone else for that matter - the 20-add panes I used for this mosaic.  I can share the linear files (.fit or .xisf  format), each around 250MB.  How is the vest way to do this?  I guess dropbox might be the best way, but I would rather not pay for an account.

Brian
Like
MichaelRing 4.64
...
· 
·  Share link
Al little training data will be nice, if you want you can put the data here:

https://www.mycloud.ch/s/S00B9DF137C56385DEA23517B17090B96F38766499E

Data will be automagically deleted after 30 days, until then whoever wants to play with the data can access it via the link.

Michael
Like
Die_Launische_Diva 11.54
...
· 
·  Share link
I'd like to raise a small warning for the brave project participants: You need to study quality widefield images before modelling gradients in the individual mosaic frames, otherwise you will remove natural sources of gradients, like for example our Milky Way glow.

Here are two great examples from both sides of the galactic plane demonstrating this:
https://astrob.in/ey7qe2/0/
https://astrob.in/318123/0/

Note the Milky Way glow gradient has been carefully treated and not removed.

Unfortunately many images (and especially mosaics) have been "flattened" because their creators have overdone it with DBE and subtracted light which wasn't light pollution.
Edited ...
Like
siovene
...
· 
·  Share link
Brian Boyle:
Sticking to these principles, how about we simply setup a new user account for the survey?  The downside of this is that the password would need to be shared between those posting to it, unless an account could be created with image universal read/write access only [slightly violating principle 1].  Even then, security issues might not allow this to happen. @Salvatore Iovene would this be an option If an account requires a subscription to be paid, I will happily do this.


Unfortunately this wouldn't work (and it if did, I'd be happy to sponsor the account). The reason is, as you guessed, security: AstroBin is going to detect attempted logins from multiple countries and force an email verification, so you'd have to share the email address too.

And if I added an exception for the account, that would be a security weakness, especially with many people knowing the password, the password being emailed around, saved somewhere insecure, etc.

I suggest you do create an account, I sponsor an Ultimate subscription for it, and people email you the images (and by "you" I mean whoever will be coordinating thi).
Like
Astrogerdt 0.90
...
· 
·  Share link
@Die Launische Diva (Hi von Deutschland aus!) raised an important point. 

Currently, I don't really know how to solve this issue. My first idea was to do the gradient reduciton after the combination to a mosaic, but that would create problems with the color calibration, as there would be a color gradient in each image, which would render a background neutralization impossible. 

This leaves us with the option of multiscale gradient reduction I mentioned earlier. Which is by far the most technical complex method I think, but it could work. 

Are there any other good ideas to decide what is signal in one single panel?

@Michael Ring Thanks for providing the cloud space for us! Just to make that clear, how much space can we use? 20 files of 250mb each will take up some space....

@Brian Boyle I think XISF files would be best suited as we generally agreed on using PI. 

CS Gerrit
Like
MichaelRing 4.64
...
· 
·  Share link
This idea of creating one user sounds like a very good, here some reasons why:

It is easy with the Astrobin API to extract all images of an user, the resulting JSON Files include RA/DEC for each published image so it should be easy to in the future create a website that shows the already done/reserved tiles..

For this user it would also make sense to create Collections, perhaps one for each mooncycle so that in the beginning people can easily see where data is needed. One crazy idea would be to use the StarMask process of PI to generate an initial picture of each tile so people can actually check if their data matches the tile's size requirements.

Keeping track of Licensing of the data should also be very straightforward as this is in control of this one user (at some point we will have to open the can of worms that comes with licensing, https://creativecommons.org/about/cclicenses/ ) I guess for a start people should share their indivudual content under CC-BY to allow to apply a more strict license lateron if needed.

This one user would also ideally hold all the data, this way the tiles will stay accessible for everyone as long as the user exists. From what I can tell in first tests the link to xisf data is currently not protected, so when we publish the link to raw data anybody can access it and work on the data.

Sharing data might also get a bit easier for the contributor, once the data is available in that special user account it can be deleted in the account of the original creator.
Especially lite/free users and likely also Premium users will not be able to share directly high quality data because of the 25/50 MB limit for files so their data must anyway be collected in another way (unless data is provided as a full tile with data reduction so that it fits in the 25/50MB limits but I think this is not the plan atm but I may be wrong here, this thread is getting longer and longer....)

The big question is if Salvatore can provide the storage that this user will require in the end or if this is not within the limits of his business case, I guess in the end 100-300GB online storage would be required when we want to keep good quality base material for future use (The small sized final tides will of course only use a fraction of this space), not sure what this costs today but we as the users should in my point of view take this cost, I will happily donate one year ultimate (or more, if required) to make this Survey work.

But now back to something else:

@Astrogerdt : If Gerrit wants to use the storage I provided then there will be enough room for him, I will disable uploads once he is done.

I will now experiment with data sharing, will share a 4 panel mosaic of the Orion region to see if I find limitations or if all works as expected.

Michael
Like
siovene
...
· 
·  1 like
·  Share link
Michael Ring:
The big question is if Salvatore can provide the storage that this user will require in the end or if this is not within the limits of his business case, I guess in the end 100-300GB online storage would be required when we want to keep good quality base material for future use (The small sized final tides will of course only use a fraction of this space), not sure what this costs today but we as the users should in my point of view take this cost, I will happily donate one year ultimate (or more, if required) to make this Survey work.


The 300 GB are going to cost me around $6.90/month just for storage, i.e. not including any download/upload bandwidth, but as I said, I will happily donate it to the cause!
Like
andreatax 9.89
...
· 
·  Share link
In anything other than >B4 skies and with integrations equivalent to 1hr @ B1 skies the LP effects should be minimal if  proper flat fielding is achieved (that is a rather big "if"). Having assumed that the raw integrated lights (let's assume RGB for the moment being) should not be flattened with more than ABE 1st degree (i.e., linear) for mosaicking to work (since the procedure will try to match and feather the gradients in the overlapping zones thus is best to leave that task to it alone). Keep in mind that with pristine skies Gegenschien and Zodiacal Lights effects tend to creep in especially in larger fields as proposed here, although this effects are most conspicuous around the Ecliptic Plane.

As for the general assembly procedure the general procedure presnted in the IRIS website should probably serve as guideline (reported here below verbatim):
>>1 - Search of distortion law of the lens. The step is mandatory for wide-field imagery since one uses optical focal length lower than 135 mm (to fix the ideas). The distortion parameters, estimated once, are a constant of the instrumentation.2 - Correction of the optical distortion in each image which one wants to assemble by using the parameters found at the preceding step.3 - Astrometric reduction of the individual images. This amounts associate celestial coordinates (right ascension and declinaison) with each pixel.4 - Projection of each image in a common celestial map. Map projection is a transformation that convert celestial sphere coordinate on a flat map. Iris proposes up to 7 distinct projections.5 - The fusion of all the images projected in the same cartographic reference frame into an unique image.<<

See: http://www.astrosurf.com/buil/iris/new550/new550_us.htm
Like
Astrogerdt 0.90
...
· 
·  Share link
The problem of zodiacal light and gegenschein impact on gradients in the images mentioned by @andrea tasselli could cause some serious problems I guess. 

Since we have to do color calibration to the individual fields, we need a clean background for background neutralization, and since we should not flatten the background too much to preserve actually wanted gradients and enable the mosaicing process to work, we can't really reduce gradients in the individual fields. The demands of both processes are contradicting. 

Zodiacal light also eliminates my idea to use MultiscaleGradientReduction. 

One other idea I have is somewhat described here in section 3.4: https://iopscience.iop.org/article/10.1086/648480#pasp_121_885_1180s3

It was used for another all sky survey due to similar problems. However, I can't rally find the space based sky background data mentioned in the paper. If someone else here has more success with that, this could ultimately solve our gradient problems. 

CS Gerrit
Like
whwang 15.16
...
· 
·  Share link
I echo @Die Launische Diva ’s comments about Milky Way glow and gradient subtraction. Having mosaicked the 360deg Milky Way myself, I don’t believe there is a perfect way to do this without using external data.  Even with external data, I doubt it would be ever possible to do this perfectly. All you can do is to minimize the removal of (or adding to) the Milky Way glow.

What I did is to take a very wide field picture (much wider that each mosaic panel) near zenith under good conditions, and to use that to model the gradient. This can sort of work because in a very wide field picture, you have sufficiently large areas without the light from your target (e.g., The Milky Way), so you can model the true sky background much better. 

I have used the above technique to create seamless mosaics for more than ten years. It’s also part of the key ideas behind PI’s photometric mosaic script. It works really well for mosaics of several tens of degrees. However, I am very skeptical if it can work perfectly for extremely wide field mosaics that we are discussing here. One of the challenges I found is the difficulty of getting perfect flat field corrections for ultra-wide lenses. The other is the existence of zodiacal light, which is seasonal, and is also 360 deg. I believe one can achieve a ver clean-looking 360-deg mosaic with such a technique, but I don’t believe any one can guarantee that no Milky Way glow has been removed (or added).
Like
MichaelRing 4.64
...
· 
·  Share link
I experimented a little with uploads, here are my results:

I took the Ha part of an orion mosaic consisting of 4 parts, converted the parts to RGB to simulate RGB Data (I have not done RGB panos yet)
The files are from an IMX571 sensor, this sensor has 26mpx.

Then I saved the xisf files uncompressed and then compressed them with zip and also saved them as compressed xisf and also zipped the results.
Saving compressed xisf files was more efficient, but total filersize was 770MB.... a little too much for my taste, for people that only need to upload 2 parts because they have wider field the amount of data will half but still will be a lot.

As we do not really need full sized data for the final pano I also reduced the data by downsampling to 50% width, this created in the end a zip file of 190MB, that looks like a more reasonable filesize.

I also checked how much extra data it would cost to save pre-positioned tiles from MosaicByCoordinates and as expected the extra dark space arround the data does not increase the filesize much, so we could consider asking people to provide pre-positioned files, and by this we could ensure that people actually check that their 4 tiles match up and do not leave holes. This will of corse have a negative impact when data has to be re-edited for gradients, not sure here what actually costs/saves more time for the QA team.

You can have a look at this data here, the download link to the zipped xisf files is in the description. There is a little hack involved, I had to rename the zip file in the end to xisf to be able to upload.

https://astrob.in/d663mt/0/

I also tested if it works to upload the tiles as revisions, but this does not work, the 32bit tif's created by saving in PI are not accepted by Astrobin.

Michael
Like
Astrogerdt 0.90
...
· 
·  Share link
@Wei-Hao Wang Your idea sounds pretty similar to the multiscale based approach I suggested earlier. I think for such a wide field, we will need either data with short focal lengths (maybe something like 20mm or a little longer?) or full sky images using all sky cameras. 

Regardless which we choose, it will be pretty hard to find someone who operates one of both systems and produces sufficiently high SNR data to build an accurate gradient model. And we would need that multiple times, from all over the world, to account for the complete sky and airglow, lightpollution, zodiacal light, etc. 

I at least doubt that this approach will be sufficient to perform an all sky gradient reduction. 

I will try to reach out to the MDW survey team, maybe they have a good solution to this, although I don't know how severe their problems are considering they work with h alpha data only. 

CS Gerrit
Like
messierman3000 7.22
...
· 
·  Share link
Whatever you people do, I advise you all - if it's possible - not to share data too publicly. Or at least secure everything somehow. I'm just saying - mind the thieves; they could take and use data without crediting any of the original data owners and without their permission.
Edited ...
Like
profbriannz 17.56
Topic starter
...
· 
·  Share link
Michael Ring:
Al little training data will be nice, if you want you can put the data here:

https://www.mycloud.ch/s/S00B9DF137C56385DEA23517B17090B96F38766499E

Data will be automagically deleted after 30 days, until then whoever wants to play with the data can access it via the link.

Michael



Hi Michael,

I have put all my linear .xisf files used to create  my Scorpius/Ophiucus/Libra/Lupus mosaic in this location.  WBPP processed with order=1 ABE, SPCC and light BXT (star reduction 0.1) and default NXT applied.


Brian
Edited ...
Like
profbriannz 17.56
Topic starter
...
· 
·  Share link
Fascinating discussion everyone.

If we can't get the large scale mosacing to work, I still believe an atlas on individual fields on 10deg field centres will be useful.   

No doubt, there are many more experienced people than me at doing mosaics.  I have done two large scale mosaics (300-600degs) around Orion and the Sco/Oph/Lib/Lup region (data uploaded) and 5 or so smaller scale mosaics.  My learning from this have been.  

1) MosaicByCoordinates in PI works really well to register images.  GradientMergeMosaic doesn't work at well in merging.  Obvious seams are left, no matter with values I put in for shrink or feather radii.  I can't rule out if I am doing something else wrong….
2) APP works well where GMM doesn't, but appears to put a lot of green in the multi-band blending process and has obvious field distortions nears the edges. I used APP to mosaic the panes in my large mosaics, but had to crop.
3) Commercial products (e.g. PTGui) work really well at blending - but only on stretched data and who knows what they are doing to the astrometry.
4) Where is RCAstro with a SeamXterminator when we need one?

I have had no significant problem with either galactic background light or zodical light on either of my large mosaics.  As a rule, I always try to observe as far from the sun as posible, so that may help with zodiacal light.  And I may simply not have noticed the galactic light problem.  

CS Brian
Like
Astrogerdt 0.90
...
· 
·  1 like
·  Share link
Hi Brian, 

I looked at some of the images you uploaded, but it seems not all images are online yet, so I couldn't do all that much as of now. 

Some things that came to mind: 
1. As you mentioned, all the images have NXT and BXT applied. Both processes alter the star shape, especially BXT. Juan from the PITeam always stresses, that any change to the stars shape can heavily affect photometry. So this may be one reason why GMM doesn't perform all that good with your data set. This is one of the reasons, why I only put the absolutely essential and least destructive processes into my processing guide. 
2. At least to me, these images look like there are some gradients left that go all over the place in a pretty irregular shape. I don't know how exactly GMM works, but I imagine that it only adapts gradients near the edges. That would ensure that the frames match each other, but would not affect the rest of the precious data in the image. So maybe gradients covering the whole frame are the main problem here. 
3. Also some corners seem pretty dark to me (20_oph for example). Maybe GMM struggles with especially strong gradients. 
4. NXT tends to produce a medium scale noise in the background, which can be clearly seen on the darker places of the images with STF applied. Since GMM uses a multiscale based approach for gradient adaption, this could further complicate the matter. 
5. I guess the zodiacallight issue will become more of a problem if different people work from different places over a long period of time. In your images, it probably remains relatively stationary relative to the sky. But if you change continents, it appears on another place. So our image may well end up with multiple zodiacallights if we do not remove that carefully. 

Don't take any of this as an insult to your data. They are great!

CS Gerrit
Like
profbriannz 17.56
Topic starter
...
· 
·  Share link
Hi Brian, 

I looked at some of the images you uploaded, but it seems not all images are online yet, so I couldn't do all that much as of now. 

Some things that came to mind: 
1. As you mentioned, all the images have NXT and BXT applied. Both processes alter the star shape, especially BXT. Juan from the PITeam always stresses, that any change to the stars shape can heavily affect photometry. So this may be one reason why GMM doesn't perform all that good with your data set. This is one of the reasons, why I only put the absolutely essential and least destructive processes into my processing guide. 
2. At least to me, these images look like there are some gradients left that go all over the place in a pretty irregular shape. I don't know how exactly GMM works, but I imagine that it only adapts gradients near the edges. That would ensure that the frames match each other, but would not affect the rest of the precious data in the image. So maybe gradients covering the whole frame are the main problem here. 
3. Also some corners seem pretty dark to me (20_oph for example). Maybe GMM struggles with especially strong gradients. 
4. NXT tends to produce a medium scale noise in the background, which can be clearly seen on the darker places of the images with STF applied. Since GMM uses a multiscale based approach for gradient adaption, this could further complicate the matter. 
5. I guess the zodiacallight issue will become more of a problem if different people work from different places over a long period of time. In your images, it probably remains relatively stationary relative to the sky. But if you change continents, it appears on another place. So our image may well end up with multiple zodiacallights if we do not remove that carefully. 

Don't take any of this as an insult to your data. They are great!

CS Gerrit


Hi Gerrit,

No insult taken.  My images may well suffer from all these flaws, and they weren't taken with this survey in mind.  I could go back and re-reduce everything for give you just the output from SPCC (before BXT and NXT), but that will take some time.   I do know that, whatever flaws my data had, APP did a much better job than GMM on my data (as far as gradients were concerned). 

Perhaps we really need to work with data taken under "survey conditions".

CS Brian
Like
profbriannz 17.56
Topic starter
...
· 
·  Share link
@Salvatore Iovene  many thanks for the offer of an account for the data.  That appears to have support from this forum, so I suggest this is the way to go.

Would anyone like to volunteer to be the owner?  Since I started this thread, I really should be prepared to volunteer myself.  

Just bear in mind that if I do it: 

1) It is mid-winter here in NZ, so I will be a bit distracted with astrophotography.
2) It is also Matariki in July, and I have a number of charitable events (art exhibitions and presentations) that will further take up my time.
3) As you can see from the above, I can do reasonable small mosaics with existing software packages, but I will need help to do anything larger scale.   My programming skills these days are non-existent.

The other option may be for someone living in the far North to coordinate during their summer as a way to combat AP withdrawal syndrome....

But I am still happy to do it, if the team can put up with a little 24-48 hour delay from time-to-time.  

Please keep the comments/advice coming, as we still have a bit of "down time" while the moon is full.

However, I would look to start the first "Pilot" lunation of the ABC survey around last quarter in the second week of June.  This will be used to assess technical feasibility of all aspects of the survey pipeline, as well as the likely take-up by photographers.  I trust it will quickly establish whether or not the survey is possible at all. 

CS Brian
Like
GoldfieldAstro 0.90
...
· 
·  Share link
This has been a really interesting discussion. I've done a few mosaics myself and still working on creating a 500 panel mosaic of the galactic core with a Nikon D810 @ 675mm FL. I haven't added any more data since COVID began and I've since sold the telescope I was using. Having endured several winters in lockdown (Melbourne based) I'm now trying to figure out how to complete it with the data I've got.

Gradients, has have been mentioned, are some of the most irritating things to work with in a mosaic because as has also been mentioned, identifying what's real nebulosity and what's a gradient towards the edge of a frame isn't easy. My workflow had me chopping back and forth between PixInsight and AstroPixel Processor for processing. I've only recently updated PI after quite a number of years so I do need to try some of the new processes.

Without having a larger template like what @Wei-Hao Wang has suggested I started creating small (3x3) mosaics in APP (for normalisation and gradients). I would then move back to PI to use CatalogStarGenerator to create an accurate star field and then register each image against each star field plus a bit of overlap. This creates astrometrically accurate images for the larger mosaic creation.
Like others I've tried to use PI for creating mosaics and haven't had a lot of success. There is a few more tools in PI now from when I last attempted to use PI for the whole process so it would be interesting to go back and give it another shot.

For creating each final mosaic I have been using AutoPano Giga by Kolor which I've just found out is no longer available. I did try PTGui and personally found it to be pretty terrible for this kind of work. Microsoft ICE I found to work quite well, a lot better than PTGui, not quite as well as AutoPano Giga but it is free.

A few other things to keep in consideration. If you're thinking of imaging at F/2 with 135mm lens' there could be some pretty shocking seams at times due to distortion. I have found that APP does a pretty good job at blending these but it does at times leave some pretty weird artefacts. Also, if there is going to be multiple different lens'; there are quite a number of 135/200mm or even 85mm with a DX sensor (remember that a QHY268C with an 85mm lens gives 9"/pixel and the same FOV as a QHY600C with 135mm) there will be a whole bunch of different lens distortions that you'll have to be able to account for. Samyang 135mm F/2, Zeiss 135mm F/2 and Sigma Art 135mm F/1.4 all have different types of lens distortion.

One more thing I came across is lens reflections. You'll find kinda small strange circular lens reflections from out of field stars that are faint enough to hide as a bit of circular nebulosity.

40 hour 70 panel mosaic
16 panel mosaic captured with Nikon D7200 and Sigma Art 85mm at F/2.8
Constellation of Lupus in maybe 15 panels
36 panels around Scorpious and Lupus
Failed 15 panel mosaic of Rho

Thinking on it, I've already got a bit of data on the galactic core captured with the D7200 and 85mm. Each panel probably doesn't have as much data as you'd like but it's there if you want it.
Like
Astrogerdt 0.90
...
· 
·  Share link
I got an answer from one of the guys working on the MDW survey about how they deal with gradients in their images. 

He told me that they get rid of most of the gradients by rejecting frames with a high background value or complex gradients. This is the first step in reducing gradients. After the integration, he uses MaxIm DL's flatten background tool with simple bilinear 4 point correction. As I do not own MaxIm DL myself, I can't say anything about these tools, but maybe someone from else here has some experience with this tool and can. 

@Brian Boyle I don't know if it will be necessary for you to reprocess all these data, that will surely take a lot of time as you said. We will soon enough get some real life survey data, it seems. They will be a much better test for our ideas, as they will be taken with different cameras under real conditions. Until then we could still use your data. But if you want to reprocess them, I am not going to hinder you ;)

CS Gerrit
Like
whwang 15.16
...
· 
·  Share link
The MDW approach that you described doesn’t sound like something that can guarantee perfect gradient removal which preserves the Milky Way glow.
Like
Astrogerdt 0.90
...
· 
·  Share link
Yes, that is also somewhat the problem I see in that approach, but as I said, since I have not used that tool personally, I can't be completely sure of that. 

If there are any better ideas, we should discuss them. 

CS Gerrit
Like
andreatax 9.89
...
· 
·  Share link
PhotometricMosaic script is the way to go for moderately large mosaics (several tens of degrees) IF lens distortions is small AND you have closely matched PSFs in the feathered areas AND the mosaic tiles form a regular grid (rectangular in [phi, psi]). I mightly doubt you can achieve a seamless full sky mosaic with an heterogeneous array of locations, focal lengths and sensors. But you might be able to create a number of large enough seamless overlapping patches covering the full sky to be of practical use. And let's remember that perfection is the enemy of the good (enough).
Like
MichaelRing 4.64
...
· 
·  1 like
·  Share link
@Brian Boyle, I do not want to be the owner of the account (the honour is yours...) but I would like to help so I'd like to also have access to the account as I am on the Northern Hemisphere, so I will be able to find time to take over tasks when you are distracted by other things. 

I looked arround a little bit and the best will likely be to create a fresh google account for mail that we can either share (for login to Astrobin purposes) or we can configure forwarding in this account to our own private mail accounts. This should hopefully cover the fact that we will log in from two totally different parts of planet earth and for that reason Astrobin will send us an account verification mail on every login.

Hope that works for you and of course, also for @Salvatore Iovene

Michael
Like
Astrogerdt 0.90
...
· 
·  Share link
@andrea tasselli The distortion issue can be solved by using the AlignByCoordinates script. It can use the generated distortion maps by the image solver script to undistort the images. That does not solve the star shapes, but at least the field distortion. 

The regular grid part is also already solved by the planned centers and rotation of the cameras. 

The only problem left is the PSF. I guess we will have to live with that one...

CS Gerrit
Like
 
Register or login to create to post a reply.