| Would you be interested in contributing towards an AB all sky survey? | |
|---|---|
| No. I wouldn't find such a survey useful. | |
| No. Satisfactory data already exists for me elsewhere. | |
| No. I would find such a survey useful, but I don't have the time, location or equipment to contribute. | |
| Yes, I would be interested in taking part. One or two fields maximum. | |
| Yes, I would be interested in taking part. Prepared to do multiple fields. | |
| Login to vote and view results. |
17.56
...
·
 |
|---|
James Tickner: My thoughts entirely…. |
1.20
...
·
·
4
likes
|
|---|
@Michael Ring Here's a first-pass stitch of Todd's Polaris data. I haven't tried to correct for field-to-field background levels (which are quite variable) or any colour normalisation differences - my main focus was to test the stitching and alignment. I've only uploaded as a JPEG as the 16-bit RGB PNG image is about 170MB and a RGB 32 bit image (the original precision) would be double this. The colour of any mosaic pixel is simply calculated as the average RGB value of any fields overlapping that point. I haven't tried to do any more sophisticated 'blending' yet. As the fields are quite jumbled, the number of overlaps ranges from 1 to 5. Overall, I think the alignment seems pretty good - stars in the overlap regions are nice and round. The final mosaic measures 9125 x 10357 pixels, has a pixel scale of 10 x 10" in the centre and is oriented with RA = 0h pointing straight up (I hope!). I used a gnomonic projection centred on the N pole. Note that as the image is cropped to be just large enough to fit the 10 panels, the pole doesn't line at the centre of the image. Hopefully that all makes sense! Compute times are reasonable: - 10-15 s per panel for initial plate-solving - About 12-15 s per panel for reprojection - About 2 s per panel for image combination So altogether about 5 mins, with about half of this due to the initial plate solve which is a one-off cost. And as everything is scripted the 'human time' factor is pretty low. I'll have a play with correcting background and colour normalisation over the next couple of days.  |
0.00
...
·
·
1
like
|
|---|
@James Tickner, that looks fantastic! And yes, the alignment of the frames was a little bit of a hot mess. Manually framing around the pole with a Skyguider Pro was... interesting.  |
4.64
...
·
|
|---|
@James Tickner Really nice progress! Have you checked if montage can take it from there? Main problem for me in montage were the circular patterns because of not so perfect alignment, but perhaps with prealigned data it can continue it's magic. Compute times are super reasonable compared to MosaicByCoordinates, looks like you are on a very good way.... |
|
17.56
...
·
|
|---|
It is coming up to the end of the test lunation, so I thought I would try to summarize where we are up to. Three positives. 1) Despite a relatively small number of observers (and terrible weather conditions for most), we have extensively tested the observing and linear processing pipeline based on a 9 x 6 deg field with 7.5 x 5 deg field centres with just under 1% of the survey fields. 2) The field centre distribution, observing and linear-processing pipeline appear robust. The booking form works well. 3) Stitching of data at the 4-6 field level [200 sq deg or less] has been achieved [including with data from different users], but PI fails or is too cumbersome for larger mosaics. @James Tickner is making good progress on an alternative software solution for the stitching. Stitching appears to work best with SPCC only [ie no ABE, BXT, NXT] applied to the output data from WBPP. Note also that, due to computational issues I did resample to 10arcse/px during the MosaicByCoordinates process] Outstanding issues [for constructive debate] 1) Although data has been successfully shared, the use of DropBox has proved to be problematic for some. We need to settle on a solution workable for all. [I have paid for an annual subscription to DropBox, so it would be good to use it if we can, but I don't want to force a time-consuming solution on others.]. We although need to settle on an agreed naming convention for uploaded data. I propose Fxxxx i.e F0001 - F1120.xisf for the auto-cropped linear output image from the WBPP as per the processing pipeline. And that's it. Once the pipeline is going and many fields are coming in, I don't think we will be to cope with anything more complex. And certainly not more than one image per field. 2) Curation and presentation of the data. Assuming that we can generate much larger mosaics, would we still want to present data in smaller chunks [say 15 x 15 deg ] covering the sky in 200 frames or so? The smaller fields might be better to visualise the data, and highlight fainter DSO in the non-linear processing. @James Tickner wuold it be easy to generate a larger 15 x 15 field centre grid, listed those 7.5 x 5 fields that would make up each larger field. Might be useful for regular data release? 3) When do we resample and photometrically calibrate in the process? As mentioned above, I have run SPCC on the WBPP output and then initially generating the mosaic pieces using MosaicByCoorindates, resampling to 10arcsec/px at that stage. This is primarily driven by compute time of me, as I guess we can see what James recommends using his routine. This will be a tradeoff between quality and time. 4) How do we carry out any non-linear post-processing? My post-mosaic pipeline [run on the images above] has been BXT -> NXT -> STG (auto) -> Arcsinh [auto black, 1.25 stretch] -> HT on stars only [Black point raised by 25%, with starmask from SXT]. This is almost certainly too much, but I wanted to really push on the visibility of seams. Personnel I believe we have established the feasibility of the survey, but we do need the personnel. Over 100 people have now indicated that they would be prepared to take part via the survey, but we currently have only a few active imagers. We will need additional people on stitching/QC as well, but we do need to come to a position on mosaicing pipeline first. Good to get people's thoughts on how we can further promote the survey. Note that the Survey Announcement thread has been very inactive. In response to a member's comment, I did ask the Astrobin moderators whether it could be pinned, but I got no response. I am not going to push this. I get the sense this is not a priority for AB at present. I think that is fair enough, as we do need to demonstrate progress and take-up first. For that reason I have not yet paid for the ABC AB account yet. A free account allows up 10 images. So, if we agree that 15 x 15 deg field releases are a good way to go, I will only need to pay for an Ultimate subscription once we have completed around 5% of the survey. And then I think we can mount a more vigorous campaign. So at the moment, I suggest we focus on getting to the initial survey products of a few large mosaiced fields, using whatever pipeline and software we decide on. The initial fields that we could use are Fields at -70 to -85. North Pole @Todd Stephens data Scorpius [my data] Milky Way Plane at -30 to -45 and perhaps also focus a coordinated campaign next lunation on a few fields that could make up 2 or 3 more 15 x 15deg "chunks" for promotional purposes. What do people think? Brian |
|
0.00
...
·
·
2
likes
|
|---|
I’m happy to contribute additional panels, especially around Polaris once the wildfire smoke clears. I’m building up a second rig with my old iExos-100 that pairs nicely with a 135mm lens. But the way things are going, I’m concerned that it might last all summer long. if it looks like it would work, I do have additional existing panels that extend from Polaris to the Milky Way through IC1396, so that’s a pretty big chunk of sky. |
5.25
...
·
·
2
likes
|
|---|
In the last weeks the weather has been poor here in Italy. As soon as we get to the next lunation and there are clear skies, I'll try to contribute by imaging some panels. With the Samyang 135mm (with plate solving I noted that the real focal lenght is 129mm btw) and a Canon 400D the FOV would be the one required for the mosaic, but the un-modded camera would lead to color balance issues. Anyways, I'll try my best to be part of this project and to deliver some good data. CS Daniele |
|
1.20
...
·
·
1
like
|
|---|
Todd Stephens: Completely understand the challenges of aligning fields near the pole - I had exactly the same issues with my south polar attempt. Even with platesolving and auto alignment the scope thrashes around and struggles to settle. |
|
17.56
...
·
|
|---|
James Tickner:Todd Stephens: The Polar field in the UK Schmidt Photographic survey - done back in the 80s and 90s - was also notoriously difficult to obtain. Each passband (BVRI) took a number of attempts. So, I can feel your pain. |
|
17.56
...
·
|
|---|
Daniele Borsari: Grazie Daniele, Benvenuto nel gruppo. Brian |
|
1.20
...
·
|
|---|
@Brian Boyle Responding to some of the points you raised above:
|
|
4.64
...
·
|
|---|
Lots of stuff to respond to.... This post is on Brian's points: Dropbox: I think what you have can work, but unfortunately most of the finalizing work will have to be done by you, @Brian Boyle as the business model of dropbox does not seem to support collaboration in an efficient way. What I would propose is that you create an 'incoming' folder and send an upload link to that folder to anybody asking for it or even publish it in a well known place. The uploaders do not need to have a dropbox account so the upload issue should be solved. The link can stay constant so strictly speaking people will not even have to ask Brian for the upload link. The hairy part will be how to continue from that point. My proposal would be a 2nd directory named 'accepted' where the upload link is only known to the qa team. qa team members should download data from incoming, do checks on quality and put data to accepted. Then Brian can put the data to the final place and delete it from incoming. This should keep load on Brian to a minimum unless he himself is a member of the qa team.... As an alternative we can use my MyCloud account, the business model behind this Cloud storage is different so it is less restricted. It seems I can easily give write access but have not yet tested if people can alse move/delete files. Storage is big enough.... On naming convention I think we at least need the astrobin name of the uploader so that qa team can get in contact with the uploader, besides this the field number should be fine. And never say never to not more than one image per field. As we are talking about raw data here that should stay as intact as possible we will habe more than one file for a field when we accept already existing data like Todd's. The raw data will always be the point where we go back to when James has created new magic or some new processing tool has evolved. For the incoming folder I agree, perhaps we can design the qa process in a way that the result of it is a proper 9x6 field as we expect. @James Tickner , a very good help would be that your script can go through a dataset and after platesolving and orientation correction can put it in folders for proper 6x9 fields, possibly duplicating files in the process. This can ensure that qa can check if all data for a 9x6 tile is available. Curation: My main goal is still to create a survey for Nina (and perhaps other tools like stellarium) so my only interest at the moment are those smaller chunks. I think I read somewhere that 10x10deg is the max size for tiles in Nina but your 15x15 will work well as they should be fairly easy to convert to the smaller size. So two thumbs up from my side for the 15x15 superfields. Those 15x15 fields can also be a nice targets for collaboration of several people. Sampling: As long as we always keep the original data downsampling to 10arcsec/px is fine at some point, the later that point is in the process the better. Promotion/Aquisition: Several things come to my mind, once we can import Todd's mosaics with acceptable effort I will contact other creators of large mosaics and ask them if they are willing to share their data. I think this will be the most efficient way to fill up the skies. Another idea would be to try to activate few youtubers that they talk about the initiative and perhaps also share a few fields of their own work. But for this it is still a little early, we first need to show off some of our work so that we can proove that the project is serious. Also Nina Discord or Astrobiscuit Discord could be a place to go for extra audience, but unfortunately Discord is a very shortlived medium so we will not gain much. We should definitely create an Astrobin Group, this is one of the easy ways to reach people outside of this thread without much effort. I may sound like a broken record but this thread is way to big now. It must be horror to walk through for a person that is new to the topic but it is filled with valuable information that may be found easier in seperate threads. This way of working is possible in a group when I understand astrobin's group concept correctly. It is also the place to promote our finished data. And we should think about putting final data in the IOTD process, winning that would be a perfect promotion.... However, main thing will be to activate the people that actually agreed to help creating data, judging from activity on this thread we will likely have to find a way to contact them via Astrobin PM as you already proposed. For Northern Hemisphere it could make sense to wait another month so that nights get longer and imaging gets easier, for SH this could start already for the next run. Michael |
|
4.64
...
·
|
|---|
Now my take on James points: Unmodded Cameras: Glad you brought up the point about the unmodded cameras, are we able to mark areas as 'unmodded friendly'? I could double efficiency if I could use my unmodded Nikon Z6ii, then I can do 2 fields at once which helps in my Bortle-4 skies. And I am willing to photograph those boring parts of the skies were little is going on as I let this camera run independant of my own aquisition. Still, it is painfull to see that I spend 8hrs of integration on 2 fields where not much is going on when I could have done the same in 4hrs. Whole Sky Image: Three thumbs up!!!!! Also on the idea to define areas of interest so people can work together on those areas. giving out some 15x15 areas with Field numbers and asking people to concentrate on those will give us a boost (I hope) because others can join the discussion and make friends with such a common goal. Once again, a astrobin group could help for this, we could open up threads for several 15x15 regions and let people organize themselves under those threads. XISF: A quick solution would be to download GraXpert, open a file there and save it as fits. I am pretty sure that many will forget to save in the correct format, so conversion will have to happen anyway. Debayering: This one I do not understand, but it may be because you do not use PI. The result of stacking in PI will always be a properly debayered image, the debayering itself happens very early in the Weighted BatchPreProcessing script. Programmatic Approach for Processing: Two thumbs up, too many artistic choices, 3 people will come up with at least 6 different ways to process the data, we need a baseline to work with, when people feel the need to optimize an area then they should do that, we can replace areas lateron with better interpretation. But as BX is PI only we would have to see how we can incorporate this in your automated process……. Code sharing: The best would be that you set up a project on github, there we also can use the Wiki to do documentation, it could also evolve as a good entry point for the whole project where we share upload links, rules, guidelines, etc…. so best choose a project name like Astrobin All Sky Survey or similar. From what I understand Matlab requires a license to run code, if this is true then I can try to port your code to python, there are some quite big libraries for Astro available in python so it is a good choice for code that can also be run by others in an open environment. Data: For me not the underlying tile repository would be the ultimate output of the project, the repository of all the raw stacks that make up the tiles are the true gold…. |
|
4.64
...
·
|
|---|
Now to my own points: Licensing: I may sound like a broken record but I think it is essential that the project works under a official and well known/prooven license and that contributors agree to that license. So here my vote would be for either CC BY-NC or the most permissive license CC BY with a personal bias to CC BY-NC https://creativecommons.org/licenses/?lang=en Core Team Metting: We should also find a good way to organize a small core team and to perhaps learn more about each other and our goals and strengths in a conf-call once Brian is done with his other tasks for organizing his event. Optimizing Imaging Time: Does anybody know a good website that allows planning of imaging sessions based on Moonrise/Moonset? July will again be a month were it can be possible to image with a rather bright moon because it sets early or raises late. This oportunity to image is sometimes overlooked and helped me a lot in June to cover most of the clear skies we had after 5 months of 90% clouds... Coding: Kudos to James for his work, what he has already done is amazing!!!!!! |
|
4.64
...
·
|
|---|
@James Tickner : found this project, works fine on my mac: https://github.com/vrruiz/xisfits I also created a small script that starts with xisf, creates fits+jpeg and then platesolves the jpg, was quite easy to implement in python with the help of stackoverflow. So far this is arround 30 lines of code, so chances are that we will find an equivalent of what you do in python. I already found a function to create a mosaic, hopefully I will get there later today, this will at least help qa team to easily judge if a field is properly filled with data. There also seems to an array that can tell me if an image contributes to an area or not. This is an interesting read: https://reproject.readthedocs.io/en/stable/celestial.html Michael |
|
17.56
...
·
|
|---|
Thanks @James Tickner @Michael Ring - great work. Unfortunately I don't have much time over the next two weeks to respond as I will be busy with exhibition opening and a number of public speaking events around Matariki. So if I don't respond, it is not that I am not interested - just busy. [I will still be booking/taking fields - since over the past two weeks I have made a purpose built enclosure for my wide-field rig allowing me to run each night automatically with minimal setup.] Just a few general comments. 1) I am hugely impressed by everyone's coding skills. And it reinfornces that I should not be the person coordinating the curation effort. 2) Our current pipeline excludes un-modded cameras and requires PI. The majority of our observers break that rule currently  I have no issue with rule breakers - they tend to be the innovators of course, but I think we do need to show that it doesn't introduce biases or significant extra work into the survey. The test for me will be combining @James Tickner data with me in the south polar region without any obvious colour gradients. I would also observe that the new PI WBPP with LocalNormlization is a very good linear processing pipeline which really appears to maximise SNR. [Much better than old WBPP - at least in my experience] 3) Format conversion would not appear to be a time-consuming issue. It was straightforward enough to convert James' .fits into .xisf. But we would have to agree on numerical accuracy. [Same as default .xisf?] 4) Naming convention. I agree with @Michael Ring that we do need a little more info. FXXXX_astrobinID_focallength_camera. ie F0191_BrianBoyle_200_2400MC 5) I look forward to the development and demonstration of the tools currently under development and support @Michael Ring comments about creating appropriate licences. I will continue to take data around the meridian crossing for suitabley southern declinations over the next lunation, focussing where possible on large super-fields. Note that we haven't really agreed on a super-field size yet. Nor specified their centres. Two options: 18 x 18 deg size on 15 x 15 centres Pros: large area, most impressive and can cover the size in 100 fields per hemisphere. Cons: difficult to create - requiring three dec strips and therefore up to nine fields. More likely to be "interrupted" by weather in the early stages of the survey. 12 x 12 deg size on 10 x 10 centres. Pro: Easier and quicker to create - requiring typically half the number of fields as the larger super-fields. Cons: Less impressive and many more fields need to cover entire sky. Thoughts? Brian |
|
17.56
...
·
|
|---|
An update. I have managed to recover 8 ABC fields from my Scorpius mosaic. I have also managed to image a further 3 new fields and complete observations on a further 3, during the early part of the night when the moon is down.
|
|
4.64
...
·
|
|---|
That explains the mails from the google sheet…. I was able to add more time to two panels that were missing some integration time, Monday and Thursda night look promising for acquisition. Anyone else in the northern hemisphere taking the opportunity to gather some photons? Those Bortle 2 guys from the other hemisphere are stealing us the show…. 😉 Michael |
15.16
...
·
·
1
like
|
|---|
I am wondering if we can ask to create a forum dedicated to this survey. This thread is getting too long to follow, and the other announcement thread is buried too deeply to find. Ideally the topics should be split into different focused discussion threads and a few threads (announcements, FAQs) can be pinned on top. All these are better achieved with a dedicated forum. I planned to take some northern panels in this coming new-moon weekend. The weather has been excellent in Taiwan these days. Unfortunately the forecast for this weekend bad. We will see. |
|
17.56
...
·
·
1
like
|
|---|
@Wei-Hao Wang I think you are absolutely correct. Time to refresh. I did tag @Salvatore Iovene in the survey announcement thread to see if it could be pinned, but did not get a response. In many respects, I understand this - as we have yet to prove we can do this. But the time may be soon... I do think we need a better "presence" in the Forum. @Michael Ring@Astrogerdt and @James Tickner have all suggested separate forum discussions for different aspects of the survey. And this would work well if we had our own Forum. As mentioned above, I do think we have to earn the right to have our own Forum. But I believe we have shown that we have a good pipeline, field centres and - even if we still do not know how to mosaic the whole sky, we are getting there..... Here is a 4 (in RA) x 3 (in Dec) grid of survey fields covering just over 1% of the sky. Comprised of data taken specifically for the survey, and data retreived from a previous mosaic. Post-pipeline processing was SPCC -> MosaicByCoordinates [PA 0 deg, 10 arcsec/pix] -> TrimMosaic [5pix] -> PhotometricMosaic [stiched in RA, then Dec] and that was all. [The annotated image has had standard STF applied and I had to bin2 to reduce file size to post here. For our first 1% I don't think it is too bad.  |
 ...
·
·
1
like
|
|---|
Brian Boyle: Hi Brian, sorry about missing that (another hint that, as @Wei-Hao Wang mentioned, this thread is getting too long). I recommend you get a dedicated space for this project by creating a group: https://www.astrobin.com/groups/ You can classify it as "Ad-hoc collaboration" I suppose. I can also move this thread there after it's created (the link will be redirected so anybody clicking an old notification or a bookmark will still reach it). Then you can have as many topics as you want, plus, when there's a new topic in a group that people have joined, they get notified. |
|
4.64
...
·
|
|---|
@Salvatore Iovene : Yes, I think that for a lot of reasons creating a group is a good choice, however, there is one thing I fear: When we move to a group we will not reach new/old participants of this thread anymore. As groups only send notifications to group members we will loose the notifications to people that already actively read this thread. Is it possible to create a new forum? That way everybody can participate in ther discussions they care about without the 'need' to join. And my hope would be that when you move the thread to that new forum the subscriptions will keep working. When this is not wanted/possible it would at least help to pin the announcement thread, then we can give status updates there from time to time and ask interested people to join the group. Thank you, Michael |
...
·
|
|---|
Hi Michael, you can: - post here asking people to join the group - manually invite them to group using the moderator tools in the group - when somebody joins a public group, two things happen, to help with the group's visibility: 1. a new activity stream action is posted to the front page 2. followers of that user get a notification "User X, whom you follow, joined public group XYZ" When I move this thread to the group's forum, notifications for this thread will still work. Adding a new public forum on AstroBin would have the benefit that your topics will be on the "Latest topics" for everyone, but: - what about people who don't want to see that? - what about other requests to add new forums? There's groups exactly for that, already So I think I need to decline the request to created a public forum for this project, and continue recommending a group. And yes, I can pin this topic here, but I don't know how much it will help. Pinning it means that it's at the top of the "Astrophotography / Other" forum, but not at the top of the "latest" topics, so only people who actually go and browse the "Astrophotography / Other" forum will be more likely to see it. I recommend you keep posting updates here, with links to the group, so it also sits at the top of "Latest topics" more often, which gives more people chances to learn about the project. Hope this helps! |
|
4.64
...
·
|
|---|
Thank you for the additional hints! Pinning the survey introduction thread (not this one) would be nice, at least then the survey has a fixed entry point that is visible for those browsing through the forums. Besides this I think it is now @Brian Boyle who should create the new group (Brian, I can also do that if you want, your decision, you're the boss....) so that you can move the thread over there. Thank you, Michael |
|
17.56
...
·
|
|---|
Thanks… I have created a new user forum for the Astrobin Community Survey and including a number of threads under it. Please feel free to additional threads.
|
