Python script to generate target acquisition.csv file from image files in a format suitable for upload using AstroBin's upload csv dialogue Stefan Berg Nighttime Imaging 'N' Astronomy (N.I.N.A. / NINA) · Steve Greaves · ... · 57 · 3546 · 4

SteveInColorado 1.43
...
· 
·  1 like
·  Share link
Thanks again for your work, much appreciated!!
Like
fbitto 0.00
...
· 
·  1 like
·  Share link
Hi Steve,

That’s a wonderful script. It will save me a lot of time.

I noticed that some filters have 5-digit numbers in Astrobin, so I changed the value in  line 914 from 9999 to 99999 so the filter number is not rejected.

Do you think it’s possible to extract darks and biases information from Pixinsight ‘Master Dark’ and ‘Master Bias’ file types?

I use a master dark library and a master bias library containing XISF files already integrated by Pixinsight, in order to save disk space. But, so far, the script cannot extract info from them.

Thanks a lot for your work.

Francisco
Edited ...
Like
AstroBadger 6.62
Topic starter
...
· 
·  Share link
Francisco Bitto:
Hi Steve,

That’s a wonderful script. It will save me a lot of time.

I noticed that some filters have 5-digit numbers in Astrobin, so I changed the value in  line 914 from 9999 to 99999 so the filter number is not rejected.

Do you think it’s possible to extract darks and biases information from Pixinsight ‘Master Dark’ and ‘Master Bias’ file types?

I use a master dark library and a master bias library containing XISF files already integrated by Pixinsight, in order to save disk space. But, so far, the script cannot extract info from them.

Thanks a lot for your work.

Francisco

Hi Francisco,
Thanks very much for your comment.
I am working on a few bug fixes and other mods so I will look at it.
It looks like there is a field in the Pixinsight Master header that gives the number of integration images, so it is possible.

CS
Steve
Like
airscottdenning 1.43
...
· 
·  1 like
·  Share link
Hi Steve,

Sorry to report, I am still getting errors running this on a big directory full of images. Different error now though!

Full output is attached.

Hope this helps!
Scott
errors.txt
Like
AstroBadger 6.62
Topic starter
...
· 
·  Share link
Scott Denning:
Hi Steve,

Sorry to report, I am still getting errors running this on a big directory full of images. Different error now though!

Full output is attached.

Hope this helps!
Scott
errors.txt

Hi Scott,
Thanks.
It seems the problem was how the .csv files were being read in.
Have replicated it and fixed it here and with one other user. I have decided to make a few other requested mods before I release the next version
Explicit support for SGP, beyond using the defaults.csv
Support for Pixinsight master calibration frames. I pass the pre-release to you to test it you have time.
Thanks again for your help
Steve
Like
airscottdenning 1.43
...
· 
·  1 like
·  Share link
Wonderful! I will be more than happy to be a beta tester as long as you wish. 

Great work!
Like
AstroBadger 6.62
Topic starter
...
· 
·  1 like
·  Share link
Update 10th December 2023

I have just released the latest version of the script. 
There are quite a few stability and some feature changes, these are listed below.
I am currently working to include Pixinsight generated Master files. 
Looking at how to be able to deal with these I decided to make major changes to my coding approach.
Once released, what started as a hack for me, now carries quite a responsibility to get it right!
The next release will include quite major changes and quite a bit of testing and I wanted this more stable release to be available.

Many thanks all who commented and tested the script, particularly Ken Bates and Scott Denning.

Please let me if you like the script, find any issues or have feature requests.  

CS
Steve

The script can be found here

https://github.com/SteveGreaves/AstroBinUploader 

The release bundle can be found here
https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.0.4

Version 1.0.4
6th December 2023 changes

1.No longer required to manually create configuration csv files.
   Checks if csv files exist and creates them if they do not exist.
   Default values can still be edited in csv files but also in the configurations dictionary at the start of the code. 
   Default keywords changed to lower case
   To ensure csv edits don't cause issues the data read from files is:
    stripped of leading and trailing spaces
    keywords are converted to upper case to match FITS header keywords
    column names are converted to lower case to ensure code can work with them
    corrected data frames are saved back to csv files to format issues are resolved
2. Improved extract header function
    converts floats to 4 decimal places
    converts dates to format %Y-%m-%d, rounds input to microseconds to ensure conversion works
    creates a subset of the header data that matches AstroBin requirements
3.Sites.csv file latitudes and longitudes saved with 4 decimal places but processed to 2 decimal
   places to ensure the same site is not recorded multiple times.
4. Corrected issue with Bortle and SQM values not being updated correctly
5. Corrected issue with Keywords from .XISF files not being read correctly
6. Improved code to correct file data reading and saving logic
7. Runtime option to stop program if new csv files are created to edit them.
8. Corrected program logic related to import, access and storage of external parmeters.
9. Refactored code to improve readability
10. Updated docstrings
11. Works with files generated by both Sequence Generator Pro (SGP) and NINA (.FITS, .FIT, .FTS, .XISF)
12. Looks for filter in FITS headers and converts them to 5 digit codes used by AstroBin ( used to be four digit codes)
Like
AstroBadger 6.62
Topic starter
...
· 
·  1 like
·  Share link
New release of this script that contains major changes and bug fixes can be found here:

https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.3.3

CS
Steve
Like
fbitto 0.00
...
· 
·  Share link
Hi Steve,

Thanks a lot for continuing improving your script. I've been getting some errors with the last versions, including V1.3.3. But that only happens in some folders, the ones that have only Light frames work OK.

I've attached the .csv debug files to see if it can be of any help.

LBN 535 180mm.zip

Best regards,

Francisco
Edited ...
Like
AstroBadger 6.62
Topic starter
...
· 
·  Share link
Francisco Bitto:
Hi Steve,

Thanks a lot for continuing improving your script. I've been getting some errors with the last versions, including V1.3.3. But that only happens in some folders, the ones that have only Light frames work OK.

I've attached the .csv debug files to see if it can be of any help.

LBN 535 180mm.zip

Best regards,

Francisco

Hi Fransisco,
Could you also send me the .log file from the run.
Thanks and best regards
Steve
Like
fbitto 0.00
...
· 
·  Share link
Steve Greaves:
Francisco Bitto:
Hi Steve,

Thanks a lot for continuing improving your script. I've been getting some errors with the last versions, including V1.3.3. But that only happens in some folders, the ones that have only Light frames work OK.

I've attached the .csv debug files to see if it can be of any help.

LBN 535 180mm.zip

Best regards,

Francisco

Hi Fransisco,
Could you also send me the .log file from the run.
Thanks and best regards
Steve

Yes of course, please find it attached.

AstroBinUploader.log

Thanks a lot
Like
AstroBadger 6.62
Topic starter
...
· 
·  Share link
Francisco Bitto:
Steve Greaves:
Francisco Bitto:
Hi Steve,

Thanks a lot for continuing improving your script. I've been getting some errors with the last versions, including V1.3.3. But that only happens in some folders, the ones that have only Light frames work OK.

I've attached the .csv debug files to see if it can be of any help.

LBN 535 180mm.zip

Best regards,

Francisco

Hi Fransisco,
Could you also send me the .log file from the run.
Thanks and best regards
Steve

Yes of course, please find it attached.

AstroBinUploader.log

Thanks a lot

Hi Francisco,
Can you send your e-mail address to [email protected]. I can then send you a version of the code that I think resolves your issues. If it works I will then make a wider release.
Best regards
Steve
Like
herman 0.00
...
· 
·  Share link
Giving this script a try.  It seems to walk through the directory structure I give it, outputting messages "Extracting headers from directory: \\M42\2022".  After these "Extracting headers" messages it then outputs the message "Read Headers to DataFrame" and then quits without any files being written.  I tried it with a –debug flag and no change.

I have given it multiple directories of PixInsight-generated .xisf files that resulted from WBPP sessions.  I have also tried it on .fts files produced from MaxIm DL.  Same results in all attempts.

I am running Windows 11, Python 3.13, AstroBinUploader 1.3.3

Any ideas what I might be doing wrong?
Like
herman 0.00
...
· 
·  Share link
Update:  My acquisition software uses the value "Light Frame" for IMAGETYP instead of "LIGHT".  AstroBinUploader converts it to "LIGHTFRAME" but it's still ultimately expecting "LIGHT".  I manually changed the value to LIGHT on a test frame and that worked.  I then got errors with the site-name  that came back from the lookup (see below).  I subsequently changed the site name in config.ini and now it all works.  I think now I just need to figure out how to get the script to recognize "LIGHTFRAME" as a light frame rather than requiring "LIGHT".


 Message: 'Processing site: 颍上县, 阜阳市, 安徽省, 中国'
Arguments: ()
— Logging error —
Traceback (most recent call last):
  File "C:\Users\testuser\AppData\Local\Programs\Python\Python312\Lib\logging\__init__.py", line 1163, in emit
    stream.write(msg + self.terminator)
  File "C:\Users\testuser\AppData\Local\Programs\Python\Python312\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 60-62: character maps to <undefined>
Call stack:
  File "C:\Users\testuser\Downloads\AstroBinUploader-1.3.3\AstroBinUpload.py", line 213, in <module>
    main()
  File "C:\Users\testuser\Downloads\AstroBinUploader-1.3.3\AstroBinUpload.py", line 181, in main
    summary_txt = summarize_session(aggregated_df, logger,headers.number_of_images_processed)
  File "C:\Users\testuser\Downloads\AstroBinUploader-1.3.3\utils.py", line 545, in summarize_session
    target = site_details(target_group , site,logger)
  File "C:\Users\testuser\Downloads\AstroBinUploader-1.3.3\utils.py", line 318, in site_details
    logger.info(f" Site: {site}")
Message: ' Site: 颍上县, 阜阳市, 安徽省, 中国'
Arguments: ()
Traceback (most recent call last):
  File "C:\Users\testuser\Downloads\AstroBinUploader-1.3.3\AstroBinUpload.py", line 213, in <module>
    main()
  File "C:\Users\testuser\Downloads\AstroBinUploader-1.3.3\AstroBinUpload.py", line 194, in main
    file.write(summary_txt)
  File "C:\Users\testuser\AppData\Local\Programs\Python\Python312\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 97-99: character maps to <undefined>
Edited ...
Like
AstroBadger 6.62
Topic starter
...
· 
·  Share link
Hi Jeff ,
looks like you are sorting this out. Can I ask you run with the debug option activated and send the files to [email protected]. This way I can see exactly what is happening .
thanks
steve
Like
AstroBadger 6.62
Topic starter
...
· 
·  Share link
Update 1st October 2024

I have just released the latest version of the script.

The code can be found here:

https://github.com/SteveGreaves/AstroBinUploader

and a release bundle can found here:

hhttps://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.3.10

there are quite a few bug fixes along with following changes:
  • Corrected utf-8 encoding error with logging
  • Reset index on group in summarise session such that group['target'].iloc[0] returns the correct value
  • formatted time output seconds are now shown to 2dp
  • allow the script to be called from an image directory and process the images in that directory but allows for calibration directories to be passed as arguments e.g. AstroBinUploadv1_3_7.py "." /home/user/images/calibration
  • output files are written to the current image directory under a directory called "AstroBinUpload". This keeps session files with the imaging data that was used to generated the data
  • modification to accept .fits files from MaximDL and ACP
  • handle the case where a fractional part of a second is present in some date-obs keyword but not in others
  • handle the case where the filter names in the .fits headers have trailing white spaces.



I have also update the readMe files to reflect the changes. Please read this first before to run the script. I think all you need to know is contained it the file.

I hope that this is it!

If you have any questions or comments please let me know.

Hope you find this useful and of course:

CS
Steve
Edited ...
Like
daniele.borsari 5.25
...
· 
·  Share link
Looks like an interesting tool. I'm not very into coding so I ask:

Is it possible to use somehow the date of observation -12 hours to group each session by its starting date (e.g. I start a session tonight and it finishes tomorrow morning, I want the selected date for all the subframes to be 10/01/2024 and not half 10/01/2024 and half 10/02/2024) and also to customize what kind of information needs to be extracted (I want just the # of frames, the duration, the filter, the date -12 hours, the f ratio and the Bortle scale, without all the calibration frames etc. etc.).

Thank in advance,
Daniele
Edited ...
Like
andrewjbarton 0.00
...
· 
·  1 like
·  Share link
Hi Steve, 

I used your script recently for the first time and it worked fantastically. This is going to save me so much time! Thanks for your contribution to the community.

CS,
Andrew
Like
AstroBadger 6.62
Topic starter
...
· 
·  1 like
·  Share link
No problem Andrew.
Thank you for your kind comment.
CS
Steve
Like
AstroBadger 6.62
Topic starter
...
· 
·  Share link
Daniele Borsari:
Looks like an interesting tool. I'm not very into coding so I ask:

Is it possible to use somehow the date of observation -12 hours to group each session by its starting date (e.g. I start a session tonight and it finishes tomorrow morning, I want the selected date for all the subframes to be 10/01/2024 and not half 10/01/2024 and half 10/02/2024) and also to customize what kind of information needs to be extracted (I want just the # of frames, the duration, the filter, the date -12 hours, the f ratio and the Bortle scale, without all the calibration frames etc. etc.).

Thank in advance,
Daniele

Hi Daniele,
I will look at this when I next look to update the script.
CS
steve
Like
Bennich 5.02
...
· 
·  1 like
·  Share link
Hey @Steve Greaves

Testing your script now - and it looks very promising. 
Thank you for making this!!

One quick piece of feedback, as I am currently testing it against a large dataset. 
It would be great to see some counter or progress indicator here, when processing: 
image.png
Edited ...
Like
AstroBadger 6.62
Topic starter
...
· 
·  1 like
·  Share link
Thanks Christian,
I am glad you find it useful.
iI have noted your request and will look at it, along with other requests, for the next update, CS
Steve
Like
daniele.borsari 5.25
...
· 
·  Share link
Hi @Steve Greaves,

I tried to run the script but I get this error and there is no generated csv file. Do you know how to solve this problem?
Screenshot 2024-10-08 143251.png

Daniele
Like
AstroBadger 6.62
Topic starter
...
· 
·  Share link
Hi,
Please send me the log file for the run
Thanks
Steve

[email protected]
Like
daniele.borsari 5.25
...
· 
·  Share link
I've sent the log file to your mail, thanks in advance.

Daniele
Like
 
Register or login to create to post a reply.