Python script to generate target acquisition.csv file from image files in a format suitable for upload using AstroBin's upload csv dialogue Stefan Berg Nighttime Imaging 'N' Astronomy (N.I.N.A. / NINA) · Steve Greaves · ... · 57 · 3549 · 4

Update 28th October 2024


I have just released the latest version of the script. v1.3.11V1.3.11 changes

1.0 Fixes bug where running the script for the first time from the installation directory would fail
2.0 Deals with the case where the filter names in the light frames have trailing white spaces.
3.0 Modified the script to take a new default parameter for the config file:

The parameter is USEOBSDATE and if set to True, the actual date of the observation session is used when aggregating data for the Astrobin .csv output. If this parameter is set to        False, then the date the observation session was started is used.

4.0 Added progress counter to the HFR processing section of the screen output

The USEOBSDATE was introduced  to allow all images captured during a session to be associated with the start of the session. The current behaviour is to associate the image with the date of capture. To use the script in this new mode set  USEOBSDATE = False in the config.ini file. This can be placed in the [defaults] section, an example is given below:

[defaults]
#FITS keyword Default value
IMAGETYP = LIGHT
EXPOSURE = 0.0
DATE-OBS = 2023-01-01
XBINNING = 1
GAIN = -1
EGAIN = -1
INSTRUME = None
TELESCOP = None
FOCNAME = None
FWHEEL = None
ROTATOR = None
XPIXSZ = 3.76
CCD-TEMP = -10
FOCALLEN = 540
FOCRATIO = 5.4
SITE = Papworth Everard
SITELAT = 52.2484
SITELONG = -0.1231
BORTLE = 4
SQM = 20.5
FILTER = No Filter
OBJECT = No target
FOCTEMP = 20
HFR = 1.6
FWHM = 0
SWCREATE = Unknown package
USEOBSDATE = TRUE

I have added a progress counter that appears as the HFR values are being calculated, it seems some of you process such large image data sets that you were not sure if the script had hung or not!

A comment based upon some user feedback:

1. If the script does not find the filter name that is used in the fits headers in the config file then it will use the filter name found in the header in the astrobin .csv file. You need to ensure the filter naming convention used in the header file exists in the config file, along with the code for that filter.

CS Steve


The scripts can be found here

https://github.com/SteveGreaves/AstroBinUploader

The release bundle can be found here
https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.3.11



*************************************************************************************************************************************************************

Update 10th December 2023

I have just released the latest version of the script. 
There are quite a few stability and some feature changes, these are listed below.
I am currently working to include Pixinsight generated Master files. 
Looking at how to be able to deal with these I decided to make major changes to my coding approach.
Once released, what started as a hack for me, now carries quite a responsibility to get it right!
The next release will include quite major changes and quite a bit of testing and I wanted this more stable release to be available.

Many thanks all who commented and tested the script, particularly Ken Bates and Scott Denning.

Please let me if you like the script, find any issues or have feature requests.  

CS
Steve

The script can be found here

https://github.com/SteveGreaves/AstroBinUploader 

The release bundle can be found here
https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.0.4

Version 1.0.4
6th December 2023 changes

1.No longer required to manually create configuration csv files.
   Checks if csv files exist and creates them if they do not exist.
   Default values can still be edited in csv files but also in the configurations dictionary at the start of the code. 
   Default keywords changed to lower case
   To ensure csv edits don't cause issues the data read from files is:
    stripped of leading and trailing spaces
    keywords are converted to upper case to match FITS header keywords
    column names are converted to lower case to ensure code can work with them
    corrected data frames are saved back to csv files to format issues are resolved
2. Improved extract header function
    converts floats to 4 decimal places
    converts dates to format %Y-%m-%d, rounds input to microseconds to ensure conversion works
    creates a subset of the header data that matches AstroBin requirements
3.Sites.csv file latitudes and longitudes saved with 4 decimal places but processed to 2 decimal
   places to ensure the same site is not recorded multiple times.
4. Corrected issue with Bortle and SQM values not being updated correctly
5. Corrected issue with Keywords from .XISF files not being read correctly
6. Improved code to correct file data reading and saving logic
7. Runtime option to stop program if new csv files are created to edit them.
8. Corrected program logic related to import, access and storage of external parmeters.
9. Refactored code to improve readability
10. Updated docstrings
11. Works with files generated by both Sequence Generator Pro (SGP) and NINA (.FITS, .FIT, .FTS, .XISF)
12. Looks for filter in FITS headers and converts them to 5 digit codes used by AstroBin ( used to be four digit codes)

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Update 28th November 2023

I have just released the latest version of the script.

The code can be found here:

https://github.com/SteveGreaves/AstroBinUploader

and a release bundle can found here:

https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.0.3

28th November 2023 release changes are: 

• Handles pre and post text spaces in data from csv files

          Fixes the error that some reported when completing the  input .csv files

• Can process FITS and XIFS files or a mixture of both

          Broadens applicability of script.

• Focal ratio now extracted from header and reported.

           For completeness

• Exports a session summary report

          Session summary as a .txt file

I have also update the readMe.md to reflect the changes. Please read this first before you run the script. I think all you need to know is contained it the file.

I hope that this is it! If you have any questions or comments please let me know.

Hope you find this useful and of course:

CS
Steve

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Update 24th November 2023

I have just released what I hope is a better functioning release. 

The code can be found here

https://github.com/SteveGreaves/AstroBinUploader

and a release bundle found here:

https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.0.2

Major changes to how the code handles missing Keywords from FITS headers.
The code use a defaults.csv to enable the user to configure values for missing keywords
These default keywords are then applied to all missing header keywords allowing for a more complete upload of information to AstroBin.
HFR is also recovered from the defaults.csv file, instead of the command line entry.

The changes attempt to make the code agnostic to the types of FITS headers processed.
ReadMe.md has been updated to reflect the changes

I think it is worth trying this release and I would welcome your feedback

I am looking at modifying the code to read .XISF files next

CS
Steve

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Hi,

Sometimes I need many imaging sessions to complete a target. My last target, a two-panel mosaic of NGC 7822, required 20 sessions . When uploading the image to AstroBin, the submission of all the relevant acquisition data gets very tedious.

To help I have created a python script to automatically generate the data in the correct format for upload using the AstroBin's upload csv dialogue.

The user runs the script from the command line, where all relevant image directories are passed to the script . The script then uses the  FITS header file information along with externally retrieved data to generate the required data in the correct format. It then produces a acquisition.csv file from this data. The contents of this file can be cut and pasted into the AstroBin's upload csv dialogue. The script will only work with images captured by N.I.N.A.

The code can be accessed here, with what I hope are detailed instructions:

https://github.com/SteveGreaves/AstroBinUploader

I hope some of you find this a useful tool, let me know what you think.

CS
Steve

I have created a release bundle the enables the easy download of all the files. It can be found here:

https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.0.1


-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
An example of use is given below:

Executing the script:

python3 AstroBinUpload.py "/mnt/HDD_8TB/Preselected//NGC 7822" "/mnt/HDD_8TB/Preselected/Calibration data/30th April 2023" "/mnt/HDD_8TB/Preselected/Calibration data/20th April 2023/DARK" 

Execution output:

Reading FITS headers...

Observation session Summary:

LIGHTS:
  Filter Blue:     90 frames, Exposure time: 1.0 hrs 30.0 mins 0 secs
  Filter Green:     63 frames, Exposure time: 1.0 hrs 3.0 mins 0 secs
  Filter Ha:     126 frames, Exposure time: 21.0 hrs 0 secs
  Filter OIII:     110 frames, Exposure time: 18.0 hrs 20.0 mins 0 secs
  Filter Red:     94 frames, Exposure time: 1.0 hrs 34.0 mins 0 secs
  Filter SII:     127 frames, Exposure time: 21.0 hrs 10.0 mins 0 secs

Total session exposure for LIGHTs:     64.0 hrs 37.0 mins 0 secs

FLATS:
  Filter Blue:     50 frames, Exposure time: 9 secs
  Filter Green:     50 frames, Exposure time: 8 secs
  Filter Ha:     50 frames, Exposure time: 23 secs
  Filter Lum:     50 frames, Exposure time: 2 secs
  Filter OIII:     50 frames, Exposure time: 25 secs
  Filter Red:     50 frames, Exposure time: 6 secs
  Filter SII:     50 frames, Exposure time: 28 secs

BIAS with GAIN 0:     100 frames, Exposure time: 0 secs

BIAS with GAIN 100:     100 frames, Exposure time: 0 secs

DARK with GAIN 0:     50 frames, Exposure time: 50.0 mins 0 secs

DARK with GAIN 100:     50 frames, Exposure time: 8.0 hrs 20.0 mins 0 secs

Used existing Bortle of 4.0 and SQM of 20.52 mag/arc-seconds^2 for lat 52.248, lon -0.123 from sites.csv.

Completed sky quality extraction

Data exported to NGC 7822 acquisition.csv

Contents of NGC 7822 acquisition.csv file generated:

Thanks, that sounds super useful, I created something similar at https://github.com/phtnnz/astro/blob/main/astro-countsubs.py but without support for the upload format required by Astrobin. Will have a look ...

Yes, very useful.

The newer system for recording dates and details of imaging on Astrobin is better than the old one but still tedious  to the point that I gave up on being that specific since my imaging sessions for an object are seldom less than about 10 nights.

This will help!

Steve Greaves:
Hi Scott,
It would need to be modified but as long as I can decompress and extract the header it should be possible.
I will look at it.
CS Steve

Hi Steve,

The FITS headers in the compressed xisf files from NINA are plain text.

I don't think you need to decompress but you would need to find the xisf instead of fits files in your script and extract the <FITSKeyword name= ... />"  tags.

Thanks for this!
Scott

Steve Greaves:
Hi Scott,
It would need to be modified but as long as I can decompress and extract the header it should be possible.
I will look at it.
CS Steve

If you didn't run across it already, there's some development here:  https://github.com/sergio-dr/xisf

@Scott Denning

Hi Scott,
I have looked at the .XISF standard from PixInsight and there is no direct python support for, i.e libraries. It is not , however, that complicated to extract the header once you know the file structure. I have written a couple of functions that do this and will integrate them into the code. It would be helpful if you could test this for me before I make a general release. If you want to please send me your e-mail address.

CS
Steve

@Scott Denning
Hi Scott,
This is the dataframe the code reconstructs from the XML you posted. So I think it will work. I will post the code , probably tomorrow.
CS
Steve

To all,
The data Scott sent me shows me that I need to make a few changes to the code to make it more robust. I will address this in the next release. The issue is that I, naively ,assumed the header file from NINA would hold standard fields that are populated if the device is connected and not if the device is mission. This is not the case. If anyone wants to download the code, please wait until I have fixed this tomorrow, UK time. I don't want to waste your time.
Thanks and CS
Steve

Thanks for the script, great work!!  I just tested it successfully (Python 3.9.10, Windows 11).  One small note:  I was getting an exception in reading the filters until I noticed that in editing filters.csv I had inadvertently added white space (a single space) after 'code' in the first line.   You might want to use strip() to remove leading and trailing whitespace, since some of your users might have trouble diagnosing the issue.

This Github repository might help with the xisf format.  XISF Object  Written in Python by Sergio who wrote and maintains the PI Auto Integrate tool. Astro Image Tools

Daniel

A note for users.  With my mono setup and EFW, NINA sets the 'FILTER' field in the FITS header according to how you've named your filters in NINA.  Works great with the script.  With my OSC setup, I use a filter drawer and don't bother with the 'FILTER' field in the FITS header.  

Assuming you've used the same filter throughout with an OSC, you can still get the script to populate the filter field with the code as follows:

Add a filter name and code to filters.csv.  With my Optolong 2" UV/IR cut, I added:  OUVIR,5532.  Then in defaults.csv, change the FILTER line to FILTER,OUVIR,Optolong UV/IR Cut.  The acquisition.csv 'filter' field will then be populated with the code  (5532 in this case).   

Alternatively, one could manually edit the acquisition.csv file.

Thanks for sharing this code Steve. How do you get an apy key for https://www.lightpollutionmap.info/QueryRaster/?

Guy

Update 28th November 2023

I have just released the latest version of the script.

The code can be found here

https://github.com/SteveGreaves/AstroBinUploader

and a release bundle can found here:

https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.0.3
28th November 2023 release changes are: 

• Handles pre and post text spaces in data from csv files

          Fixes the error that some reported when completing the  input .csv files

• Can process FITS and XIFS files or a mixture of both

          Broadens applicability of script.

• Focal ratio now extracted from header and reported.

           For completeness

• Exports a session summary report

          Session summary as a .txt file

I have also update the readMe.md to reflect the changes. Please read this first before to run the script. I think all you need to know is contained it the file.

I hope that this is it! If you have any questions or comments please let me know.

Hope you find this useful and of course:

CS
Steve