New version of Python script to generate target acquisition.csv file from image files in a format suitable for upload using AstroBin's upload csv dialogue [Deep Sky] Acquisition techniques · Steve Greaves · ... · 9 · 790 · 3

Update 28th October 2024


I have just released the latest version of the script. v1.3.11V1.3.11 changes

1.0 Fixes bug where running the script for the first time from the installation directory would fail
2.0 Deals with the case where the filter names in the light frames have trailing white spaces.
3.0 Modified the script to take a new default parameter for the config file:

The parameter is USEOBSDATE and if set to True, the actual date of the observation session is used when aggregating data for the Astrobin .csv output. If this parameter is set to        False, then the date the observation session was started is used.

4.0 Added progress counter to the HFR processing section of the screen output

The USEOBSDATE was introduced  to allow all images captured during a session to be associated with the start of the session. The current behaviour is to associate the image with the date of capture. To use the script in this new mode set  USEOBSDATE = False in the config.ini file. This can be placed in the [defaults] section, an example is given below:

[defaults]
#FITS keyword Default value
IMAGETYP = LIGHT
EXPOSURE = 0.0
DATE-OBS = 2023-01-01
XBINNING = 1
GAIN = -1
EGAIN = -1
INSTRUME = None
TELESCOP = None
FOCNAME = None
FWHEEL = None
ROTATOR = None
XPIXSZ = 3.76
CCD-TEMP = -10
FOCALLEN = 540
FOCRATIO = 5.4
SITE = Papworth Everard
SITELAT = 52.2484
SITELONG = -0.1231
BORTLE = 4
SQM = 20.5
FILTER = No Filter
OBJECT = No target
FOCTEMP = 20
HFR = 1.6
FWHM = 0
SWCREATE = Unknown package
USEOBSDATE = TRUE

I have added a progress counter that appears as the HFR values are being calculated, it seems some of you process such large image data sets that you were not sure if the script had hung or not!

A comment based upon some user feedback:

1. If the script does not find the filter name that is used in the fits headers in the config file then it will use the filter name found in the header in the astrobin .csv file. You need to ensure the filter naming convention used in the header file exists in the config file, along with the code for that filter.

CS Steve


The scripts can be found here

https://github.com/SteveGreaves/AstroBinUploader

The release bundle can be found here
https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.3.11



*************************************************************************************************************************************************************



Hi,

I have release an expanded and updated version of the script I released in November last year. The main focus of the original script was to ease the upload of acquisition data to Astrobin when you have many nights of imaging to report. This version extends that capability.

As before the user runs the script from the command line, where all relevant image directories are passed to the script . The script then uses the  FITS header file information along with externally retrieved data to generate the required data in the correct format. It then produces a acquisition.csv file from this data. The contents of this file can be cut and pasted into the AstroBin's upload csv dialogue. It also produces a summary.txt file that details the target observation information. 

The main focus of the changes in this release is to improve the collection and reporting of observation session information. The session summary information seemed well received so I have expanded that capability. 
Key features include:

• The ability to pass multiple directories via the command line: Multiple directories can be passed to the script via the command line. All images results contained within the directories will be accumulated as part of the target.
• Structured and unstructured directories: Image files, including calibration files, can be collected into a single directory, the root directory. The root directory structure can be flat or contain sub-directories.
• Symbolic links to directories: Symbolic links can be used within the root directory or passed directly via the command line, this is useful when reusing calibration directories. The first directory passed should be the root directory.
• MASTER calibration files: If MASTER calibration files are found, these will be used. If the non-MASTER versions of the MASTER files are also found, the non-MASTER versions will be ignored.Only MASTER files that can be matched with LIGHT file are processed.
• Processing of PixInsight's Weighted Batch Pre-processing (WBPP) output: When the target is a WBPP directory the script will use the calibrated LIGHT frames as well as any MASTER calibration files found in the directory.
• Multiple panel mosaic imaging sessions: Mosaic imaging sessions are detected from the OBJECT entry in the FITS headers. LIGHT frames are processed on a per-panel basis, whilst calibration data is processed per target.
• Multiple site support: Multi-site collaborative target acquisition or remote observatory image capture is supported. Site location data is reverse-geocoded from HEADER location data. Data from multiple sites is reported with summary outputs that correctly identify the site contribution, for instance equipment, LIGHT, and calibration data. All data is, however, aggregated in the AstroBin.csv file for the image target.
• Support for multiple file formats: Extracts headers for all FITS/FIT/FTS/XISF files in specified directories. Directories can have a mix of files.
• Accepts files generated by N.I.N.A, SGPro and PixInsight
• Sky Quality Retrieval: Recovers SQM and Bortle scale classification based on the observation location coordinates.
• Auxiliary Parameter Calculation: Calculates additional parameters like Image Scale (IMSCALE), and Full-width Half Maximum (FWHM) from measured/estimated HFR values for each image.
• AstroBin Compatibility: Formats aggregated data for upload to AstroBin's import CSV file dialogue.
• Target summary: Creates a detailed summary text file for a target acquisition session. Caters for single or multi-site data as well as single or mosaic imaging data.
• Logging and debugging file output: Creates a detailed .log file as well as data dumps to .csv file if required

The changes in this release are quite extensive,  so please consult the readme file before running.

Many thanks to @Ken Bates for his detailed testing of the script and his many useful comments.

An example run output is given below. 

I hope you find it a useful tool.


CS
Steve

The scripts can be found here:

https://github.com/SteveGreaves/AstroBinUploader/tree/v1.3.0

The release bundle can be found here:

https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.3.0


Example Session Summary Output. Single site, 2-panel mosaic, WBPP directory


Summary.txt ouput

acquisition.csv output

Update 1st October 2024

I have just released the latest version of the script.

The code can be found here:

https://github.com/SteveGreaves/AstroBinUploader

and a release bundle can found here:

hhttps://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.3.10

there are quite a few bug fixes along with following changes:

• Corrected utf-8 encoding error with logging
• Reset index on group in summarise session such that group['target'].iloc[0] returns the correct value
• formatted time output seconds are now shown to 2dp
• allow the script to be called from an image directory and process the images in that directory but allows for calibration directories to be passed as arguments e.g. AstroBinUploadv1_3_7.py "." /home/user/images/calibration
• output files are written to the current image directory under a directory called "AstroBinUpload". This keeps session files with the imaging data that was used to generated the data
• modification to accept .fits files from MaximDL and ACP
• handle the case where a fractional part of a second is present in some date-obs keyword but not in others
• handle the case where the filter names in the .fits headers have trailing white spaces.

I have also update the readMe.md/pdf to reflect the changes. Please read this first before to run the script. I think all you need to know is contained it the file.

I hope that this is it!

If you have any questions or comments please let me know.

Hope you find this useful and of course:

CS
Steve

David Hickey:
Hi Steve,

Great tool!  I just tried it and found a couple things.  When it is first run without any parameters and without a config.ini file in the directory, it bails out with an error that argv[1] is not set.  If I move a config.ini into that directory from a previous version of this script then it runs ok, it just doesn't seem to want to create it.  The second thing I noticed is the .csv file that it generates has the filter names in it (RED, BLUE, etc) rather than the index numbers that AstroBin wants.

Dave

Hi Dave,

The situation at the moment is:

1. Run from installation directory with no config file and no arguments passed  : -error, as you have found.
2. Run from installation directory with no config file and arguments passed : script  exits and asks user to modify config
3. Run from installation directory, config file exists and arguments passed : script  runs
4. Run from any other directory with no arguments passed : script exits with indication that arguments are needed
5. Run from any other directory with  '.' passed as the first argument but no config file : script exits and ask user to modify config
6. Run from any other directory with  '.'  passed as the first argument, config file exists : script runs

I will rectify this in my next release.

Regarding the Blue and Red etc, The code will return the filter name if it cannot read the code as a number, This can happen if  there are extra spaces after = and after the code. I though I had fixed this in v3.1.10, may be i didn't. There is a new release coming  where both these issues will be fixed. In the mean time ensure your config.ini filters section looks like

[filters]
        #Filter     code
        Ha = 4663
        SII = 4844
        OIII = 4752
        Red = 4649
        Green = 4643
        Blue = 4637
        Lum = 2906

ie just one space after the = and no space after the code

Dude, if I haven't already told you this, I owe you a beer–or three–or twenty. This script has literally changed my life. I've always been very anal about entering accurate stats into the AB form by hand. It was really painful and took an hour or two for each image–especially mosaics shot over three to six months with 150+ hours of integration time. Now, it takes two minutes and is completely painless. Thank you!

I just found this today after wanting to up my game on accuracy in session information. This is fantastic!

A couple things that I would be curious about. I have a AAG Cloudwatcher that connects to voyager so I have ambient temp and SQM info for every image. They are saved as AOCSKYQU, AOCAMBT in the fits header. Would there be a way to use those instead of SQM and I believe FOCTEMP?