How do you organize your lights, flats, biases and dark frames?? [Deep Sky] Processing techniques · peng155 · ... · 8 · 327 · 1

1.20
...
· 
·  Share link
(deleted)
Edited ...
Like
ChuckNovice 8.21
...
· 
·  1 like
·  Share link
Darks:
/Calibration/Darks/<temperature>/<gain>/MasterDark_<exposure>.xisf

Flats:
/Calibration/Flats/<yyyyMMdd>/MasterFlat_<yyyyMMddsmilefilter>.xisf

Lights:
/Projects/<target>/<filter>/


If I had two target in one night due to the first one going below horizon mid-night then the flat taken is good for both targets. This is why I keep them separate and referenced by date taken. Don't need to keep the flats forever. Once you're done processing your target you can keep the calibrated lights and dump the raw lights and associated flats. Later, if you add more data you only need to calibrate your new data and include the already calibrated ones when stacking.
Like
jhayes_tucson 26.84
...
· 
·  3 likes
·  Share link
Phil,
My favorite tool in PI that goes a long way to sorting out the kind of problem you outlined is the FITSkeyword script.  You can throw a huge pile of data at it that contains different exposures, filters, temperatures, binning, … whatever and it will sort it all out and neatly organize everything by file folders.  For me, it is a huge labor saver.

If you don't use PI, this isn't much help but maybe there's another similar tool you can find to do the same thing.

John
Like
1.20
Topic starter
...
· 
·  Share link
(deleted)
Like
1.20
Topic starter
...
· 
·  Share link
(deleted)
Like
Ewen 0.00
...
· 
·  Share link
John Hayes:
Phil,
My favorite tool in PI that goes a long way to sorting out the kind of problem you outlined is the FITSkeyword script.  You can throw a huge pile of data at it that contains different exposures, filters, temperatures, binning, ... whatever and it will sort it all out and neatly organize everything by file folders.  For me, it is a huge labor saver.

If you don't use PI, this isn't much help but maybe there's another similar tool you can find to do the same thing.

John

Thank you for this! I had never heard of this script before. It looks like a huge time/labour saver.
Like
afd33 9.38
...
· 
·  Share link
Using a cooled camera I just have a folder with my master dark and master bias in it. I update those maybe once a year or so. Then for lights and flats I just keep in separate folders. My folder structure is Camera\telescope\Object\date\exposuretype\filter\

So mine typically would look something like B:\2600mm\FRA300\M31\2024-08-11\light\Red\….
Edited ...
Like
j_paul 2.11
...
· 
·  Share link
Here's the layout of my folder structure. It's been incredibly robust and keeps everything very well organised, and has made my stacking and processing very painless. I take advantage of the keyword functionality in the WBPP script in PixInsight, not sure if Siril has similar, but this kind of strucutre should work regardless.

D:\ 01. Calibration \ Camera \ Year \ DARK   (I also name each file with Gain, binning, set-temp and exposure time)
D:\ 01. Calibration \ Camera \ Year \ BIAS (as above)

D:\ Target \ YR_Cam_Scope_Rotation
  • (eg. D:\ M42 \ 23_2600_106_R145.00 [b]\[/b] (this tells me the year, the camera, the scope and the rotation of this imaging run, this way i can keep all of my imaging grouped by target, and can easily replicate to add new data to any set I wish)

D:\ Target \ YR_Cam_Scope_rotation \ 01. INT
  • (all my files required for integration)

D:\ Target \ YR_Cam_Scope_rotation \ 01. INT \ FLAT \ S_#
  • (I use the "S_#" keyword, where "#" is an incremental number, so i can easily match the flats with the lights, very handy because I usually go a few months on the same flats. When i take a new set of flats, I create a new folder in the FLAT and LIGHT folders and increment the # number.)

D:\ Target \ YR_Cam_Scope_rotation \ 01. INT \ LIGHT \ S_#   (as above)
D:\ Target \ YR_Cam_Scope_rotation \ 01. INT \ LIGHT \ DateD:\ Target \ YR_Cam_Scope_rotation \ 02.STK   (WBPP/Stacking output)

D:\ Target \ YR_Cam_Scope_rotation \ 03. PRC   (all processing tasks/intermediary files/pixinsight project files)

Here is the folder structure in practice for a Mosaic.
image.png
Edited ...
Like
jimmythechicken 19.44
...
· 
·  Share link
Perhaps this is a good time to introduce Lumidex, it is an open source and free astro-image file management program, it reads in header data from .FITS and .XISF files. Everything is loaded into a searchable database allowing you to filter your data and keep track of it in a file structure agnostic way. This in my opinion is the most ideal way to work with your files, it massively simplifies searching for and locating raw and calibration data, even if its spread out over many different locations. We're working on bringing more features to it and would love more community feedback if you're interested. Its very straight forward to get set up and using it and there are tons of ways to filter and sort your data.
Like
 
Register or login to create to post a reply.