Currently, I'm saving my master files, the PixInsight project files, the result as a high-quality TIFF (maybe 250MB), and finally the resampled image as a PNG for uploading (maybe 25MB). I've started to keep the masters so I can re-process my images as my skills improve.
What files do you save, long-term?
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
For each dataset I image, I keep the following files.
1. All my lights that pass selection processes.
2. Calibration masters used (per image)
3. Master Lights (and any drizzle integrated master light).
4. PixInsight Project containing final Starless and final Stars image, plus the individual channel data, and the rescreened result.
5. Final processed image as *.tif (16bit unsigned integer)
6. Final processed image as 100% scale, 100% quality *.jpg
7. Final processed image as 25% scale, 100% quality *.jpg (for posting online)
If HDD space was tight, I'd get rid of the master lights first… You can re-create those using the master calibration frames and the lights if need be.
Following on from that, I would next delete the PI Project file for any image that I considered "Complete" that I was 100% happy with.
If HDD space was still tight, I'd buy more HDD space.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
Alex Nicholas:
For each dataset I image, I keep the following files.
1. All my lights that pass selection processes.
2. Calibration masters used (per image)
3. Master Lights (and any drizzle integrated master light).
4. PixInsight Project containing final Starless and final Stars image, plus the individual channel data, and the rescreened result.
5. Final processed image as *.tif (16bit unsigned integer)
6. Final processed image as 100% scale, 100% quality *.jpg
7. Final processed image as 25% scale, 100% quality *.jpg (for posting online)
If HDD space was tight, I'd get rid of the master lights first... You can re-create those using the master calibration frames and the lights if need be.
Following on from that, I would next delete the PI Project file for any image that I considered "Complete" that I was 100% happy with.
If HDD space was still tight, I'd buy more HDD space. What's the reason to keep the uncalibrated lights + the calibration masters, versus keeping just calibrated lights?
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
I personally keep all my lights/flats/darks.. Tools like WBPP are constantly innovating and improving and I want to be able to go back to the source if needed.
HDD's are very cheap.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
I have a 4TB HDD and i store everything on it. Mostly i do planetary and have files of around 20GB.
Once full i start deleting.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
If it gets to the stage that i process it I keep it,(unless I think the data is hopeless). Clouds don't allow much data generation anyway so If i doubt its kept,as theres always the chance that processing software and my processing techniques will improve later.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
AmyWarble:
Currently, I'm saving my master files, the PixInsight project files, the result as a high-quality TIFF (maybe 250MB), and finally the resampled image as a PNG for uploading (maybe 25MB). I've started to keep the masters so I can re-process my images as my skills improve.
What files do you save, long-term? I save all files including the PI project files for maybe a week after I finish just in case I discover that something needs to be changed. After that I save the final integrated components, the calibration masters and the calibrated files. I delete everything else. This is on my processing PC. The original lights are still present on my acquisition PC although in 30 years I don't think I have reprocessed more than 10 or so images out of hundreds. Too many objects, too little time.  I do move the files I save on the processing PC from a 4 Tb fast M.2 to a 20 Tb mechanical drive when the week is up so as to free space on the fast working drive for images yet to be processed.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
I keep everything I capture raw, plus masters and my processed results. I don't keep calibrated/registered/whatever preprocessed frames once I'm done with a target because if I reprocess later I will regenerate those.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
All the original light and calibration frames plus the final render for the subject. Everything else goes in the bin.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
I keep my lights, flats, bias, and Pixinsight Projects (one project per target). I have not been using darks since I have a cooled camera and shoot at -15 or -20. If I did have darks, I would keep them as well. One thing I did not see mentioned was backup. I have iDrive and it will automatically backup any file dumped into a folder structure (and subfolders). I don't backup the PI project so that is a risk. I also have a folder structure in which all of my final jpgs for publishing are stored and backed up. Finally, I have a folder structure for all of my master lights which are also backed up. So, my process. - Blink all of my subs removing bad ones.
- Move flat masters to folder structure (backed up)
- Move good subs to a permanent folder structure (gets backed up)
- WBPP to produce the master
- Save the master light to the permanent folder (just the integration, I do not keep the high/low rejection images).
Regarding the WBPP output folders, I save them until I need to cleanup my disk space. I place them on a 2TB internal drive. They are not backed up. Hope this helps Rich
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
I keep every lights, calibration frames, final tifs and Gimp project. Every thing else gets deleted
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
Hello Amy,
Currently I keep all my lights and calibration files. In the beginning I though just keeping the masters was enough to save space, but I've learned the hard way not to do it anymore, mainly because stacking to create them tends to change from time to time, and what's its good now, it's not tomorrow. So the best policy is to keep the originals. From that you could do anything you want form them.
This was also something that I've learned from my boss related to our company archive. Keep the files as much original as you can. Because after that, the process is "destroying" and "degradate" the RAW product, that could be photo, video, audio, analog or digital.
Another suggestion: don't save data in larger disks. Do the exact opposite. Minus disk space, and more disks. Yep it's expensive, but if you value your archive, this way you minimize the data loss if one of them stopped working, loose data, etc. It's more easy to loose a smaller full disk with data, rather than one with a lot of them. Another value lesson that I've learned with the years.
Best regards,
Cesar
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
Saved everything the first 3 years doing this. Now I delete the WBPP calibration, register, and debayer folder. Save sorted/blinked original raw FITS, masters and processed finals with stars and starless, along with any intermittent experimentations in native PI .xisf. And obviously the portable finals for publishing, which are fairly inconsequential for capacity usage when compared to the raws and native finals.
Yeah, I was starting to really hit the ceiling and tired of spending for more storage. Dumping those 3 batch processing folders saves a ton. Just costs time if you want to restack later. Which, I have actually done more than once. But what the heck. Just set it up to run while sleeping or at work.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
During import, I blink my raw subs and delete any bad ones (eg cloudy cover, star trailing). I keep all raw subs and flats.
I make a bias/dark library and once I've created masters, the raw files are placed into a ZIP folder to reduce size.
After WBPP, I keep the master lights but discard the lights in between (eg calibrated, cosmetic corrected, registered) as these can always be generated again in future from the raw subs. Now I have a master flat, the raw flat frames are then put in a ZIP folder to reduce size.
I will keep a Pixinsight project as it saves your history of processing, so if you ever want to tweak your final image using a newly learned technique, you won't have to start from scratch. When a project in complete, I export the final image as a JPEG for uploading to websites.
I may revisit the above approach and keep the calibrated light frames and ZIP the raw lights because I may have a project spanning a couple of years which uses different master darks which can be annoying to group with keywords during WBPP.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
Noah Tingey:
Alex Nicholas:
For each dataset I image, I keep the following files.
1. All my lights that pass selection processes.
2. Calibration masters used (per image)
3. Master Lights (and any drizzle integrated master light).
4. PixInsight Project containing final Starless and final Stars image, plus the individual channel data, and the rescreened result.
5. Final processed image as *.tif (16bit unsigned integer)
6. Final processed image as 100% scale, 100% quality *.jpg
7. Final processed image as 25% scale, 100% quality *.jpg (for posting online)
If HDD space was tight, I'd get rid of the master lights first... You can re-create those using the master calibration frames and the lights if need be.
Following on from that, I would next delete the PI Project file for any image that I considered "Complete" that I was 100% happy with.
If HDD space was still tight, I'd buy more HDD space.
What's the reason to keep the uncalibrated lights + the calibration masters, versus keeping just calibrated lights? The reason to keep the raw lights and raw calibration frames is that stacking algorithms change over time. Should I want to go back and re-stack the the data later, utilizing new calibration techniques or stacking algorithms, I want to be able to do so... part 2 - calibration masters is not 'calibrated light frames' its the master dark, master bias and master flat for each filter. I keep these per image, as if I want to extract an image's files and re-stack that data - I want to have the cals that I used for that specific image available, I don't want to also have to try to rememeber which date the flats I used for that image were taken on.. Once I'm done processing an image, everything other than the final JPG images are compressed fairly dramatically for archiving.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
For me files from 5 years ago I only keep the masters, have 15TB of data and still deleting files, cameras and telescopes and software is getting better and better and old ones are sold, so in my opinion old data will never be re-used (except to good ones) .
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
Storage is cheap BUT it can still be a pain collecting it all and having a whole basket full of portable drives! Having said that, I keep all good subs and Pixinsight Masters although the chance that I'll revisit those files is slim to none …
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
After I have calibrated, stacked, and processed an image I zip the master calibration files with the raw fit files together. It does not take long to calibrate the raw files. This way I am storing zipped 16bit files which are half the size of the calibrated 32bit files. I only keep these zipped files for about 2 years and then I delete.
How long to keep raw data really depends where you image from. If you image from a place where it will take 3 years to get 20 good hours on a target then keep the data 4 or 5 years. If you are lucky and image from a place where it only takes a couple of weeks to get 20 hours of good data then you do not need to keep the data for very long.
Once you have a really good stacked master image it becomes increasingly more difficult to create a better stacked master. More masterLight processing experience and new masterLight processing tools make the biggest difference in the quality of the final image.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
I keep effectively, all files which are generated any any point in my process (except of course, the temp files which are automatically deleted). I could probably cut down on data usage a lot, particularly with the files generated through PIPP as intermediates, but I am not going to, because I have many terabytes of storage lying around.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
For every target I store:
* all light frames
* master dark
* master bias
* master flat
* PI project (so I have history, etc)
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
