I am curious what the arguments are for keeping old data? Especially (or particularly) from the early stages of becoming an Imager. I feel like this would be akin to keeping my grade school art projects.
On the one hand, data storage is cheap. A 1 TB flash drive easily contains all of my old Lights, Master Flats, Master Darks, and Master Bias files. Toss it in the safe and forget about it.
But my thinking is more along the lines of I will never use old data again for several reasons:
1) It was taken from the old house under Bortle 6.5 skies, compared to my very dark Bortle 3.9 skies. (I know the scale does not use tenths, I use them for emphasis.)
2) Much of the old data was taken with an alt-az mount which used a platform rotator to eliminate field rotation, but did not correct camera angle between resets. Thus many of the old integrations consist of wildly varying camera angles.
3) Some of the scopes I used early on have been sold off.
4) Much of the old data was taken using an ASIAir and OAG. Notably inferior to my unguided 10Micron results and NINA HocusFocus.
I have re-processed the old data many times for practice as I learn new skills. I also feel I have wrung out all of the improvements the old data can offer.
Also, it seems to me that adding poorer quality old data to new projects would be dilutive, not additive. Am I incorrect on that?
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
I usually keep everything as long as I have space on my NAS.
Trying to reprocess old and bad data may still be interesting, in my opinion.
First, it's always nice to produce an image from a session.
More importantly, it's a good exercise for one's processing techniques.
And lastly, depending on the weather, this might be the only data left to process…
I just had a holiday for Christmas, and unfortunately, it was cloudy almost all the time.
So, I decided to go back to my old unprocessed data.
It was satisfying to salvage some of my failed sessions and produce an image.
The data for some of them was not that bad, but my lack of skills prevented me from stacking and processing the images correctly at the time.
The data on some others was awful, but with a little more experience, I could produce at least something.
The final images are not necessarily good in any way, but I did learn new things, which is always great.
In the end, a not-so-good image is better than no image, in my opinion.
Otherwise, it's just wasted time outside in the night.
The difference with keeping your grade school art projects is that you cannot improve them in any way.
You still can with old data.
Now, obviously, I started taking pictures in 2024, meaning the amount of data is really manageable.
I think it really depends on how far back your old data goes.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
Also, processing software improves from year to year. There might be things that were impossible to do when you collected the old data that would give t new life now.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
If the outcome I get from a data set is exactly what I wanted, I will SOMETIMES boot the raw frames, but keep the fully integrated/calibrated master file.
However, if I find that the result is not what I'd hoped for, I will zip up the lights, darks, flats and flat darks for the image and store them away with the NINA target information so that I can come back to the data next year, point to the exact same coordinates and add more data…
I have some data sets from 2009 still here on my hard NAS, but then I have images I made at the start of 2024, where I've just drawn the line in the sand, got rid of the raw data and I just have the unprocessed, calibrated/integrated master light in case I want another shot at processing the data.
I then keep Full res, 1/2 res and 1/4 res copies of all the processed images.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
I dont think its necessary to keep very old data. Mainly because I have improved my acquisition to the point that the old data probably does not meet the new standards.
things like autofocusing and better guiding have improved my new subs, as well as better cameras and optics.
also in the early days I would shoot low in the sky, and that NEVER happens now, so the old FWHM numbers were generally much higher.
the subs I have been shooting lately would be worth keeping long term, so I take detailed notes on these for future reference.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
Old data doesn't necessarily mean bad data. State of art camera 10 years ago can be almost as good as today's. Light pollution 10 years ago was much better. So old data can be as good as new ones. If they are of good quality, keep them of course. You may or may not need them in the future. But if you do need them one day, you don't want to realize that you had deleted them a few years ago.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
my storage and data is.. well.. maybe a bit of a mess haha…
I know I have a lot of old data still around, but it's spread out over several different hard drives and such..
I regularly plug in a hard drive I find in a drawer and it has loads of data on it that I forgot about.
I do have plans to go all out, and get a proper large harddrive tower for all of it, but.. that's a future problem.
As for what to keep.. well.. over the last years I've done a lot of upgrading to the rigs. So the data that I'm getting now is obviously significantly better than I got xx years ago.
My rigs are at a point now where I don't really need to upgrade things, I'm happy with the camera+filter combos that I'm using, and I have several scopes that I feel will cover me for quite a while.
so.. I reckon that from this point onwards it's becoming more important to start saving up data as the rigs and data should vary a lot less. Any older data I don't mind discovering on a random harddrive and maybe try a re-process of it. But I doubt I'll put the old data into a current project, as I don't think it will benefit the image. It is actually nice to re-process those images, as you obviously become better at it, and the software improves too.
But there are also other factors. it also depends on your location and how much you can capture on a target; loads of cloud-filled nights means it's more important to save the limited data that you can get. Also you might want to consider what F-stop your scope is and such and then determine how much is worth keeping.
with an F8 scope I feel it's a lot more beneficial to store data vs a F4 scope for example.
Either way, at some point there is a "limit" to what is sensible. the difference between 1 hour on a target or 10 hours is significant. But at some point these jumps become a lot less noticeable. But if you have stored 30 hours on a target, you might need to get a total of 60 before you'd really see an improvement… wether or not that is worth it..
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
I've only been at this for 5 years so I don't really have any old data, but as you said storage is really cheap. I keep my lights and calibration masters. I also tend to keep my processing lights (calibrated, cosmetic correction, debayered) so I can add data over the years. Generally use a 20TB drive for my data and another for my retrospect backups. Seems to work well so far. Cheers!
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
Also, don't forget that the digital world is fragile. Formats and hardware change, HDs and DVD degrade and become unreadable, formats change It might be that the best solution is to make a final image from your old data and then make an archival print. This might sound really old school but think about your old family photos. The reason they survive is mostly because it's a simple print in a shoebox in a closet verses an old scuzzy interface Zip tape drive from the 80's.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
This is a very personal thing, I guess. Whatever you feel comfortable with I would say. For me, I keep my old raw data, my pixinsight project files and final output files. All stored on both a local SSD and a cloud service. Storage is cheap these days, so I see no benefit in throwing it away. What is very important in any kind of archiving is to keep structure. If it is impossible/very difficult to find things back, you will never look at it again. But if it's readily accessible, you might run into a use-case at some point. For me it is a matter of piece of mind. If at any point there is a use-case for it, it would be nice to have the data. Or perhaps better stated, if there is ever a use-case that's interesting, I'd be quite bummed if I had thrown the data away for the sake of freeing up an SSD. The most obvious use-cases are of course the re-processing with better tools, trying out alternative colour palettes, etc. Also you develop your own taste over time. The way you process your images now is probably different than 5 years from now. Not sure if you want to re-process all images based on a new taste, but perhaps there are one or two gems for which you're be happy to go through the trouble. There's one type of use-case that always lingers a bit in the back of my mind as well. How about the things that actually changes in the night sky. Think of novae, asteroids, double stars, etc. Most of what we see changes at time-scales way longer than what we could realistically capture in a life-time. But not all. Novae of various kinds are obvious examples. When AT2019abn flared up in M33, I was lucky enough to have data from before the event, so I could make a 1:1 comparison. Comparing with the final image would probably also work, but in such a case it is nice to have the data from two periods processed the same way, giving maximal focus on the event itself. Besides novae, it might be interesting to see how a Crab Nebula would look like in 10 years time, or a double star system with an orbital period of 20 years. Just my 2 cents.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
I would be careful about any file format that's tied to any particular commercial software as that can go away at any time. I had that experience with CAD program I used back in the 80's. Now, nothing can read it. I do agree, that for historical reference, all old data has value.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
I have digital astro data from a first production run STL11000m camera I purchased back in 2004, more than 20 years ago. This data was taken using wider field equipment from dark skys when i was traveling to do imaging. These data are still good and I go back to the data, using new processing tools and looking for objects to image with my newer-larger equipment. I consider this a valuable resource. All of this is on-line residing on several NAS drives, and backed up on blu-ray DVD's.
I still have Astro-image film data, from before the digital age, as far back as the early 60's. Its value is limited, but provides an emotional connection to younger years.
All this takes very little space and costs very little to keep. Your choice, but for me I kep it all…
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
I don't have all your variables to consider. Most everything taken at the same location except for a few travelers. Equipment does vary. But I am really glad I have saved the majority of my data. Things change so much with the processing technology and my skill/comfort using it that when I consider how much clear skies are at a premium here, the idea of just flushing adequate images seems wasteful. Not to mention we are all subject to our own changing personal aesthetic that would have us creating something different from the same media set over time.
Don't get me wrong. I have thrown out some absolute garbage over time. And indeed may consider some material I am currently keeping to be unusable in the future.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
Gary Seven:
A 1 TB flash drive easily contains all of my old Lights, Master Flats, Master Darks, and Master Bias files. Toss it in the safe and forget about it. According to the Beta 5 Computer instructions, all Class 1 Supervisors must store all gathered data indefinitely via the Beta 5 Exceiver Circuit; *however* Class 1 Supervisors "Can" override previous instructions on the Beta 5 Computer !
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
Gary Seven:
A 1 TB flash drive easily contains all of my old Lights, Master Flats, Master Darks, and Master Bias files. Toss it in the safe and forget about it.
According to the Beta 5 Computer instructions, all Class 1 Supervisors must store all gathered data indefinitely via the Beta 5 Exceiver Circuit; *however* Class 1 Supervisors "Can" override previous instructions on the Beta 5 Computer !
Well done sir ;-)
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
So I guess it is almost unanimous. Keep the old data.
As to mixing it in with new data, I can always give it a try.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
So far I have kept most of my 3.5TB - and whenever I deleted some parts, I deeply regretted it a year later, when I had found a new way to calibrate or process better.
What has not worked out so well was to intehrate new with old data, as my imaging skiils have improved so old data is poorer data as well so far.
But since my 4TB SSD will soon be full, I'll have to move out to external disks - which will then end up in a drawer to be forgotten, which is a modern way of throwing it away, isn't it.
Arny
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
Once I am happy with the integrated image from an observing run, I delete all intermediate data files (I mainly use PixInsight, so there is a considerable volume) and archive the original (acceptable) lights and the various intermediate and final integrated images.
My logic is that I can always re-run the processing sequence from the original light frames, or test some other new bright idea, which I do on occasion. I find this approach cuts down the archive data volume requirement by at least a factor of twenty - my original light frames are ~50Mb, while my intermediate file sets are all -300Mb per frame.
Of course, there is an implicit assumption that future software will be able to read and process archival formats, but that is probably a reasonable risk (says he who lived through Y2K in the IT industry).
Regards
Tim Anderson
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
Yes I tend to keep everything - you never know when you can add to it or reprocess it, which is great fun and good practice when the weather is bad.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Copy the URL below to share a direct link to this post.
This post cannot be edited using the classic forums editor.
To edit this post, please enable the "New forums experience" in your settings.
