The new forums will be named Coin Return (based on the most recent vote)! You can check on the status and timeline of the transition to the new forums here.
The Guiding Principles and New Rules document is now in effect.
BACKUP - a good way to keep TB's of data backed up?
BlindZenDriver
Registered User regular
Registered User regular
So I have like 40 TB and growing that I would like to backup safely, convenient and cheap. What to do?
Currently I use external* hard drives for the job, but it seems to me there must be a better way only I have not found one.
Back in the day where it was more like hundreds of MB putting the data on DVD's worked well, but that isn't feasible with TB's of data. At work I have also used DLT tapes, but they were expensive and also did not have the capacity for multiple TB.
*As in ones in USB enclosures and also some that are really for internal use that I have an external interface for.
Currently I use external* hard drives for the job, but it seems to me there must be a better way only I have not found one.
Back in the day where it was more like hundreds of MB putting the data on DVD's worked well, but that isn't feasible with TB's of data. At work I have also used DLT tapes, but they were expensive and also did not have the capacity for multiple TB.
*As in ones in USB enclosures and also some that are really for internal use that I have an external interface for.
Bones heal, glory is forever.
0
Posts
Roughly speaking my current setup looks like this:
About half of my external drives I have here at home, just sitting in a cupboard, and the other half is in a locked desk at work. It is a growing collection.
Now I have considered a NAS and also running a RAID setup, however neither is really what I need as it is more an backup meant for a total disaster than a day to day one I aim for. It is about being able to recover from say theft, fire or some sort of total IT disaster with crashed or erased drives. The big majority of my data doesn't change much, so I am fine with backing up taking some time and effort. Only using hard drives for like store once and then read maybe one or two times and otherwise just sit there seems a silly waste, there ought to be a medium better suited.
Backblaze's B2 commercial stuff is $72/TB/year (so... thousands for your 40+TB) but their unlimited backup for regular people is unlimited and $100/machine/year. I'm not sure what the difference is (I assume how you can access/download it) but maybe electricitylikesme knows.
EDIT: I realized, I bet one difference is that for the personal backup you have to keep the drives connected to Backblaze, and stuff disappears after 30 days if not. I have gotten poking emails from them when I've gone on vacation for 2 weeks and left my machines off.
You are of course totally correct when it comes to verifying the backups regularly. I sort of do that, but not in a systematic fashion and I really should put more care into that. Due to the amount of data the verify process is rather time consuming, thankfully though the hard work is done by my computer
As for hard drives being the only really viable option, it has also become my conclusion - simply put I have not been able to find an alternative which does not cost significantly more.
@tsmvengy Than you, but your cloud suggestion is not for me - I want to be more in control and I have seen stories of companies either going bust or doing something stupid ie. loosing data. Cost would also likely be a factor, but I have not even looked into those.
I am now of to buying more hard drives and making a plan for systematic verification of my backups
Or you could plop some in a safe deposit box or very small storage unit. If you aren't willing to check cloud options, then you have to think more analog for off-site solutions that avoid immediate risk of flood or fire (or theft) to your residence
That is also a way to do the off site storage, however my current solution using my desk at work is fine safety wise and as I go there most weeks it is easy to swap which drives I keep there.
The important thing is really about keeping a backup in a different location that your main location, it is something I preach to people I know if they ask me about backup. Simply put - one needs to think of the worst and then how to deal with it, otherwise a backup isn't fully secure.
You install a client on the computer, and it'll just back up everything to the cloud, or only specific drives/files if that's what you want.
The intial backup will take a long time, obviously, and will be a problem if you have a data cap with your ISP, but honestly I consider the price for backblaze extremely reasonable for the fact that I can backup unlimited data from a computer to an offsite location.
Well, as said above if you want all the data on one device, you need some kind of multi-drive solution like a NAS.
But you then still run into the conundrum that it doesn't matter if all of your data is on a NAS or external hard drive if they're sitting beside each other on a desk and your house burns down you lose everything anyway.
As for local backups. Yeah unfortunately at 20TB + there isn't really a consumer grade solution that's inexpensive just because of the sheer amount of data. you really are talking about having a second device and backing up from one device to another, which means double the cost. To give you an idea, I do this for a living as a sysadmin, and the medium size business I work for has 40TB of data currently backed up, total. 20TB at a personal level really is at the high end.
We back up from our servers to a local storage device, and then that all gets offloaded to a cloud provider. That cloud provider provides us our ransomware protection through their service via immutable backups, but that's not really a consumer grade solution. Our ransomware protection is not on premises.
At a consumer level, backing up to an online provider is the most cost effective solution. Backing up locally requires a second copy of all data, which does mean having at least that much storage all over again. There isn't really a great way around that.
Also, I'll just point out that you really should ask yourself if you *need* to back up all 20TB. You may, and that's fine. If that 20TB is all photos and personal documents, I get it. If that 20TB is mostly blu ray rips, movies, etc, yeah it would suck to lose most of that but ask yourself, is that truly mission critical that you need to go through the expense of backing up that blu-ray rip of The Avengers that you may watch again at some point.
Again, everyone is different and everyone's needs are different. But as someone who works in the space that's something we always ask. Is the data mission critical? If it isn't, does it actually need to have the expense of that backup solution?
My dad has almost 8TB of photos he's taken with his SLR camera over the last 15 years. Obviously that all gets backed up to backblaze because if his house burns down or he gets ransomwared, there's no getting those back if they're gone. I have about 6TB of data on my computer but I'm backing up only about 500GB worth because the rest is stuff that I don't really care if I have to redownload, or don't really care if it goes away.
And again, I'll just highlight that I do this for a living, and at a personal level my backup solution, and the only thing I recommend to people, is an online solution like backblaze, carbonite, etc. And I mean, really, most people don't even need that level. I generally just tell people to use onedrive/google drive, etc, as the vast majority of people who ask me are just looking to back up a few things and photos. Photos go into services like Google Photos, Amazon Photos, OneDrive etc. But when it is identified that a service like OneDrive or Google Drive isn't sufficient, my recommendation, 100% of the time, is an online service because it's the most idiot proof and cost effective solution.
Really depends on ISP, which is why i said it. Not every ISP has a data cap, If you do it's obviously a problem. Where I am you have to try very hard to actually find an ISP with a data cap. the only one that does even waives it for the first two years of service with them (as an enticement to sign up).
Edit: this is why for a while, we were transporting drives for work between Philadelphia and Eastern CT. We were handling so much throughput that physically moving the media was nearly on par or slightly faster than a data connection.
Single 20 TB drives is a thing, just not in USB enclosures from what I have found.
And they are 20.000.000.000.000 bytes and not 21.990.232.555.520 as one would expect, so in reality they hold 18.6 TB instead. You should be able to find for like $300 and then you just need a SATA to USB 3 gizmo to connect it to your computer, those can be had for like $3 and up - just make sure to get one that will power a 3½" drive as some only work for less power hungry 2½"drives.
There is also 22 TB drives that then actually hold 20 TB, however as it is a new thing they cost like almost twice what the 20 TB drives can be had for. And if money doesn't matter I think Intel something able to hold a few more TB for 10x the price.
Funny, because it is also true. It is still a thing if enough data is involved, be it for a business purpose or just for my personal backup with a ton of video material (and me not running a personal server off-site). But only really when talking moving full data sets, not the day to day thing unless it is some crazy science stuff.
Even moving servers them self is also a non-thing, since virtualization means just moving the data to new physical hosting.