FORUMS: list search recent posts

QNAP TVS-871T and Amazon S3

COW Forums : NAS - Network Attached Storage

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Michael Raff
QNAP TVS-871T and Amazon S3
on Feb 3, 2017 at 8:59:36 pm

We recently acquired a 48 TB QNAP TVS-871T (saw Bob Z's great post-NAB review) which we are now using as a Nearline Storage solution and also as the home of our archive of deliverables, masters & submasters.

We have a smaller and aging EditShare system as our online shared storage solution, but it's going on 7 years old and has only 24 TB at RAID 6 (so more like 18 TB usable) and we are debating whether to upgrade or replace it with something else. In the meantime, the QNAP gives us some valuable additional storage space.

I see in the QNAP's Backup Station application that there is an option for Cloud Backup to Amazon S3. However slow this process might be, it represents a potential option for us to store a complete backup of our archives offsite, which would be an added layer of insurance that my bosses would appreciate.

I know nothing about Amazon S3 or its costs, but assuming we can afford it, can someone explain in simple terms how the QNAP backs up to the cloud? (I am correct in assuming that after an initial full backup, it can do incremental backups?) How easy is this to set up and run?

Any insights would be appreciated.

TIA,

Mike Raff
Williamsburg, VA


Return to posts index

Bob Zelin
Re: QNAP TVS-871T and Amazon S3
on Feb 4, 2017 at 6:21:34 pm

I am glad that you are having success with your QNAP 871T. QNAP continues to release amazing products at dirt cheap prices.
You are using your QNAP as near line storage for your aging EditShare, but what you will find is that even if you are using AVID Media Composer that requires bin locking, simply running Indiestor Mimiq on your edit system computers will allow you to use the QNAP as your main shared storage system. And of course, the 871T while wonderful, is the small unit. They have larger (not much more money) products like the TVS-EC1680U that are incredibly powerful, and perform as well as the most expensive shared storage systems out there today, all at 10G (and even 40G) speeds.
And as you know, the QNAP's come with the 10G card pre installed, so just add a 10G switch from Netgear (and now Buffalo) and you have a super fast system that is dramatically faster than what you have in your aging system.

With that said - today speed is everything. Observe the slow transfer from your EditShare to your QNAP as near line storage. It's slow - perhaps painfully slow. You could add LTO tape backup. This is even slower than near line storage.
Now lets consider cloud backup, like Amazon S3. You think near line is slow, and LTO tape is slow - well, you aint' seen nothing yet. This defines "the turtle". Backing up or restoring ProRes, DNxHD, R3D, XAVC files to the cloud is justification for suicide. Basically, it's totally useless in my opinion. It's great for your QuickBooks Pro app, or any accounting, spreadsheet, sales data, or even h.264 and h.265 files, - but for ACTUAL VIDEO MEDIA - the cloud is totally useless, as if you actually needed something, because your place burned down, or was robbed, your clients would leave you anyway, because it would take you forever to download those files again (and it will take you forever to upload those files). A 100 Mb/sec uplink (which I bet you don't have) is still painful to transfer 100 Gigs of Media - to transfer 1TB - forgetaboutit! Now try to transfer 48 TB - are you kidding me. In 2017, it's completely unrealistic, unless you have the good fortune to have Google Fiber at your place that offers 800 Mb/sec upload speed (not even Disney has this).

So what's a cheap way to do this ? Buy a QNAP TX-800P, plug it into your 871T, clone your drive (which is still a slow process) and move this offsite to a storage facility. And when your place gets robbed or burns down, most of your media will be on this second redundant array. But what about your "on going " new media. Well, that's a process that you have to put into place in your company. With larger companies, you setup an off site backup, via a fiber (10G) cable in another building. Or make tapes (or redundant) backup array and move them off site.

But to realistically think that backing up 48TB to S3, and have it actually usable for you - well, that's just silly.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Neil Sadwelkar
Re: QNAP TVS-871T and Amazon S3
on May 29, 2017 at 10:20:51 am

I just came across this old thread, and thought I'd pipe in with realistic costs for cloud storage. I had done a comparative study for a client who eventually went with cloud storage for smaller project files and ProRes masters, but left raw files for a cloned disk system.

Cloud storage plans are usually in $ per GB per month and are deceptively low when one sees them expressed in those units. Like fractions of cents. But the real cost quickly adds up when that data is kept backed up for long periods. And there is a steep cost of getting that data back too.

Between Backblaze, Amazon S3, Microsoft Azure and Google cloud, I found the costs to upload and store to be about $5 to $ 26 per TB per month. the download cost is between $ 50 to $ 80 per TB per month. Once you take about 10 TB into consideration and, say, 2 years of storage then restore, the cost balloons into tens of thousands of dollars. Way more than what a second nearline storage off-site, or LTOs stored off-site would cost.

Then there's the time to upload/download. At speeds 100 Mbps and under, the time to upload and download, say, 10 TB, ranges from 10 days (for 100 Mbps) to 3 months (for 10 Mbps)! And, obviously, 100 TB is exactly 10 times that.

So, as Bob has written, cloud storage for pro video files - not yet.
We'll get there when we all have 40 Gbps Internet easily available.

-----------------------------------
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India


Return to posts index


Bob Zelin
Re: QNAP TVS-871T and Amazon S3
on May 29, 2017 at 10:59:09 pm

a wonderful reply from Neil as usual.

Here is the simple answer to near line storage. As you may have discovered, the QNAP is equally as fast as the
EditShare, at a fraction of the price. Want near line storage - buy a second QNAP and run QNAP Backup Station RTRR (and if you don't like that, run ChronoSync, Carbon Copy Cloner, etc.) - want it across town, in another building - you are at the mercy of your internet provider. Unless you have google fiber, you will pay a FORTUNE to have 800 Mb/sec.
Excuse me - you will pay a fortune to have even 100 MB/sec upload speeds, so in the US, keep it in another part of your building and you will have wonderful near line storage (and if it's a 10G connection in your building, you will have backup that is not nearline - but full speed).

People always ask me "what"s next". What's next is already here - but the politics of installing fiber lines (like Google does) is not going to happen in the next 20 years in the US. Companies like Spectrum (Charter Cable which was Time Warner) find it much cheaper to make a 10 Million Dollar campaign contribution to multiple state Governer's and Senators, than to spend the money to actually dig up the streets, or GOD FORBID allow Google to enter the state so they can put them out of business with 800 Mb/Sec for everyone. The future is here. It's now just politics to allow it to happen.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Yuval Dimnik
Re: QNAP TVS-871T and Amazon S3
on Jul 31, 2017 at 12:16:09 pm

"Once you take about 10 TB into consideration and, say, 2 years of storage then restore, the cost balloons into tens of thousands of dollars."

Here are my calculations for the described scenario. Would love feedback.

Standard S3:
Storage costs: 10TB * $0.023 per GB (Standard Storage) = ~$230/M = $5,520/2Y
Egress costs: 10TB * $0.090 per GB (Data Transfer OUT From Amazon S3 To Internet) = $900
Total: $6,420/2Y

Glacier (cold archiving)
Storage costs: 10TB * $0.004 per GB (Glacier Storage) = ~$40/M = $96/2Y
Egress costs: 10TB * $0.090 per GB (Data Transfer OUT From Amazon S3 To Internet) = $900
Glacier Data Retrievals: 10TB* $0.0025 per GB (Bulk - 5 - 12 hours acceess time) = $25
Total: $996/2Y

https://aws.amazon.com/s3/pricing/

yuval.dimnik@noobaa.com
http://www.noobaa.com


Return to posts index

Yuval Dimnik
Re: QNAP TVS-871T and Amazon S3
on Jul 31, 2017 at 12:17:56 pm

The cloud is not usable as an online storage, especially if your workload is not hosted on the cloud.
That said, it is a great solution for DR, archives and even nearline.
In these cases you won't be exposed to the high read out of the cloud costs (egress costs) or metadata costs, and it's an opex model on pay per use on data you actually store.

You can push 1TB of data per day with one dedicated line (you don't want it to kill your email....) of 100Mb/s.
In the states, you can get 500Mb/s if you add $100/M (Not local so this is what I found https://www.verizon.com/home/fios-fastest-internet/) and then one can push 5TB per day.

The cloud providers provide a tool to import and export large amounts of data that can help with initial load and DR if the site was burned.
http://docs.aws.amazon.com/snowball/latest/ug/whatissnowball.html
https://cloud.google.com/data-transfer/docs/introduction

yuval.dimnik@noobaa.com
http://www.noobaa.com


Return to posts index


Bob Zelin
Re: QNAP TVS-871T and Amazon S3
on Jul 31, 2017 at 3:38:37 pm

please tell me why someone would prefer to use Amazon S3 or Google Drive or DropBox, for massive amounts of data, when you can simply buy a QNAP, keep it at a remote location, and accomplish the same thing ?
At the beginning, you can bring the QNAP to your main shared storage system, copy over the data, and now move it to a remote location (like your home, instead of your office). So you bought the $3000 QNAP and the drives, but now your expense is over, and you can use MyQNAPCloud or CloudLink to access your data, or backup additional data.

If you have ever used Amazon Snowball, you will see that the process is not easy - it's not plug and play. There is no easy to use "app" for Mac or PC to upload or download from an Amazon snowball - its all CLI commands in terminal.
So for those that say "I just don't want to deal with this, I would rather pay someone" - well, you will be paying someone to get your Amazon Snowball to work.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Yuval Dimnik
Re: QNAP TVS-871T and Amazon S3
on Aug 3, 2017 at 12:54:57 pm

Disclaimer: I do NOT work for AWS or any other cloud provider.
In addition, there is no argument about the fact that just storage - GB for GB, the cloud is more expensive.
That said, off the top of my head, here are some reasons S3 is great:

Variable OpEx vs. CapEx.
==============
See this for easier than any text: https://www.cloudtp.com/wp-content/uploads/2016/04/traditional-view.svg

Data Redundancy
==============
If you use QNAP with large drives and raid you risk raid rebuild failures and data loss. On the cloud - not your concern
http://www.enterprisestorageguide.com/raid-disk-rebuild-times

Logistics and logic
=============
Buy a QNAP, bring it on site, migrate, take it to your home and connect it to power/network while it takes some space. Need reasonable network at home to support data replication assuming anything is done at the office during the day.

vs.

Write your data to S3.
Done.

...Get
DR
A repository where you can share from to other users.
Work on your data using cloud services (e.g. https://aws.amazon.com/digital-media/aws-elemental/)
If cloud storage prices drop you enjoy that drop.
If you delete data, you don't pay for that data anymore (unlike storage).
If a drive fails... Somebody else's problem.

Flexibility, agility, the ability to adapt and change, adopt new technologies offered by the cloud etc are some additional benefits.

Yuval
yuval.dimnik@noobaa.com
http://www.noobaa.com


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]