video storage questions
Hello there... guestions about video storage. In you DIY6 you list two 250gig WD's for storage and G-SPEED eS as an option. Do i need the two internal HD's if i use the G-SPEED eS ?
Besides their storage capacity what benefits do the two internal HDs have for video editing while using the G-SPEED eS
Does the G-SPEED eS showup as 4 HDs or one large drive ?
Short answer is that the G-Speed should be fast enough to capture/edit right onto it.
However, I think the DIY6 system wanted to make sure the best hard drive performance was available for the capture/edit and that is why they recommended a RAID 0 which should be very fast. Not very safe, but fast where the G-Speed is safe and almost as fast.
On one hand i want to save money. On the other hand, to have good equipment. On both hands to be as hassle free as possible.
Somewhat slower with the G-Speed does not sound to bad if safety is built in.
I am a newbie and will have many questions... i'll try to make them good questions.
In the DIY6 configuration the G-speed array would be aligned to the raid0 3hd internal setup. Would the PCIe x4 RAID Controller for the G-speed have any relationship between the OShd and the two WDhds.
In trying to understand this, i figure i need a primer on relationships between the OS and raid arrays internal and external.
There is not any special relationship, most of us are just trying to handle 3 issues: We want more speed, we need space and we need our data to be safe. And of course we want all of it to be easy! (Things you mentioned too.)
So, for speed, my preference (for years) is to seperate the O/S from the data. Install two hard drives. Put the O/S on the first and create your own folders on the 2nd and save everything there. Make the 2nd drive as big as possible. That is how it all started.
Now I do backups, but I still don't like restoring or reinstalling the O/S. So I started doing a mirror for the O/S. Not quite as fast, but safer and once setup, easy. Haven't lost any data or had to restore from backup for about 3-4 years now.
Next, I needed more space and I could buy more drives, but then its more complicated (D:, E:, F:, etc.) and drives failed so it wasn't easy or safe.
That bring us to RAID 0 which can string 2 or more drives together for speed and a lot of space, but not very safe. Next choice is RAID 5 where you can string 3 or more drives together and it is a LOT safer, a little faster and provides almost as much space as RAID 0. This can be done on most modern motherboards today. Just add drives.
For me, I bought a big case, got as many SATA ports as I could find and did RAID 0 (O/S) and RAID 5 (Data). Very safe, lots of space and pretty fast. Still, I knew there was more speed to be gained if I wanted to spend the money.
Along comes eSATA which is much faster than USB or Firewire and you can put the drives outside your case and do RAID 0/RAID 1 or even RAID 5 if you have enough drives. Or use it for fast backup. It is just a logical extension of your motherboard and just as fast.
So as hardware and software has evolved it becomes easier and cheaper to handle those 3 important issues.
So no special relationship, just using the technology as I see fit. I am sure other people have their way and I wouldn't argue with them. Technically, you could have one drive for the O/S and have everything else on the G-Speed. Or you could have (3) 750 GB drives in a RAID 5 which will give you 1.5 TB of available storage.
Or my new way: You could buy your own RAID controller ($400), add (4) 500 GB hard drives ($400) and do it yourself. 1.5 TB of high speed RAID 5 for $800 if the drives fit in your case. Faster than the G-Speed, faster than single drives AND very safe but it took me two days to setup because of a strange motherboard issue. Now that it is setup, it is also easy to handle.
Feel free with any questions.
I am confused as to the efficiency of the the system as compared to the task given. Correct me if i'm wrong... that what i want to achieve is both, the ability to edit in close to real time, and render a file... as fast as possible on the same system.
I'm trying to figure out these two functions in relationship to the system.
As i understand, NLEs such as Avid Liquid have the ability to render in the background as you continue to edit.
I like the idea of a dedicated controller. Not sure how manage the controller, the NLE, the editing, and the rendering all at once.
I've heard about raid0+1... not sure i understand it... sounds good, speed and safety.
do i understand correctly that i can have the following:
* i want to have a raid0 setup to split up the edit environment so that it is as seamless as possible
* i want to save/render the file as efficiently as possible... maybe at the same time that i'm editing....
* i want to have all my files as safe as possible at same time.
.... nice graphics, with lines connecting the OS, NLE and hardrives would be helpful.
here is a RAID article i've found
Sorry, not good at graphics, but you need to realize the hard drives are setup in advance and you don't worry/think about the RAID when you are working on your PC.
Lets take the example that you already have a PC running XP. You have 1 hard drive that is labeled "C". That is the normal way PCs are setup.
Now, you want to add, speedy, safe storage for your video editing. RAID 0 is NOT used because it is not safe. With 2 drives we can do RAID 1, 3 drives can be RAID 5 and 4 drives can be RAID 5 or RAID 10 or RAID 0+1.
Lets put 3 (500 GB) hard drives into your computer and connect to a new RAID controller. So you would turn off the computer, install the hardware and then turn on the computer. Very early in the boot process (before XP) you would see new messages coming from your RAID controller. It might say press CTRL+I to setup. So you press CTRL+I, go into the RAID configuration and setup a RAID 5 using the 3 drives. After you are done the system reboots and this time you don't press CTRL+I. You let XP load.
XP will need the software drivers to recognize the RAID card and then you use the XP Disk Management program to create a new volume (1 TB in size) and assign a drive letter - "D". XP DOES NOT KNOW this is a RAID array. Your controller monitors the drives.
Now you still have the "C" drive where XP is loaded and your NLE is loaded. But you also have a speedy, safe 1 TB "D" drive to use.
All of that setup is done one time and from now on you use the "D" drive to save your data, your renders, etc. The drive is used the same way as any other drive. Neither XP nor your NLE know or care it is a RAID array. There is nothing special about it while inside XP. You create folders, files, delete items, etc.
So, when go to capture some video you shot, you save in a folder on the "D" drive. When you edit the video and render, you save the render to the "D" drive again. When your NLE is reading the video to show you a preview, it will be reading off the "D" drive because that is where you are saving it.
You can even open Photoshop and configure it to place its temp work area some where on the "D" drive.
Now, one morning you wake up and you need to finish editing for that important client. Your turn on your PC and XP boots, but the RAID controller software reports that your RAID is degraded. Probably it means a drive has failed.
Normally this would be a disaster. When did I do a backup last? Did I backup after I did that 6 hours of editing 2 days ago? Do I remember exactly what I did since the last backup?
However, with the RAID 5 protecting you all you do is go buy a new hard drive. Replace broken drive with new one. Use RAID software to rebuild array (About 3 or 4 mouse clicks) AND YOU CONTINUE to use the system, even while the RAID is rebuilding. NO restoring from backup. No panic that you might have lost something. Everything is exactly where it belongs. You could even take a risk and continue working WITHOUT replacing the drive. Everything will work, it will be a little slower. However, if a 2nd drive fails BEFORE you replace and rebuild you will lose all your data.
This post is a little long. I am going to break into two posts.
You mentioned RAID 0+1. Each version of RAID has pros and cons. RAID 1 is mirroring. You put in (2) 500 GB drives and you only get the space of (1) 500 GB drive. So a mirror loses half of your space and lets you recover from a 1 drive failure. With RAID 5 you need at least 3 drives and you lose 1 drive. So (3) 500 GB drives in RAID 5 gives you a 1 TB volume. (4) 500 GB drives gives you a 1.5 TB volume. Each one can recover from a 1 drive failure. RAID 10 or RAID 0+1 lose half the space and not all controllers let you set this up. RAID 6 loses two drives and I think it is a (4) drive minimum, but you get recovery from (2) drives failing at the same time. So (5) 500 GB drives in RAID 6 gives you 1.5 TB of space.
Here is how I have my ssystem setup:
About 4 years ago I bought a large computer case. It has room for 11 5.25 drive bays in the front. So now I use hard drive enclosures that allow me to get hard drives in/out without opening the case or even turning off the computer.
I installed a mirror and put Vista x64 on it. The mirror is my "C" drive and I install all programs there along with the O/S. Both drives are in enclosures.
I have (4) 500 GB hard drives in a RAID 5 connected to an Adaptec 5405 RAID controller. 1.5 TB of space. Drive "D". I put ALL my data here. I put all the drives in enclosures. The Adaptec is new and very fast.
I have an external enclosure (Venus T4s) that sits on my desk. It has a simple RAID card and eSata connection. I use (4) 300 GB drives in a RAID 0. This is my temporary space. "E" drive. I have setup Photoshop, Sony Vegas and any other programs to use this as temp space. Fast, but not safe is perfect for that kind of storage.
Anyway I arrived at this configuration after trying different setups over the years. It is fast and works for me. I am not saying it is the only way to go and probably if I started all over I would make a few changes, but I just wanted to give you ideas.
Let me know if any of this helps.
... for the speedy response... i'll need a while to crunch this. from the start it looks to me as helpful info. As i see it, the OS/programs stands between my ideas and what the hardware can do.
i think there always has to be a common point of reference. i think the OS/programs, is it. the chicken lays the egg.
after learning a bit about raid i've decided based on economics that, to start, i will set up an internal raid5 with a controller card. your model of having a raid0-temp system makes sense but, i would need to save more money for that.
how do you link your raid0-temp ? by usb, firewire or eSATA ? does the MoBo have eSata connectors or do i need another raid controller card. if it is by firewire, does the DIY6 MoBo have a firewire port and even if it does is a firewire card a better choice, and if so is firewire800 a better choice yet.
you stated that you had some headaches with "strange motherboard issues" in setting up your raid controller. could you give a walk-through of setup issues related to the various components of the DIY machine. in otherwords, a step by step of how you would proceed if you had all the components in front of you on a table.
i have found various bits of advice scattered throughout the video guys site, such as "storage controller needs it's own IRQ" 'DMA settings' 'install updated drivers before you do this or that' but a step by step walk-through of hard/software must-dos, basic options and considerations for building from scratch would be a great help for all newbies.
If the walk-through is set up like a logic tree (where, for example "step 2 is this or that see step 4", and so on.
That the component and software choices for a DIY system do not change to fast, this walk-through logic tree could be updated as the industry introduces new standards, technologies and options.
This might be a large task but i think it would be a great benefit to new consumers and serve as a center point for all the info at the Videoguys site.
I really like the RAID 5 option myself.
USB and Firewire are slower than eSATA. USB is the slowest, Firewire faster, Firewire 800 faster, but eSATA is very close to regular, internal SATA.
The RAID 0 for the temp area was also something I built over time due to budget constraints. My RAID 0 is a Venus T4s external box that comes with its own eSata controller card. It is easy to use and less money than other solutions. Probably not as fast as other more expensive solutions, but it works fine for the temp area.
The problem I was having with the RAID controller has to do with Option ROM. Your motherboard has a BIOS that basically gets the PC running by recognizing hard drives, CD-ROM, RAM, etc. One of its duties is to find a boot device. Any hardware device that can be bootable puts some code into the Option ROM area. This is an area of memory that was setup way back in the days of DOS and is fairly small.
Well, after installing the RAID controller, my system wouldn't boot because the Option ROM area was full and the BIOS couldn't find my mirrored drives to boot from. After doing a lot of troubleshoting and figuring out what the problem was, I basically had to disable as many bootable devices as I could in the BIOS and turn off the bootable feature in the new RAID card. This got me working again, but it took me many hours over about 4 days to figure this out.
As far as the IRQ and DMA settings you mentioned, motherboards today can usually handle that on their own. I have not had to specify IRQs for years.
I will think about the "walk-through" procedure. It would take some time to create it.
thanx for your thoughts Jeff
If you can start a logic matrix for the DIY series, i'm sure it would be helpful and topical.
Maybe it could be a series that starts with the mother-board and all it's considerations related to OS/NLE and other supportive hardware.
Then it continues with the perspective of another piece of hardware such as the graphic card or raid controller... and so on.
Maybe someone could contribute a graphic or mindmap rendering of the logic. This way at a glance, people could find where they fit, with what they have to spend and what they expect out of it.
The DIY folk, from my reading seem to have done alot of work, and put alot of thought into establishing benchmark systems. The next step would be to create assembly instructions for those specific systems and systems in general.
I look forward to this being a new topic. In good time.