Son Of Leonid Meteor Video Digitization
by Leonid Don Quixote
on
Dec 18, 2006 at 11:52:55 pm
Nearly a year ago, I bumbled into this forum asking absurd questions about how to digitize some NTSC video from a NASA airborne mission to study the Leonid meteor storm in 1999. I am a volunteer working with Dr. Peter Jenniskens of the NASA SETI Institute on the project. We've been wanting to digitize this stuff for years, but Peter could never get NASA to properly fund such an effort. I was poking around trying to find out just how cheaply it could be done. You fellows gave me some excellent (albeit disheartening) pointers and I followed up on the leads you suggested, trying all manner of possibilities. Alas, nothing seemed to work.
Well, I'm back. Over the last year I have decided that those old tapes are degrading and something MUST be done to digitize them. I am short on both money and time because I've been working at a startup. But I decided that I had to make it work myself, and so I tightened my belt, scraped together my pennies and purchased the following equipment:
1. Aurora Igniter-X board.
2. Macintosh dual 867MHz G4s, 133 MHz bus, 1 GB RAM
3. two external LaCie 256 GB FireWire drives in a RAID0 (striped) array; max sustained transfer rate per drive: 41 MB/s
I am using QuickTime to record the data -- since I don't do any editing, there's no need for anything better (I think).
I set up everything for uncompressed storage and tried it out. It didn't work -- records only 18 fps. Curses! I went back and reviewed the specifications in the IgniterX Manual. The IgniterX is outputting data at 28 MB/s, and the manual states that I need sustained data rates of at least 30 MB/s. Firewire can support 40 MB/s, and the two drives are each individually fast enough to handle the bandwidth. So what am I doing wrong? Here are my guesses:
A. QuickTime Pro?!?!?!? C'mon, Chris, get serious! Get proper software! (What would that be?)
B. FireWire 400?!?!?! Sure, they say 400 Mb/s, but if you believe that, I have a bridge to sell you. You have to use a SCSI or ATA drive system.
C. Your system should work fine. You just forgot to push the Magic Button. Do that and you'll be fine.
D. You're not even close.
Re: Son Of Leonid Meteor Video Digitization by John Pale on Dec 19, 2006 at 7:08:00 pm
[Leonid Don Quixote]"B. FireWire 400?!?!?! Sure, they say 400 Mb/s, but if you believe that, I have a bridge to sell you. You have to use a SCSI or ATA drive system."
This is probably the problem. Nobody should really be trying to edit uncompressed video with FW 400 drives, even in a RAID format. In fact, if you are using two separate units striped together in a homemade RAID, you are actually splitting the bandwith of the FW bus, thus slowing everything down even more.
You might be able to pull this off with FW 800 (Lacie Big Disk or G Tech G RAID), if you are determined to use FW. I think you will have to get a FW800 PCI card for that computer, as I dont think it has FW 800 ports.
Re: Son Of Leonid Meteor Video Digitization by Leonid Don Quixote on Dec 19, 2006 at 10:10:47 pm
Thanks for the explanation. I was a little suspicious of FireWire because it's a serial bus -- but I figured that Firewire 400 was still fast enough. Obviously not. So now I'm going to research all the alternatives. I'm back to square one, so if I'm going to a card and new drives, I can go with anything: Firewire 800, USB2, or one of the internal parallel busses. Any suggestions?
Re: Son Of Leonid Meteor Video Digitization by Leonid Don Quixote on Dec 20, 2006 at 12:51:18 am
OK, I have spent a few hours doing some background reading and I've figured out a lot:
1. I should kick myself for using a FireWire drive; I should have used the ATA-100 bus inside my machine.
2. The ATA-100 can support burst speeds of 100 MBytes/second. That's three times the speed required. However, when I actually use it, I record at only about 19 frames per second. Sheesh, what good are specs when they don't mean anything?
3. Obviously, what I need to do is get two more ATA-100 internal hard drives and plug them into the system, then put them into a striped RAID-0 array. That should be fast enough to do the job.
The question is: will two more ATA-100 internal hard drives be fast enough? Does anybody know?
Re: Son Of Leonid Meteor Video Digitization by Cofe on Dec 20, 2006 at 1:53:57 am
I don't know about 3 but 4 works.
I have a G4 1.42DP and have all 4 internal drive bays loaded with 80GB drives (Seagate) striped as one Raid 0
System drive is an external firewire drive on the 400 bus.
works fine for uncompressed.
But why do you actualy try to work uncompressed?
DV is a very efficient codec and if you do not intend to go to broadcast well suitable to be converted to web based codecs like mpeg 4 or H264.
I think the Igniter can do DV on OSX and the firewire drives are good enough to do DV.
Re: Son Of Leonid Meteor Video Digitization by Leonid Don Quixote on Dec 20, 2006 at 5:35:24 pm
My reason for requiring uncompressed is that the video was taken of the sky through image intensifier tubes during the Leonid meteor storm. The video itself is not very impressive -- image intensifier tubes produce simple green-and-black output and they're a bit fuzzy. But the meteors we're looking for are often very small and transient -- exactly the kind of thing that compression algorithms erase. We've experimented with a variety of compression algorithms and we know that they throw away at least 30% of the meteors on the video.
BTW, the video itself is quite impressive at the peak. Sometimes there are as many as five meteors visible in a single frame.
Re: Son Of Leonid Meteor Video Digitization by Chaos Wrangler on Dec 22, 2006 at 1:56:41 am
I would avoid DV and go with uncompressed, I have not seen the footage but i assume a high contrast ratio and fine details and DV might sawtooth that up a bit. Try some test with DVCPro50 (if it will allow you to capture that setting) for a lower bit rate. For drives I believe that you can put 4 drives in your G4 (make sure they are all the same model, check with dealmac.com for good prices) and combine them in a 0Raid. if you only have 2 drives make sure they are on different busses and Raid them together. If you make the 4 drive raid put your system disk in the firewire 400 case.
Re: Son Of Leonid Meteor Video Digitization by John Pale on Dec 22, 2006 at 4:09:59 am
Hey there...
I just looked up your original post about this project.....you have 80 hours of footage to digitize...if you want to do this uncompressed you will need 6.1 Terabytes of drive space.
One option would be to capture smaller amounts, output to Digital Betacam (or another mildly compressed format----NOT DV or DVD). Wipe the drives, then capture more footage.
Re: Son Of Leonid Meteor Video Digitization by doka15 on Dec 22, 2006 at 8:29:17 am
Just tossing in my 2 cents now. I have the Pipe Pro card and do uncompressed all the time and I have nver had a problem. I installed an SATA PCI card (2 port) and purchased a 2 drive external raid set up with 2- 250 gig drivess. I bought it from these folks http://eshop.macsales.com 18 months ago for I think $500. Prices have dropped big time and that same set up is $350
Made them Raid 0 and I have never had a problem running uncompressed.
Re: Son Of Leonid Meteor Video Digitization by Edward G Downie on Dec 24, 2006 at 2:18:57 pm
I am not sure if the Igniter X board can capture dv50 but if it can that is the best way to go if not use uncompress. For the drives get a serial ata card from sonnet technology and their fusion 5 storage case and use five hitachi 7K500gb sata drive or western digital 500gb drives the 3gbs / 16 mb buffer ones, and when you are capturing you use selective capturing only capture what you need and all will hold on 2.5 tb. and will cost you a little over a thousand dollors. I am not sure if pipe pro card is available any more but you can buy one or from black magic or aja good luck.
Re: Son Of Leonid Meteor Video Digitization by Leonid Don Quixote on Dec 29, 2006 at 1:02:27 am
Thanks all for the many useful comments. I today received a new ATA-100 internal drive, which I installed and set up in a format very similar to one described earlier: the System Drive is on FireWire, while the two internal ATA-100 drives are installed as a RAID-0 system (striped). I recorded my video and, sad to say, got only 20 frames/second recording while uncompressed. There are several possible routes to pursue at this point:
1. Buy some more internal drives. The Mac can take two more ATA-60 internal drives. They're not as fast as the ATA-100 drives, so I fear that hooking two ATA-60s into a RAID-0 with two ATA-100s will just result in a slower system. This is not my favorite option just now.
2. Perhaps I can change the capture depth. Right now it's recording 1.27 MB per frame. Since a frame holds 307 Kpixels, that's just over 4 bytes/pixel. Now, if it were ten bits deep in each of three channels, that would explain the size of the file. If it's 8 bits deep on each pixel, then we get only 3 bytes/channel -- but as I recall, many video systems use 4 bytes per pixel to fit the databus width. So, is there any possibility that I can tell the IgniterX board to record only 8 bites deep? Could I even tell it to record just grayscale? (This is output from an image intensifier, which is green-and-white anyway.)
I have gone through the Ignition control panel in the System Preferences and it does not appear to offer me any choices regarding bit depth, or even the option to ignore sound input. The manual suggests that this should be possible, but I cannot find out how these factors are controlled.
BTW, the Ignition Preferences panel informs me that my board does NOT have the Pro option and does NOT have the SDI option, but it DOES have uncompressed. The driver is 6.4.3 (the 6.4.4 file would not unzip -- the unzipper claimed the file was corrupted.)
Re: Son Of Leonid Meteor Video Digitization by Leonid Don Quixote on Dec 29, 2006 at 3:01:25 am
I just carried out an interesting experiment. I set up QuickTime to record under four different arrangements, getting these results:
ATA-100 drive alone: 16.6 frames/second
FireWire drive alone: 17.0 frames/second 19MB/second
two ATA-100 in striped RAID: 20.2 frames/second
two ATA-100 + FireWire in striped RAID: 20.3 frames per second
Re: Son Of Leonid Meteor Video Digitization by doka15 on Dec 30, 2006 at 7:26:06 am
I would listen to Davids idea and this is what I do as well. Buy an external enclosure and an SATA pci card and do a raid 0 on that. Don't use the internal bus.
What drives did you buy? Also when you record are you using the Aurora uncompressed codec? I use a G5 dual 2.5 with external SATA and do uncompressed 10 bit with no problems. I record 30 minute programs with no drop frames. The drives I use are Segate 250 GIG sata.
Re: Son Of Leonid Meteor Video Digitization by Leonid Don Quixote on Dec 30, 2006 at 6:47:41 pm
Thanks much for the advice. There's a snag, though: this is a volunteer project. Dr. Jenniskens can't pry any money out of NASA to fund it, so I'm doing the whole thing out of my own pocket -- and since I'm working in a startup, I don't have that much money to play with. Indeed, I have already spent about $1000 on the entire setup and my wife is complaining bitterly about the expenditure. So I'd really like to come up with something that won't strain marital relations any further. I realize that it might be simply impossible, but I was hoping that by accepting four-year old technology, I could end up with a workable system.
The IgniterX manual recommends a system using the internal bus and at least two -- perhaps four -- internal drives in a RAID-0. I already have two drives on the ATA-100 bus, and so I see the following as the cheapest solutions:
1. Add two more internal drives on the ATA-67 bus to make a four-drive RAID-0.
2. Figure out how to tell the IgniterX board to encode only 8 bits per channel instead of 10 bits per channel.
You ask if I am using the IgniterX uncompressed codec. I believe I am; the files coming out are certainly big enough to be uncompressed (1.2 MB per frame). Moreover, my own analysis of the luminance distributions shows no compression artifacts.
The biggest unknown for me right now is the method of controlling the bit depth of the digitization. The manual clearly indicates that there are two choices here, but I can't find any indication of how that choice is made. Anybody have any idea how that's done?
Re: Son Of Leonid Meteor Video Digitization by David Battistella on Dec 30, 2006 at 9:47:39 pm
Leonid.
I have a proposal. In re-reading this thread you have a steep learning curve and your system is severly underpowered for the knoble task you are undertaking.
I went back into the e-mails we exchanged last year.
You need about 6TB to fit the 80 hrs of footage.
You can get about 5hrs on the raid you are trying to build.
It will take about 1333 DVD-R's to store all of your data uncompressed, so DVD's are not an option as I imagine you do not have the time to make all of these DVD's.
There is a way to make this cost effective and usefull.
1. Capture everything 8-bit uncompressed to firewire 800 drives. You will need seven of Lacie Big Disks to accomplish this (can't fill them to the rim).
2. Burn DVD's as needed but watch you quicktimes from these drives and eventually keep a backup of the material on a mirror set of drives.
3. Make D-beta tapes 94minutes at a time as another option to preserve the material.
Archiving is serious business. I think you should look for a philanthropist instead of trying to figure out this end of the technology. I will volunteer to get it all converted and I will even try to help you find the cash. Your wife will be a lot happier too.
Re: Son Of Leonid Meteor Video Digitization by Leonid Don Quixote on Dec 30, 2006 at 11:27:43 pm
Thanks so much for the help; I really appreciate it. I failed to include two crucial facts in my previous explanations:
1. We don't need a lot of storage because we'll be working in small batches. We'd like to digitize about an hour at a time and then process and save onto (believe it or not) CDs. We decided on CDs because they have the lowest dollar cost per bit. It's our hope that, if we can present the bigwigs with functional hardware and tell them "All we need is a few thousand CDs", they'll spring for them out of loose money. Moreover, we want to save them, not as QuickTime files, but as individual png files. We believe that this will provide future researchers with the greatest flexibility in terms of analyzing the data. I will write the software that will split out the Quicktime files into individual frames (unless you know of something that already does that.)
2. While we're poor in money, we, like most other research groups, are rich in volunteer time. We have half a dozen guys who are happy to babysit the computer and grind out the CDs. The plan is that, once I get something working, I transfer it to the NASA SETI group in Mountain View, CA, and then the volunteers babysit the computer day in and day out for months getting all the data archived. I know it sounds absurd, but science progresses largely on the backs of graduate students and volunteers.
Re: Son Of Leonid Meteor Video Digitization by Leonid Don Quixote on Jan 2, 2007 at 5:58:37 am
Ta-da! That was the solution! QuickTime was the time bottleneck, not the hard drives. I digitized with MediaGrab, and it worked perfectly! Thank you, thank you, thank you! I have already digitized an hour of Leonid video and shot off an email to Dr. Jenniskens advising him of my success.
Lots of work remains. First we have to come up with our archiving medium. We had been planning on using several thousand CDs because their cost per bit was the lowest, but I just checked and, sure enough, the price of blank DVDs has fallen far enough that they're now the cheaper medium. One of my colleagues convinced me that we should store in QuickTime format rather than individual pngs, so that will save us a step. There is one small problem: a DVD holds 4.7 GB of data, which translates to 3 minutes and about 50 seconds of video. This would lead to a clumsy archive structure, so I'm trying to find ways to trim the video down so that I can fit 4 minutes of video onto each DVD. That yields 15 DVDs per hour, or a mere 1200 DVDs to hold the main 80 hours of data.
There's also lots of work to write the data extraction software, the statistical analysis software, and so forth. But we're on the road! Thanks for the help, everyone! If you wish, I can upload a few seconds of the Leonid video so that you can see what this is all about.
Glad MediaGrabX worked! by John Pale on Jan 2, 2007 at 7:26:57 pm
I never actually used it, being an FCP user...but I remembered that it existed. Using Quicktime Pro to capture is a very new feature that was added long after the Igniter was discontinued by Aurora. MediaGrab was made to be used with the Igniter, and I suspected it might work better.
You might want to look at Dual Layer DVD, which hold almost double the data as a standard DVD. The prices of those are dropping too.
If you still want to go the PNG route, all you need is Quicktime Pro, which you already own.
Open the movie you digitized.
Select File/Export
Then in the dialog window select Export: Movie to Image Sequence
Click Options--then select PNG. There is another options button in this window. You may want to experiment with different settings here or just use the defaults, which seem to give good results on most things.
I think using PNG image sequences, you will save space on the DVD's. PNG uses lossless compression, so it should not adversely affect image quality..
Thanks for the suggestion. I didn't realize that Quicktime provides instant breakdown into individual frames -- I thought I'd have to write my own code to do that.
Re: you may find this program useful, too by Leonid Don Quixote on Jan 4, 2007 at 10:09:46 am
Thanks for the pointer. The program looks interesting but is probably more than I need. I have to build an archive format that will be extremely easy for scientists to access a few decades from now, and so I want something very simple and common. We planned to archive it as individual frames, but there's one last argument in favor of keeping it in QuickTime format -- I have to do some research to make sure that we're right on this point.
DVD-Rs fail after a few years by Paul Provost on Jan 28, 2007 at 6:31:40 pm
Carefull with your archives! I own a post production/DVD authoring company, and we are finding DVD-R masters created as recently as 2 years ago that have deteriorated to the point of not being able to access the data.
there are digital archive master (DAM) DVD-R available that supposedly last longer but cost 2 to 3 times as much.
do some research, but I think DLT tape is considered more stable. problem is, who will be able to play those in 20 years?
