FORUMS: list search recent posts

Real uncompressed, useable codec - please help

COW Forums : AJA Video Systems

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Andras Sarkadi
Real uncompressed, useable codec - please help
on Mar 25, 2009 at 2:36:42 am

Dear AJa users,

I'm quite new to the Aja world, and i'm trying to look for the right codec to use in our post production studio. I've done some tests, but none of the codecs i have seemed to be right for the job.
Here's what I'm looking for:

Uncompressed - really uncompressed. not something that looks like one. we do lots of compositing, color grading, effects and 3D. i cannot lose image details in the workflow.
Realtime playable - HD and SD, no need to render.
8 bit is usually enough for us - 10 bit would be a nice and useful option.

We get our footage from Digibeta and HD/HDSR in TGA or DPX sequences, (captured on different machines) RED footage and Canon H264 quicktime movies. I really don't mind converting all these sources into one format as long as the format works.

The machine I use is a 2.66 QuadCore MacPro connected to an Xserve SAN (so bandwidth can not be a problem), Kona 3 with 6.0.2 drivers, quicktime 7.6, latest FCS.

I got the best results from Apple Uncompressed 10bit, but it doesn't always play back realtime.
Coming from a DPS/Leitch Velocity background, where all of these requirements were natural features of the system it's really hard to imagine that this is not possible.

Please help with ideas. I'm slowly getting angry. (Clients get annoyed much faster)


thanks for your time

Andras Sarkadi


Return to posts index

Dino Sanacory
Re: Real uncompressed, useable codec - please help
on Mar 25, 2009 at 9:43:39 am

[Andras Sarkadi] "The machine I use is a 2.66 QuadCore MacPro connected to an Xserve SAN (so bandwidth can not be a problem)"

Bandwidth can absolutely be a problem. You make no mention of the storage size or type, number or users or how it is all connected. If you are using Apple Xserve RAIDs, uncompressed HD requires practically the full bandwidth of both sides of a unit. Your particular configuration and the number of users hitting the storage could very easily cause playback hiccups.




Return to posts index

Andras Sarkadi
Re: Real uncompressed, useable codec - please help
on Mar 25, 2009 at 12:13:40 pm

The SAN is connected to two computers through fiber channel. We tested the bandwidth and it's around 2 Gbytes/sec with two machines reading at the same time. I feel that should be enough. What do you think?
Andras


Return to posts index


Jeremy Garchow
Re: Real uncompressed, useable codec - please help
on Mar 25, 2009 at 1:48:58 pm

ProResHQ isn't doing it for you?

If not, then your next best options are Uncompressed 8bit or 10bit. You won't get real time effects as FCP is not an Uncompressed workhorse like other hardware based systems, and please forgive me for being a bit frank, but you should get used to that fact. On 2GB/sec you should be getting realtime playback of uncompressed HD and then some. I can playback UNcompressed HD on my raid that gets about 400MB/sec and used to do it on 180MB/sec fiber raid years ago. If you aren't getting real time playback, I'd say something isn't quite setup correctly in your system, your SAN is bottlenecking somewhere, or perhaps it's as simple as not having your footage, timeline and output settings all match.

What have you tried so far, and how did you get to that point?

Jeremy


Return to posts index

gary adcock
Re: Real uncompressed, useable codec - please help
on Mar 25, 2009 at 2:02:52 pm

[Andras Sarkadi] " We tested the bandwidth and it's around 2 Gbytes/sec with two machines reading at the same time. I feel that should be enough"

WHAT?? 2 GIGABYTES a second?
No, not on any xsan I have ever heard of.

it took me 54 spinning disks in 2 arrays aggregated across 2-4gig fibre connections to be able to handle 1.25 GBps for a single user, (that is UC 4K at 24p)

I have an idea about how much it takes to be able to handle data at that rate and it is not on any Xsan I have ever seen nor heard about.

I think you need to recheck your through put,



gary adcock
Studio37
HD & Film Consultation
Post and Production Workflows

Inside look at the IoHD
http://library.creativecow.net/articles/adcock_gary/AJAIOHD.php




Return to posts index

gary adcock
Re: Real uncompressed, useable codec - please help
on Mar 25, 2009 at 7:00:07 pm

[Andras Sarkadi] "We tested the bandwidth and it's around 2 Gbytes/sec with two machines reading at the same time."

Andras,

It was impolite of me to comment without offering some information on your problem.

http://www.aja.com/ajashare/AJA_System_Test_v601.zip

lets start with how fast your storage really is, this app can tell you how fast your system really is,
Setup using 512MB or 1GB for the file size like this.




gary adcock
Studio37
HD & Film Consultation
Post and Production Workflows

Inside look at the IoHD
http://library.creativecow.net/articles/adcock_gary/AJAIOHD.php




Return to posts index


Arnie Schlissel
Re: Real uncompressed, useable codec - please help
on Mar 28, 2009 at 8:56:08 pm

[Andras Sarkadi] " We tested the bandwidth and it's around 2 Gbytes/sec with two machines reading at the same time. I feel that should be enough. What do you think?"

That's 2 gigabits per second. Nothing on earth delivers 2 gigabytes per second.

Arnie

Post production is not an afterthought!
http://www.arniepix.com/


Return to posts index

gary adcock
Re: Real uncompressed, useable codec - please help
on Mar 25, 2009 at 1:00:50 pm

[Andras Sarkadi] "The machine I use is a 2.66 QuadCore MacPro connected to an Xserve SAN (so bandwidth can not be a problem), Kona 3 with 6.0.2 drivers, quicktime 7.6, latest FCS.
"


[Andras Sarkadi] "I got the best results from Apple Uncompressed 10bit, but it doesn't always play back realtime. "

if your content cannot playback your storage is NOT FAST ENOUGH.

not too many people do 10bit UC on an Xsan. ( like damn near NO ONE)

it is a storage issue. 10bit UC plays fine on my systems- and is easier to handle than frame seq.

I use ProRes for all of my editing and captures now and I am not looking back.



gary adcock
Studio37
HD & Film Consultation
Post and Production Workflows

Inside look at the IoHD
http://library.creativecow.net/articles/adcock_gary/AJAIOHD.php




Return to posts index

Andras Sarkadi
Re: Real uncompressed, useable codec - please help
on Mar 25, 2009 at 11:50:17 pm

Dear Gary,
you were absolutely right, I stand humbled. I'm not getting 2Gbytes/sec. I get around 180-190 Mbytes. I ran the wrong test i guess...
So now I'm going to retest the codecs i have with that result in mind...
Jeremy, you also said ProRes would be my best bet? And yes, I am getting used to the fact that FCP+AJA is something different. :) I won't mind if i can have predictable, good-looking results after the images go through a complete workflow.
Have any of you tried Sheervideo codec? I think I will test it as well, but would love to hear your opinions about it.
Ramona thanks for the Rave idea, but I don't think that my studio could afford something like that after getting the SAN :)

thanks for your time and help
Andras


Return to posts index


Jeremy Garchow
Re: Real uncompressed, useable codec - please help
on Mar 26, 2009 at 1:00:13 am

Sheer is nice, but again, no rt in FCP.

Try ProRes or ProResHQ. I think you will be surprised. As Gary will tell you, use ProResHQ from 10bit or higher sources when doing software encodes.

180MB/sec won't be fast enough for two seats doing Uncompressed 1080, but it should be fine for ProRes and HQ.

ProRes and HQ are both compressed 10bit codecs and hold up very well to multiple generations of rendering.

Jeremy


Return to posts index

Andras Sarkadi
Re: Real uncompressed, useable codec - please help
on Mar 26, 2009 at 1:22:17 am

Thanks Jeremy, i'm testing the ProRes way right now, but I already have something strange happening. I put a dpx sequence to After Effects, rendered it out in Uncompressed 8bit, Prores HQ, and Prores HQ with 4:4:4 Chroma filtering enabled (not quite sure what it means, but why not try.)
I imported everything into FCP, and they all look similar and good, me happy. But opening the same files in AE and looking at them at the video monitor shows different results. The uncompressed looks similar to the original, but the both Prores videos look much lighter, like it had a gamma correction applied to it.
what could be causing this?
Andras


Return to posts index

Jeremy Garchow
Re: Real uncompressed, useable codec - please help
on Mar 26, 2009 at 2:22:33 am

Old version of the Kona drivers. This is a problem with AE, but AJA has found a way around it. If you download and install the latest Kona (v6.0.3) driver, this will fix it. If you don't feel like doing that, quit everything, go to your LIbrary > QUicktime folder and remove the AJAUncompressedcodec.component from the Quicktime folder. Start AE and rerender. This will cure the gamma shift, but you will lose access to any legacy AJA Uncompressed movies (not Apple Uncompressed, different things).

The new driver is easier.

Jeremy


Return to posts index


Andras Sarkadi
Re: Real uncompressed, useable codec - please help
on Mar 26, 2009 at 2:33:48 am

Thanks Jeremy, that sounds great, first thing to try tomorrow morning.
As noone mentioned any codec that could be useful and its name is starting with AJA, i have to ask why?

Andras


Return to posts index

Jeremy Garchow
Re: Real uncompressed, useable codec - please help
on Mar 26, 2009 at 3:20:56 am

I think it has to do with the AJA Uncompressed codec filling a gap a long time ago before FCP had a native Uncompressed HD codec.

The AJA Uncompressed codec correctly set the color space to REC709 when the frame size went above SD, while Apple's uncompressed did not. Apple eventually fixed it and now the AJA and Apple codec are pretty much the same, except for the name.

These days, it is not necessary in 4:2:2 uncompressed HD.

Hope that helps.

Jeremy


Return to posts index

gary adcock
Re: Real uncompressed, useable codec - please help
on Mar 26, 2009 at 12:21:54 pm

[Jeremy Garchow] "180MB/sec won't be fast enough for two seats doing Uncompressed 1080, but it should be fine for ProRes and HQ."

True
that is barely fast enough to do a single stream of 10bit UC, and the xsan controller would limit that if 2 machines were on the server at the same time to some thing considerably less -think 1/2 the throughput to start or about the same bandwidth you get with 2 raided drives over Sata.





gary adcock
Studio37
HD & Film Consultation
Post and Production Workflows

Inside look at the IoHD
http://library.creativecow.net/articles/adcock_gary/AJAIOHD.php




Return to posts index


gary adcock
Re: Real uncompressed, useable codec - please help
on Mar 26, 2009 at 12:38:26 pm

[Andras Sarkadi] " I ran the wrong test i guess..."

I think your Tech/ IT guy told you that you had a 2Gig Fibre connection- and did not explain that gigabit is not gigabytes of data.

Andras, in corporate IT worlds, some guys think they know everything about drives and storage- and they apply their boxed solutions to our storage problems.

I have had dozens of battles with IT guys about the reality of handling video on a network.
I tell IT people that I need a minimum of 6 disks per user for compressed HD, and 10 if doing UCHD on an xsan system increase that by one additional controller for every 4 users and increase the disks by 50% due to the overhead required.

I do not ever recommend editing in 4:4:4 or 2K on a shared volume unless someone really really understands the needs.

YES HD can be done on less, but over a network, I want something that will work reliably.

gary adcock
Studio37
HD & Film Consultation
Post and Production Workflows

Inside look at the IoHD
http://library.creativecow.net/articles/adcock_gary/AJAIOHD.php




Return to posts index

Andras Sarkadi
Re: Real uncompressed, useable codec - please help
on Mar 26, 2009 at 11:18:40 pm

Gary, i also have bad stories about IT guys, but this time it was really my fault... thanks for steering me to the righ tool to measure disk performance. I also take a note of what you've written above about estimating disk needs.
Jeremy, I upgraded to the 6.0.3 Kona drivers, but I still had the problem. After that I removed the AJAUncompressed codec.component file, but I still have the color shift...
I uploaded a file with jpg exports of two versions I have now. Both versions were converted in AE from a TGA sequence. The uncompressed version matches the source perfectly.
If you have the time, please check it out:
http://www.umbrella.hu/uncomp.vs.prores.jpg

thanks for all your help
Andras


Return to posts index

gary adcock
Re: Real uncompressed, useable codec - please help
on Mar 27, 2009 at 1:14:39 pm

[Andras Sarkadi] " Both versions were converted in AE from a TGA sequence. The uncompressed version matches the source perfectly. "

OK

What is your workflow and how are you doing this? Where are the targas files from?

Where and how you do the conversion from RGB to YUV makes a huge difference.



gary adcock
Studio37
HD & Film Consultation
Post and Production Workflows

Inside look at the IoHD
http://library.creativecow.net/articles/adcock_gary/AJAIOHD.php




Return to posts index

Andras Sarkadi
Re: Real uncompressed, useable codec - please help
on Mar 27, 2009 at 4:45:30 pm

Gary, the TGA sequence originates from a DPX sequence which comes from a Baselight grading session. I didn't really trust AE to do the DPX-TGA conversion, so I did it in Digital Fusion. (I know, bringing in another software rarely helps, but I wanted a TGA sequence I can trust). After the conversion, the DPX and the TGA sequence looks the same.
I am reading right now, that switching the "Match Legacy After Effects QuickTime Gamma Adjustments" in AE might help, so I'm going to do a test with that. I work with AE CS4 by the way.

thanks again
Andras


Return to posts index

gary adcock
Re: Real uncompressed, useable codec - please help
on Mar 27, 2009 at 5:01:54 pm

[Andras Sarkadi] "TGA sequence originates from a DPX sequence which comes from a Baselight grading session"

Ok first question

why are you not working with the DPX originals? I do this on a daily basis using the Gluetools Plugins with FCP.

Andras you seem to be doing an incredible amount of jumping thru hoops when something like gluetools will allow you to handle the files as graded in baselight.

gary adcock
Studio37
HD & Film Consultation
Post and Production Workflows

Inside look at the IoHD
http://library.creativecow.net/articles/adcock_gary/AJAIOHD.php




Return to posts index

Andras Sarkadi
Re: Real uncompressed, useable codec - please help
on Mar 27, 2009 at 6:55:42 pm

Gary, I'm pretty new to the mac-fcp-aja world. I had no idea i would run into problems like that. And i never wanted to work with a dpx sequence in this project. There are so many other unknown factors about using aja and fcp for me, that I wanted follow the route i know and trust, and that was 1. to have a sequence that matches the grading, but in a format i know, trust and can use in AE, Nuke, whatever. 2. make a movie out of that to use in fcp. Two weeks ago that was the idea of a workflow in the given situation.
If I feed the DPX files to AE and convert them to ProRes, I have the same problem...

Andras


Return to posts index

Ramona Howard
Re: Real uncompressed, useable codec - please help
on Mar 27, 2009 at 10:37:25 pm

Andras,

Please forgive as I have not read every word in the conversations back and forth between everyone. Also if this covers anything you already know. I am simply going to point out a few things that may or may not have come up and or you and or others don't know. Use what you can. Hope it helps.

Oh I remember the quicktime days when we did Sin City, yes it was rather frustrating....

DPX is really straight forward once you know the rules and once you know how things interact. It sometimes gets a bad rap, when it really isn't the true fault of what is happening...

Within a DPX you can pack anything, any raster, and bit depth, any colorspace, log, lin, full range or SMPTE range, etc.....

In addition you can pack metadata in both the header and user space of the header. Within the header there are defined SMPTE standards of packing timecode, reel, keycode, etc. The user space is a free for all and almost anything can be put here. We pack audio chunks and other metadata that the Rave reads and writes. When done properly anything packed in user space DOES NOT EFFECT if a DPX can be used in anything else.

This is why DPX is so prominent in the high-end workflows, it is fairly versatile.

Here are some items that may be worth checking out to see where things are going wrong for you :)

Start with your original files, what colorspace and range are they? When converting to DPX, make sure you are converting to the same. ie, if it is 0-1024, make sure the DPX is the same, log to log, etc....

This way the conversion from your original files won't be the issue and they should match bit for bit, if they don't, here lies the issue. If your using shake you should be able to open both and compare.

Next.

Are the original files 4:4:4? is the DPX sequence being converted to 4:4:4? and is the Pro-Res 4:4:4?

Next.

Find out what range or format the ProRes works in (Full or SMPTE) and that it matches the DPX files

i.e, most products default to a SMPTE range for these types of codecs and or can't support full range. So if you start with a full range DPX and convert to something else, they will not match. Same goes for 4:4:4 to 4:2:2. Most tape decks, monitors, etc do not support full range and or 4:4:4!

So if this isn't the issue, check.

Some products have issues with sharing DPX files between MACs and PCs and reading the header info correctly. This may require you to do some digging (maybe someone here can answer) if this infact is part of the problem. We support both big indian and little indian in Rave, so we don't care where the files originate from and we don't see this issue but I have know other products that have. It is an old problem and surprises me to still see programs with this issue.

Last, call AJA, they will bend over backwards to help you get to the bottom of things if they can. The AJA hardware has settings that support pretty much all scenarios and you will see issues if you have the wrong one selected.


You may be asking why in the heck would I start with a sequence that is out of SMPTE range to begin with? Because, in the film-world we do it everyday, as film does not have the limitation that video does. The workflows that use full-range are also using LUTS (Look Up Tables) to do a remapping and or colorspace conversion to go out to TV/video and some monitoring. And, yes there can be good remapping and bad, so it will depend on where it is being done.

I hope that helps you get your head around the issue and the workflow, as honestly it isn't that difficult when it works, which BTW I see almost on a daily basis. Just gotta know the rules :)

Cheers,
Ramona
http://www.spectsoft.com










Return to posts index

Andras Sarkadi
Re: Real uncompressed, useable codec - please help
on Apr 1, 2009 at 2:50:49 am

Ramona, and Everyone reading and writing
I am not abandoning the topic, sorry for the long pause, just had to finish a couple of projects in these days. Ramona thanks for the tons of information and questions, going to answer. and yes. I am learning the rules. :)
I just realized yesterday, that the other computer connected to our Xsan is reading 3x quicker than mine. around 650 Mbyte/sec. We are looking into that problem as well, will be interesting to see what causes it.
Andras


Return to posts index

Ramona Howard
Re: Real uncompressed, useable codec - please help
on Mar 25, 2009 at 5:57:51 pm

Andras,

Please take a look. We have been filling this exact need for 10 years.

http://www.spectsoft.com/index.html

Native DPX so you can render right to the unit, or drop files from other sources and they just play. Uncompressed is what this unit has always been about.

Cheers,
Ramona



Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]