ADOBE AFTER EFFECTS: Forum Expressions Tutorials Creative Cloud

Video performance requirements analysis?

COW Forums : Adobe After Effects

<< PREVIOUS   •   FAQ   •   VIEW ALL   •   PRINT   •   NEXT >>
Kendall ShawVideo performance requirements analysis?
by on Jan 3, 2015 at 8:42:10 pm

As a methodology that applies to videos in general, not one single video one time, how can one estimate whether or not their video for playback on computer will play smoothly on a roughly average computing platform, not over a network?

I know the "dumb" way, which would be to test each video in isolation, as if it were the only video I have ever seen, on a roughly average computing platform, never accumulating knowledge about what traits seem to have contributed to whether or not it plays back adequately.

Not that.

What are metrics that can be used to judge the efficiency of a video, other than video bandwidth?


Return to posts index

Brian CharlesRe: Video performance requirements analysis?
by on Jan 4, 2015 at 4:19:30 pm
Last Edited By Brian Charles on Jan 4, 2015 at 10:59:57 pm

[Kendall Shaw] "how can one estimate whether or not their video for playback on computer will play smoothly on a roughly average computing platform"

I don’t think there’s a single metric since many factors influence video performance, including but not limited to: video dimensions, codec, bit rate, potential hardware bottlenecks: GPU, RAM, bus and disk speed.



Return to posts index

Walter SoykaRe: Video performance requirements analysis?
by on Jan 4, 2015 at 4:43:51 pm

MPEG-2 and MPEG-4 attempt to address this question with levels and profiles. The idea there is that a device promises support for a specific profile and level, and any content encoded according to those specs should work.

http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Profiles

Walter Soyka
Designer & Mad Scientist at Keen Live [link]
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
@keenlive [twitter]   |   RenderBreak [blog]   |   Profile [LinkedIn]


Return to posts index


Kendall ShawRe: Video performance requirements analysis?
by on Jan 4, 2015 at 5:35:51 pm
Last Edited By Kendall Shaw on Jan 4, 2015 at 5:46:48 pm

A problem is that the target is not a device that a profile was created for, I think. It's a computer.

Also, I have seen varying performance using the same profile.

But, I hadn't thought of looking at what those profiles and media encoder profiles do. That probably is a good starting point to look for factors affecting performance.

[Brian Charles] "I don’t think there’s a simple metric since many factors influence video performance, including but not limited to: video dimensions, codec, bit rate, potential hardware bottlenecks: GPU, RAM, bus and disk speed."

Each of those is a metric. I want to come up with something more specific than each of those and figure out how you would emulate it though. For example, what is average disk speed and how much does it affect video playback performance, in isolation.

I would think that companies that produce a lot of video would have automated tests of some sort as part of QA, instead of relying entirely on people manually checking video to see if they think it looks right.


This is a made up scenario, it's not the motivation for my question: A different way to think about the factors is if one were asked at a job interview, how would you ensure that videos shipped as video files on CD-ROMs playback consistently for the average consumer? Presumably the applicant would not answer "you can't" or give some vague answer. Or, the applicant would want to demonstrate to the interviewer that they are aware of general methods to address common problems with video encoding.


Return to posts index

Walter SoykaRe: Video performance requirements analysis?
by on Jan 4, 2015 at 6:23:37 pm

[Kendall Shaw] "This is a made up scenario, it's not the motivation for my question: A different way to think about the factors is if one were asked at a job interview, how would you ensure that videos shipped as video files on CD-ROMs playback consistently for the average consumer? Presumably the applicant would not answer "you can't" or give some vague answer. Or, the applicant would want to demonstrate to the interviewer that they are aware of general methods to address common problems with video encoding."

Assuming average means they have a computer from the last five years (not a fair assumption if we're actually talking about CD-ROM drives), I'd answer that bit rate was the critical factor for CD-ROM delivery, because the access speed of the CD-ROM is likely be to the first limiting factor.

As long as you are sticking close to video standards, it's my experience that nearly any modern computer can play 1080p H.264 locally, and nearly any modern computer with a broadband connection can play streaming/progressive download 720p H.264. I stick with AME's Vimeo presets for client delivery with no problems.

What is the actual motivation for your question? Speaking practically and generally, I think that video encoding and playback is a solved problem in 2015.

This may be easier to discuss given a concrete scenario than it is in the general case.

Walter Soyka
Designer & Mad Scientist at Keen Live [link]
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
@keenlive [twitter]   |   RenderBreak [blog]   |   Profile [LinkedIn]


Return to posts index

Kendall ShawRe: Video performance requirements analysis?
by on Jan 4, 2015 at 7:22:12 pm

Typically the thread goes off the rails when specifics are brought up. But, the main motivation is that I observe that I automatically seeks a methodical solution to almost any task I am involved with, so it\'s there in my head nagging at me.

One of the nagging thoughts is how to make the process less time consuming when judging whether my after effects compositions are playing back correctly or not.

The most common problem that I deal with is that rendered video plays back in fits and starts. Sometimes uncompressed video plays back smoothly while using youtube 1080p profile mp4 in media encoder the video plays in fits and starts. Sometimes the opposite is true.

The same goes for video bitrate. Sometimes a video file with vastly higher video bitrate plays smoothly for me, while a lower bit rate video file plays in fits and starts. Sometimes the opposite is true.

If one has a faster computer, these become less apparent. But, that is no solution since the audience for the video is people who are not me and who don\'t have my computer.

So far, I have developed superstition about what affects video playback quality. Then when my routine fails I develop a new set of superstitions. I want that to stop and to know what I am doing instead.

I would also like to be able to capture knowledge of what factors affect playback quality into automated tests.


Return to posts index


Walter SoykaRe: Video performance requirements analysis?
by on Jan 6, 2015 at 10:25:57 pm

[Kendall Shaw] "So far, I have developed superstition about what affects video playback quality. Then when my routine fails I develop a new set of superstitions. I want that to stop and to know what I am doing instead. I would also like to be able to capture knowledge of what factors affect playback quality into automated tests."

I guess you need to consider all the ways that video playback can fail. I think there are a few major categories:

Insufficient bandwidth. Data cannot be read off the disk/over the network fast enough to keep the decoder's buffer full; frames are dropped when data is unavailable. This is often a problem with uncompressed media; although uncompressed is trivially easy to decode, it has massive bandwidth requirements and very few disk systems can actually keep up. Note that with poorly-constrained variable bit rate encoding, it's possible for the average bit rate to be within spec for your disk or network, but allow temporary bit rate spikes that exceed available bandwidth. This will result in dropped frames.

Insufficient compute availability. Decode for some codecs is extremely CPU-intensive. If the frames cannot be decoded at the rate by which they are needed, they will be dropped. Note that compression generally trades file size against encode and/or decode complexity; the smaller a file is (while maintaining quality), the more powerful CPU necessary for decode.

Bad data. A buggy encoder or a transmission error may cause invalid or missing data which forces the decoder to produce an error.

Display. If the application cannot push the rendered frames onto the GPU fast enough, you'll drop frames. If playback spans multiple independently-clocked monitors, you're likely to drop frames. If the GPU driver hiccups, you may drop frames.

The thing that complicates these when talking about a general purpose computer is that these situations may arise temporarily and without warning. If you're in a low-memory situation and the system starts swapping to disk, you will lose bandwidth unexpectedly. If you're running many applications simultaneously, you may have occasionally suffer reduced compute capacity.

If you want to geek out on the media file side of this, check out ffprobe. There's automated QC software from vendors like Tektronix, but it's pricey. Network shows go through QC at dedicated facilities. I watch my output because it goes out the door with my name on it.

Walter Soyka
Designer & Mad Scientist at Keen Live [link]
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
@keenlive [twitter]   |   RenderBreak [blog]   |   Profile [LinkedIn]


Return to posts index

Kendall ShawRe: Video performance requirements analysis?
by on Jan 6, 2015 at 11:42:21 pm

I want to figure out how to translate factors like those into judging the quality of a video. Video playback can be affected by CPU speed, for example. A corresponding test would be to determine somehow how CPU speed is affecting the playback of some video and what it is about the video that is being affected by the CPU speed.

The opposite is easier, presumably, given some video see if it's playback is improved by a faster CPU. This approach works to fix the playback of one video for one person, one time.

The point of all this is to have the video playback for other people with unknown computer configurations. When CPU speed affects playback it is a characteristic of the video that allows for CPU speed to affect the playback poorly. That characteristic is what I'm after, for each of however many factors there are.

I have used ffprobe/avprobe before, but aside from finding out what type of video I have, I haven't known what to do with the information for diagnosing a problem. The same goes for bitrate viewer. The documentation says it helps you to find and get rid of "bad stuff". The identifying of this "bad stuff" and knowing what to do about it is what I don't know and what I would have the software for in the first place.

You've given me what seems like 2 good ideas at least though, I can use the low level information that I won't understand coming out of ffprobe on different video files and compare them to see if I can start learning to recognize problems that way. The other is to see what the rationale for hte different mpeg profiles is and to look at the sample test case code for mpeg that's a public standard.


Return to posts index

Chris WrightRe: Video performance requirements analysis?
by on Jan 7, 2015 at 2:36:35 pm
Last Edited By Chris Wright on Jan 7, 2015 at 2:45:51 pm

what about jpeg2000? it supports adaptive playback with layers upon layers of data to increase or decrease the decode quality on the fly using a single file. lower quality just ignores the sub quality layers. Used from everything from
hospital imagery to Cinema 4K profiles DCP. You might not even need to re-encode!


http://www.3dtv.at/Products/Jpeg2000Decoder/Index_en.aspx


Return to posts index


Kendall ShawRe: Video performance requirements analysis?
by on Jan 7, 2015 at 3:59:29 pm

It certainly sounds good. My only exposure to JPEG2000 was when someone was using it to serve giant still images over the web. It's use would depend on the application of the video. So, it is not a general solution to performance problems, I think.

My question is really about video in general. Even in a scenario in which I could use JPEG2000, I would want to know how to address this problem:

Bob has a computer and will playback a video. Video version 1 and 2 are the same content. Version 1 has an unknown problem.

Video version 1: The JPEG2000 decoder adapts to "low quality" when Bob plays the video.
Video version 2: The JPEG2000 decoder adapts to "medium quality" when Bob plays the video.

I want to know how to figure out what the unknown problem is that leads the decoder to adapt to a lower quality in version 1 vs. version 2.


Return to posts index

Chris WrightRe: Video performance requirements analysis?
by on Jan 7, 2015 at 5:47:12 pm

you would need a video benchmarking tool which gives benchmark numbers on hundreds of items. another problem not discussed yet, for example, the same video played in quicktime would skip but in media player home classic, the 3rd party decoder works 170% more efficiently. Neither the hd, ram, proc, bandwidth, etc. or anything else would matter.
That's the real problem and why vimeo and quicktime suck. The poorly programed decoder...


Return to posts index

Kendall ShawRe: Video performance requirements analysis?
by on Jan 7, 2015 at 9:07:45 pm

Part of the question in the first post is more or less how does one do benchmarking. Beyond that how do you interpret the results and use the information to point out what you must change to correct the problems.

Different players rendering the video differently would also depend on hd ram etc. A player rendering the video more efficiently than another is a matter of how it performs in relation to those factors.


Return to posts index


Walter SoykaRe: Video performance requirements analysis?
by on Jan 7, 2015 at 9:16:36 pm

[Kendall Shaw] "Part of the question in the first post is more or less how does one do benchmarking. Beyond that how do you interpret the results and use the information to point out what you must change to correct the problems."

I think your insistence on a general solution to this problem makes it practically unsolveable. As Chris pointed out, I completely ignored the software stack in my basic list of failure modes. This adds dozes, hundreds, maybe thousands of variables.

I think it's much more fruitful to discuss the actual failure conditions you're seeing. Of the zillion possible failure modes, there probably only a few practical ones, and addressing those will cover very nearly all playback cases.

Walter Soyka
Designer & Mad Scientist at Keen Live [link]
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
@keenlive [twitter]   |   RenderBreak [blog]   |   Profile [LinkedIn]


Return to posts index

Kendall ShawRe: Video performance requirements analysis?
by on Jan 8, 2015 at 12:10:24 am
Last Edited By Kendall Shaw on Jan 8, 2015 at 12:17:46 am

I don't usually think of problems in terms of being solvable or unsolvable. In this case, when I talk about knowing what I am doing, I don't mean total awareness. I don't think that ever happens. I mean knowing what I am doing to some subjectively significant degree.

The choice is not between blindly dealing with each crisis in isolation vs. solving all possible problems. I am seeking a trajectory that is in the problem solving direction.

There is a fundamental difference between the two approaches. In some contexts the difference is obvious, more and more it is less obvious in relation to software because the world largely switched from learning concepts to imitating procedures around the time of y2k.

It's expected that a doctor will know some signs of a heart attack. The analogy would be knowing some signs of problem video.

It's not expected that the doctor knows all possible signs of a heart attack. Similarly, it's not reasonable to expect to know all possible factors that can result in problem video.

If the doctor can only deal with individual cases in isolation, it doesn't work well. The patient is grasping at his chest, let's try slapping him on the back. Oh, that didn't work. Let's try submerging him in an ice bath, etc. etc. The number of times this results in a dead patient might suggest that this approach is less than optimal. It could lead people to seek general knowledge that can make one prepared to deal with situations that have not yet occurred in a way that is practical.

The switch from concepts to imitation can be seen in technical documentation which used to be reference material but is almost entirely random lists of procedures to imitate in order to perform some task.

For each case in isolation, it requires less effort to follow the procedure than to learn a concept. In other areas outside of the software industry the problems with this idea are too obvious to even be debatable. We teach 4 year old kids the value of learning abstractions.

There are some related aspects to the trend that are good. Extremes in either direction are not practical.


Return to posts index

Walter SoykaRe: Video performance requirements analysis?
by on Jan 8, 2015 at 12:22:33 am

If you'd like to learn the concepts underpinning compression, there are a number of books that cover the area in depth. A quick Amazon search on "video compression" will give you a number to choose from. I read Ben Waggoner's book a long time ago; it should still cover the basics, but newer books would have better coverage of the current landscape.

Walter Soyka
Designer & Mad Scientist at Keen Live [link]
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
@keenlive [twitter]   |   RenderBreak [blog]   |   Profile [LinkedIn]


Return to posts index


Kendall ShawRe: Video performance requirements analysis?
by on Jan 8, 2015 at 2:20:35 am

Thanks. I just glanced at it (the ben wagonner compression book) on safari books and it looks very good.


Return to posts index

Ivan MylesRe: Video performance requirements analysis?
by on Jan 8, 2015 at 6:22:57 am

Just getting caught up on this thread and I think it wound up in a good place. Ultimately, the best way to understand how different variables affect the end result is to 1) develop a stronger understanding of the compression process, 2) become familar with encoding guidelines/practices, and 3) perform systematic tests. Encoding is well-suited to testing because it is easy to vary one parameter while holding others constant. Over time, compressionists learn how to improve not only the encoding settings, but also upstream activities like editing, compositing, colorization, camera movements, et cetera to ensure smooth playback.


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]