ADOBE AFTER EFFECTS: Forum Expressions Tutorials Creative Cloud

Link external data (.csv) file

COW Forums : Adobe After Effects Expressions

<< PREVIOUS   •   FAQ   •   VIEW ALL   •   PRINT   •   NEXT >>
Christine Tang
Link external data (.csv) file
on Jul 20, 2020 at 4:00:12 pm

I am having problems linking a external data file (.csv) to my composition.

I have the equation:
myData = thisComp.layer("CompiledTrainsReduced2.csv")("Data")("Outline");
i= timeToFrames(time);
if( i > 0 && i < myData("OBJECTID").numProperties-1)
{
initTime = myData("OBJECTID")("OBJECTID " + (i-1)).value;
endTime = myData("OBJECTID")("OBJECTID " + i).value;
initPosition = eval(myData("ML1_X")("ML1_X "+(i-1)).value);
endPosition = eval(myData("ML1_X")("ML1_X "+(i)).value);
linear(time,initTime,endTime,initPosition,endPosition)/7.9
}

The problem is, my .csv is too dense and eventually takes hours (10hours+) for After Effects to import.
I'm wondering if it's possible to get data from my computer and fetch data through there?
I have tried:
myData = "/c/Users/tangc5/Downloads/Trillium_Development/Data/Paul/Master_Table_2SHORT.csv";

But that does not work. I've tried both forward slashes and back slashes.
If there are no ways to fetch data from my computer, are there alternatives?

Thank you.


Return to posts index

Filip Vandueren
Re: Link external data (.csv) file
on Jul 20, 2020 at 8:29:20 pm

Don’t add the .csv to your timeline.
Just import it in the project and reference it directly within your expressions. Use footage()
It’s the adding it to the timeline that takes ages, because after effects needs to create an interface to the data

I’ve found that manually looking through the csv’s sourceText with string-functions is also a lot faster than using the sourceData methods.

If you post a small excerpt of a few lines of csv I could help in more detail

Or see this thread:

https://forums.creativecow.net/thread/227/45348#45397



Return to posts index

Christine Tang
Re: Link external data (.csv) file
on Jul 21, 2020 at 6:58:19 pm
Last Edited By Christine Tang on Jul 21, 2020 at 6:59:59 pm

Here's an excerpt of my data:


My columns show, OBJECTID, POS X, POS Y, Rotational Angle, per object I am animating through the .csv. There's roughly 50 objects with 3 values each. OBJECTID is a time value per frame.

How do I then change my code I posted earlier:
myData = thisComp.layer("CompiledTrainsReduced2.csv")("Data")("Outline");
i= timeToFrames(time);
if( i > 0 && i < myData("OBJECTID").numProperties-1)
{
initTime = myData("OBJECTID")("OBJECTID " + (i-1)).value;
endTime = myData("OBJECTID")("OBJECTID " + i).value;
initPosition = eval(myData("ML1_X")("ML1_X "+(i-1)).value);
endPosition = eval(myData("ML1_X")("ML1_X "+(i)).value);
linear(time,initTime,endTime,initPosition,endPosition)/7.9
}

ML1_X is my data set's position X. And this expression is plugged into my layer, position x.


Using your footage code, I am trying to grab the data value from POS X to link to Position X in my layer and have it play by frame by frame by OBJECTID (which is time in my document).


Return to posts index


Filip Vandueren
Re: Link external data (.csv) file
on Jul 21, 2020 at 7:03:48 pm

Can you copy the entire text of 1 or two rows (all columns)

Is this data that will change often and need to be re-imported ?
In other words, is it important that the data stays 'live' -> the CSV might change ?

If not, then it's very possible that writing a script to create keyframes is a better solution.



Return to posts index

Christine Tang
Re: Link external data (.csv) file
on Jul 21, 2020 at 7:06:40 pm

Data needs to stay live as it would constantly be changed/edited unfortunately..


Return to posts index

Filip Vandueren
Re: Link external data (.csv) file
on Jul 21, 2020 at 8:03:07 pm

Ok,

No problem.
Like I said, the fastest way is to just look at the sourceText of the csv footage,
Make substring of the line you need (with a regexp that matches a unique ID)
You can then .split(“,”) to get an array.
Then you can just get the items out of the array as column indexes.

If there’s data for every single frame, then there shouldn’t be any linear() interpolation though?

If you paste a full header and line of data from the csv I can go in more concrete detail if you wish



Return to posts index


Christine Tang
Re: Link external data (.csv) file
on Jul 21, 2020 at 9:00:09 pm
Last Edited By Christine Tang on Jul 21, 2020 at 9:07:11 pm

Thank you, please go into detail as I had help coming to my current expression...

14222_trains000.csv.zip

Here is the file. If I had help starting it off for the X axis I can continue and adapt the expression to my Y and Rotational axis. I use POINT_X_0 and POINT_Y_0 for one train and POINT_X_1 and POINT_Y_1 another train. In the example earlier "ML1_X" is the same as "POINT_X_0" in my document, I just renamed it for my clarity.

My goal is to plot pos X and pos Y from my data set to animate a Train moving by each frame. You mentioned the linear() might not needed, if there are better ways to approach this and make after effects less laggy, please let me know.

Apologies for my lack of expression knowledge. I am new to learning expressions. I took a look at your thread you linked and tried out footage() but did not figure out how to adapt it to work like my original expression did.


Return to posts index

Filip Vandueren
Re: Link external data (.csv) file
on Jul 22, 2020 at 12:25:25 am

OK, thanks for sharing.

That's about 50 minutes worth of animation data…

The good thing: there is a row for every frame, so we don't even have to search through the data to find a matching framenumber "f",
we can just take the f'th line (element f of the array of all the text split in \r's = returns ),


f=86613 + timeToFrames(time);
footage("14222_trains000.csv").sourceText.split("\r")[f];

(Nothing happens before frame 86613, so I've added that offset to my example.)

add the expression above to a text-Layer. name that text-layer "Row".
you can view the data changing every frame.
Now, we will re-use the values in text-layer in other expressions in your train layers(rotation, position,…) because that's a lot faster than starting every expression with the same lookup in the csv.

for example, point 0's position would be:

row = eval("["+thisComp.layer("Row").text.sourceText+"]");
[row[6],row[7]] / 7.9;

(because x and y are columns 7 and 8 in the csv)

point 0's rotation would be:

row = eval("["+thisComp.layer("Row").text.sourceText+"]");
row[5];

(because rotation is the 6th column)

I've done some tests:
Using After Effects' built-in dataValue() methods can render a bit faster (after a half minute apparant system-freeze), but they are muuuuuch slower to work with in the timeline.
It feels as if the whole .csv has to be read and parsed before any manipulation (or rendering), but once that's "cached" it's ok.
But, If you for example disable an expression and enable it again, then DataValue() would think again for 30 seconds.

My sourceText-method just looks up the needed values on the fly.
If the outside file changes or needs to be re-imported, that also takes about 30 seconds on my Macbook.

I was able to render the visualization of the final 1minute 47 seconds of data (the only part where something happens) in under 2 minutes .

project and testrender:
https://we.tl/t-pwpd5YqeKi



Return to posts index

Filip Vandueren
Re: Link external data (.csv) file
on Jul 22, 2020 at 10:07:09 pm

A last tip, but potentially Important:

When you import a .csv file,
- in the File Dialog enable "All Files"
- select the .csv file, but beofre importing it;
- Change the file import Format from "Comma Separated Value" to "Javascript"

Boom: loads instantly, no half minute waiting for csv parsing.
If you accidentally drop it in a comp: no 10 hour waiting, just a "useless" layer that gets added to your comp.

Javascript files behave as an inert chunk of text, just waiting to be interpreted or read by an expression, unlike a .csv or .tsv (.txt) that wants to parse itself to create handlers to make its data accessible.

Now, to be clear: You cannot use dataValue() methods on it at all.
Only manipulate the footage().sourceText.

But as I've shown - for now as of CC2020- for large datasets it is much much faster than using After Effects' own csv parsing, no crashing, no waiting.



Return to posts index


Christine Tang
Re: Link external data (.csv) file
on Jul 23, 2020 at 2:02:30 pm

Thank you Filip,

I've spent the last day working with your file and it works extremely well! Your explanations make it easy to understand with my little knowledge of expressions, thank you.

I ran into the same problem you mentioned with dataValue(), it renders faster but in that it slows after effects down extremely. Your work around is much better!

This final tip I will explore today as that was something I was struggling with. My .csv's take hours to import and when I close my after effects and re-open my file it takes hours to open as well.

I'll edit this text blurb to let you know if all is successful but once again I appreciate your time guiding me through this! Thank you so much.


Return to posts index

Filip Vandueren
Re: Link external data (.csv) file
on Jul 23, 2020 at 2:34:30 pm

Does even my example file take hours to load?
As I mentioned, it opens in about 30 seconds for me (before I added the”JavaScript file” hack)



Return to posts index

Christine Tang
Re: Link external data (.csv) file
on Jul 23, 2020 at 7:10:40 pm
Last Edited By Christine Tang on Jul 23, 2020 at 7:42:57 pm

It opens within seconds but for some reason it renders out in After Effects fine but when I try to render it out in Media Encoder, my trains with the code on it is missing but my backgrounds and other layers are rendering fine.

Any reason why it doesn't work in media encoder?



Return to posts index

Filip Vandueren
Re: Link external data (.csv) file
on Jul 23, 2020 at 7:53:22 pm

Please, for the love of god never render in AME

It’s a mystery to me what that thing wants to or does not want to render.
It’s just a black box that “kinda” renders aep.-projects but then often it kinda doesn’t...

I hate it and I suspect the feeling’s mutual :-)



Return to posts index

Tomas Bumbulevičius
Re: Link external data (.csv) file
on Jul 30, 2020 at 12:20:21 pm

Hey Filip, well, I can't agree more, but at the same time - disagree as well, haha!

Its a bummer that rendering through Render Queue disables you from using AE - thus, whenever project is not terribly complex, AME works just fine.

And not to mention additional encoding to web-friendly formats.. Thus unless person is capable in automation of things through ffmpeg and stuff, AME is pretty much time saver, till a certain point :)

Find out more:
Motion Graphics Design & After Effects Tutorials
On YT
On VH


Return to posts index

Filip Vandueren
Re: Link external data (.csv) file
on Jul 30, 2020 at 12:30:12 pm

Hi Tomas,

Well, before I started using Rendergarden, I often just used the command-line renderer that is included with after effects.

- Open a Terminal
- drag and drop the app aerender on the console.
- type the word "-project "
- drag and drop the .aep file on the console,
- hit enter.

That takes about 5 seconds and requires 0 programming/automation skill ☺
I think this is exactly what the background-render script just automates.

True After effects rendering in the background, with After effects still available for you. You can start as many concurrent renders as you want this way.



Return to posts index

Tomas Bumbulevičius
Re: Link external data (.csv) file
on Jul 30, 2020 at 12:37:47 pm

Filip, well - yes and no again from my point of view, haha!

Considering aerender.exe approach you mentioned - what you do with a rendered output, when it finishes? If it is image sequences, you also need audio to be rendered. Proper profiles defined, etc. And yes, encoding it to final output too, whether its lossless single output or sequences.

Thus, while it "kind of" helps, it adds extra work - who wants to do extra tedious work, raise your hand, haha!

This is why it is a questionable approach, which requires excessive management, if done manually, IMO ☺ So talking about general use case scenarios for most users, without many bells and whistles - I would say AME 95% of a time.

Find out more:
Motion Graphics Design & After Effects Tutorials
On YT
On VH


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2020 CreativeCOW.net All Rights Reserved
[TOP]