This thing is BUGging me since Day 1 !!

[Gregory]

Hey Tyson...

I've been postponed to post this for a very long time due to been always busy with work, but I can't take it any longer.

Granted we have a rather slow network in our office due to extreme security, VPNs etc.
But...
Every time I save a scene with the TyCaches pointing to our network folders - in order to render with Backburner in our network renderfarm - the scene takes AGES and EONS to save the file in the local hard disk. Where if the TyCaches are pointing to the local hard disk the saving is INSTANT!

So, what gives?
There must be something terribly wrong going on here; a millisecond local save of the scene file, becomes 10 minutes if that scene contains TyCaches that point to a network location instead of a local hard disk.

On the same note, when loading back those scenes containing TyCaches pointing to a network location, also takes MUCH longer to load, compared to when those caches are pointing to a local disk; not as slow as when saving, but significantly slower than when network locations are not used.

I hope these make sense and there is a cure for this; it's just TOO MUCH for me having to deal with this issue for so long.

Thanks,
Greg

[tyFlow]

I've never run that issue myself and 99% of my tyCaches are loaded/saved from a network.

Are you using mapped drives or UNC paths? Windows can sometimes take ages to resolve UNC paths, especially if the destination folder doesn't exist.

[Gregory]

I've never run that issue myself and 99% of my tyCaches are loaded/saved from a network.

Are you using mapped drives or UNC paths? Windows can sometimes take ages to resolve UNC paths, especially if the destination folder doesn't exist.

Yes, I'm using a mapped drive.
It's been like this for ever here.
Network speed is slow, for sure; measured around 2 to 5 mb/sec, based to file size.

But, still...
What does the location of the saved and cached tycache files have anything to do with the loading/saving process of the scene file?
And that's with the scene file placed locally on the PC, not over the network.

[Gregory]

It feels like as if TyFlow checks each tyCache file one by one, when saving the max scene?

Our network is slow, yes.
But the asset location is just a mere text path, right?
It should not affect the saving process at all.

It only happens with TyCaches.
Doesn't happen with any other asset used in the scene and placed on the network, like alembics, phoenix caches and other type of caches in general.
Only with TyCaches.

[tyFlow]

2-5mbps? Did you mean 2-5gbps? The speed you listed is roughly equivalent to 3G cellphone speed......that is....an incredibly slow network speed for any LAN.

It's quite strange that saving the file causes the slowdown since I have stops in my code to explicitly skip any tyCache data processing while saving.

Could you try isolating a single tyCache object to an otherwise empty file and see if you can reproduce the saving issue? Perhaps use a windows performance-measuring tool to see if you can isolate what bottleneck is happening during that slowdown (network transfer? CPU activation? HD activation? etc).

It certainly seems, based on your description, that it's a tyCache issue...I'm just not sure what exact part of tyCache code could be the problem. Some extra tests/benchmarks like the ones I described above would help me to track it down. I haven't run into that issue myself but I also don't have an incredibly slow network to test things on so it's a bit of a tricky situation.

[Gregory]

2-5mbps? Did you mean 2-5gbps? The speed you listed is roughly equivalent to 3G cellphone speed......that is....an incredibly slow network speed for any LAN.

It's quite strange that saving the file causes the slowdown since I have stops in my code to explicitly skip any tyCache data processing while saving.

Could you try isolating a single tyCache object to an otherwise empty file and see if you can reproduce the saving issue? Perhaps use a windows performance-measuring tool to see if you can isolate what bottleneck is happening during that slowdown (network transfer? CPU activation? HD activation? etc).  

It certainly seems, based on your description, that it's a tyCache issue...I'm just not sure what exact part of tyCache code could be the problem. Some extra tests/benchmarks like the ones I described above would help me to track it down. I haven't run into that issue myself but I also don't have an incredibly slow network to test things on so it's a bit of a tricky situation.

Yeah, I'm afraid it's 2-5 MBPS...

IT won't change that anytime soon; they are capping the speed on purpose.
It's 10MBPS when working from the office, but since the last year and so, working from home has a greater impact on the speed of the network and it's 5MB per second at best of times.

If it helps, it gets slower the more cached frames I have in the tyCache object.
So, in a completely empty scene with just one tyCache object, a 1000 frames tyCached sequence will make the scene file to be saved 100 times slower than a 10 frames one. That's why I'm saying it feels as if the tyCache is "checking" each individual cache file when the "Save scene" command is invoked.

The attached screen shows an empty scene with a single tyCache object, containing 502 cached frames, located on our network drive.
The scene file is 348 kb on disk.
It took 40 seconds to save on my local SSD.
I only noticed network activity spiking while saving; no CPU, GPU or HDD activity happened.

If I change the location of the cached files to the local disk instead of the network, the scene saves instantly; must be milliseconds, can't really measure that.

[tyFlow]

Ah, that's interesting you mention the linear relationship between number of files and save time.

tyCache implements a max API function which enumerates auxiliary files. I don't make use of that info myself in tyCache mesh-building code (which would explain your slowdown even though I skip mesh processing during file save)...I simply hand it off to max whenever max asks for it (one example of where max uses the info is to list dependent files of an asset in the Asset Tracker window). The function iterates over the folder that your tyCache points to and returns all relevant filenames. On a normal network this likely takes a few milliseconds, but on a network like the one you're describing, I guess it takes a lot longer for thousands of files.

So my best guess is that max is requesting this enumeration during save/load (which would make sense...so that max can tell you if enumerated files are missing when you load, because the required filenames are stored during save).

Looking further into the problem, I can see within my code that the problem is likely related to the functions I use to scan directories for files. Googling the key Windows API calls I'm using leads to various people online complaining about their performance in situations with hundreds of thousands of files (which would further explain terrible performance across your network).

I will see if I can optimize those functions. Send me an email to support@tyflow.com (so I have your email) and I will send you a test build once I implement an alternative method.