basic semi-broken things

[AlexMcLeod]

Hooray, it's awesome!

...but there are things that could definitely be improved, just to move away from Particle Flow's dismal legacy, and those improvements would probably break a lot of users' setups, so I'm going to get into them ASAP.

• There doesn't seem to be a dt. There's "framestep", but for anything like a proper simulation workflow it's imperative that the only unit of time is the second, and that simulation timestep can be decoupled from your framerate. It follows that time-related values like velocities and accelerations and birth-rates should always be in units per second, not per frame.
  
• Is the integration scheme at least semi-implicit Euler, ie: symplectic? Many effects have a significantly different result (and not just in terms of accuracy) if you change the timestep, which is undesirable, because we should only be changing the timestep when troubleshooting.
  
• It seems like tyFlow doesn't store acceleration per particle. Does this mean that velocity is always modified directly? That's going to lead to a lot of accuracy and stability problems, and it'll make it harder to parallelise any interactions between particles, or guarantee deterministic output. It's not necessary to cache acceleration, but it's necessary to have an acceleration channel, and for forces to accumulate there before being added to velocity. Velocities in sims can be interdependent, but accelerations generally cannot, so if everything goes into acceleration before velocity, less can go wrong.
  
• In the Slow operator, you have "slow by surface proximity". You also have a "follow" option in the Surface Force operator. These should probably be combined, because most of the time, we'll want surfaces to drag particles around (including stopping them), and combining the two manually doesn't quite work as expected.
  
• There's no air resistance? The Slow operator doesn't take air-speed or a particle's shape into account, so we can't do leaves or snow or cloth realistically, without scripting it every time.
  
• There should be a global Air thing, because we've never had air before, and this is the opportunity to make the best air in the industry. So instead of wind as a force, instead of turbulence as a force, as an alternative to the Slow operator, you should be able to pipe any & all of those things (Surface Force, Point Force, legacy spacewarps etc.) into an Air object, which represents the air everything is moving through and can have its velocity sampled anywhere. At any given point in space the Air would return an absolute velocity (the sum or a composite of everything fed into it), but unless you set it to water-like density, it wouldn't have a lot of mass to push things around. Then events could have an Airforce operator that just samples the global Air object and responds appropriately, taking per-particle mass into account, and an approximation (bbox, convex hull etc.) of the particle shape. This would basically replace all other wind-like forces for users wanting realism.
  
• Anywhere you have an "attraction parameter" with a falloff, you should also have a "repulsion" parameter with a separate falloff. That allows particles to stick to things (including each other) without just smashing into them, so you can make compressible gases, bouncy force-fields etc., while only calculating the neighbourhoods etc. once per frame.
  
• Anywhere you have a falloff, there should be a way to specify the falloff curve. Often, a cubic or similar sigmoid is preferable to inverse square.
  
• Display of bindings etc. and DrawMarker in scripts show the particle position (ie: GetPos()) lagging behind the particle mesh. That's confusing to look at and it's not much fun having to compensate for it every time we use DrawMarker.
  
• The UI has a mixture of spinners using percentages (0% to 100%) and spinners using floats (0.0 to 1.0). For example in Particle Physics, stiffness is in the range of 0.0 to 1.0 but friction is a percentage. Could we just lose all the percentage spinners and have spinners that go from 0.0 to 1.0?
  
• The UI has a mixture of Max units (like inches or centimetres) and plain old floats, where they should only be plain old floats. For example, if the scene is set to use imperial measurements, the Particle Bind operator's mass multiplier is specified in inches, which makes no sense.
  
• Anything that has "variation" needs a distribution function. For most uses, an exponent would be sufficient (eg: lerp(min_val, max_val, powf(GetRandFloat(sInx, 0.0f, 1.0f), exponent))), but there would probably be cases where users would want a custom distribution curve. This is scriptable but it'd mean having to script just about everything that initialises a particle's size, colour, velocity etc. and that'd get unwieldy.
  
• Interparticle friction in Particle Physics has been implemented in a way that doesn't preserve angular momentum. This means lumps of viscous stuff can't tumble! The trick to proper interparticle friction is to make sure it only applies along the vector between the two particle positions, by scaling it with the dot product of (vel2 - vel1) and (pos2 - pos1) so it has no effect at glancing angles.
  
• Particle Physics in general seems to be badly affected by not storing acceleration separately from velocity. With no other forces acting on it, a particle system should produce no net motion - but with friction enabled, I guess you're modifying local velocity while you're reading neighbour velocities, because it adds up to a small, spurious net motion. That's how you end up with a non-deterministic system that doesn't conserve momentum.
  
• Case-insensitive scripting would be very much appreciated, or just plain lower-case aliases for all the keywords. Who's going to remember that "Invert()" is capitalised but "inverted" is not? Or that "hitPoint" is lower camelcase but "OrigLength" is upper camelcase, and "rest_length" uses underscore instead? Or that "id" means "ID"?
  
  
• The Script operator would benefit from some more conventional versions of vector & matrix methods, like cross(a,b) instead of Point3.Cross(a,b), because users will want to paste in code from other APIs etc. where vector types are native, like OpenCL, OpenGL, GLM, CUDA C and so on.
  
• I don't know how you're solving springs & constraints, but to make them thread-safe, you either need to store an accumulated force or displacement in the constraint itself, which you apply to the particles in another loop. A lot of the literature on constraint solvers says it's okay to just let constraints iteratively modify particle positions or velocities directly, but that doesn't work in a predictable way if you parallelise the solver, which is why you have to accumulate & apply, instead.
  
• Dual quaternions could be a more efficient way of storing particle transforms. Matrices and scales and rotations work, too, but dualquat transforms are inherently rigid (orthogonal etc.), and they only require 8 floats each.
  

I probably have code lying around for a lot of the above, which I'd be happy to share. 

[tyFlow]

Hey Alex, you wrote a lot of things and I'll try to address them.

Things should be in units per second and not per frame.

Since offline sims like tyFlow don't have variable framerates, I decided to work in units per frame rather than second for simplicity. I personally am the opposite of you - since I often want intuitive control over particles I don't like having to constantly do back-of-the-napkin calculations figuring out exactly how far my particles will travel if my vel X and my framerate is Y and my unit is seconds. Per-frame units makes it very simple to understand how spinner values correspond to particle motion over time.

Acceleration, velocity integration, etc

tyFlow accumulates velocities within particles over each timestep by iterating over operators and adding their resulting force values to each particle's velocity channel. At the end of the timestep, the velocity is integrated and added to particle positions. Accumulated velocities are multiplied by the frame dt,

Slow by surface should be combined with follow surface

Not quite sure I understand this one...follow surface adds velocity to particles that is parallel to a surface. Slow by surface dampens velocities based on surface proximity, two different things.

Air resistance

Lift forces for cloth (which would be applicable to leaves, paper, etc, as well) is on the todo list.

Air force

I'm not quite sure what you mean by this...some kind of grid that forces are fed into? Those forces would need to be advected as well, to ensure realism. Similar effects can already be achieved by using a Fluid Force on particles with an input PhoenixFD simultaion (and FumeFX at some point in the future).

Attraction/Repulsion

Generally such spinners can be given negative values, for opposing effects.

Falloff curve

Agreed, right now everything is basic inverse-square. At some point in the future I'd like to either offer different modes, or a literal GUI curve you can manipulate.

Certain display markers lag behind particles

There's probably room for improvement here but it's most because the data for them is stored prior to velocity integration. For some markers, it's tough to predict where they'll be at the end of the time step because future operators may change the forces they're based upon. An easy solution might be to make them position/velocity independent and just save some stub of data which can be converted at the end of the time step prior to display...I'll look into doing that.

Some spinners are floats, others are percentages

Spinners that aren't marked with a "%" do not modify values by percentage, but instead by value. For some things this distinction is necessary, so it's important to note where percentage symbols exist in the spinner label, to understand how the spinner will add variation. As for the scale of the percentage variation (being 0-100%, instead of 0-1), that's just a personal preference.

Some spinners show units

I think displaying units is important for a lot of things. It doesn't actually change the internal data value, but if I'm working in inches for example...I'd like to know my force values in inches to get a better idea of how far they will move my particles relative to the current unit system. Or the radius of something in units so I know how big it will be relative to other things I've built to scene-scale. However, you're right about mass value...that's a mistake and it shouldn't be displayed in world units. That will be fixed in the next build. I'm sure there are other places where this mistake may be visible....I'll keep my eye out for them and if you see any other rogue units showing up (or not showing up), let me know!

Spinners need distribution functions

This is another thing on the (distant) roadmap that I'd like to implement. I used to have exponents in a few places where I figured they would be used most (and they're still in a few places, like the mass operator)....but eventually I'd like to add them back. It was just cluttering the UI a bit having all of those secondary spinners, so I'll be experimenting with ways to keep things tidier without sacrificing functionality.

Interparticle friction doesn't preserve angular momentum, scale with dot product.

I'm already doing this in my friction model, I think the artifacts you're noticing are simply the result of using a position-based integrator with high friction values. Although if you have a specific algorithm I could take a look at that you're referencing, I'm always open to improving things!

With no forces acting on it, a particle system should produce no net motion

If you have a test case where the opposite is occurring, let me know - because it shouldn't be. The only time the solver itself introduces motion is if it's separating inter-penetrating particles.

Case -insensitive scripting

The scripting language is C#, which is an inherently case-sensitive language. The reason for capitalization in function names is because I'm adhering to the language standard (upper case for function names, lower case for variables).

More forgiving vector/matrix methods

I can appreciate this suggestion...I'll see about adding class-independent methods for the various math-specific functions at some point.

How are you solving constraints

I use a partitioning approach you can read about in a lot of recent literature. The accumulation approach you mention has the downside of requiring a huge number of steps to properly converge.

Thanks for your thorough critique! I appreciate all of the thoughts and suggestions!

[AlexMcLeod]

Per-frame units makes it very simple to understand how spinner values correspond to particle motion over time.

Yep, I understand that from an animation point of view, but all the papers you'll find on fluid dynamics, rigid-body dynamics etc. and all the third-party physics libraries are going to be in metric units, so any time you feel like incorporating the latest & greatest simulation methods, you'll have to rework them to use your time & space units instead. I've made that mistake before when writing simulators, and I'm just trying to keep you from getting locked into the same mistake for too long, because fixing it later requires breaking things.


Maybe you see having per-frame stuff to be intuitive because you've been using PFlow for so long? But even PFlow breaks the 3dsmax convention of allowing changes to framerate without affecting the look of the animation (much like you can change the scene units), because PFlow doesn't allow retiming without a lot of mucking around. It's not often that we have to change framerate mid shot, but tying information to frames (which can be any fraction of a second) instead of actual seconds means that when a shot does have to be overcranked for slow-mo, or just for troubleshooting, the effects animation would have to be reworked.

tyFlow accumulates velocities within particles over each timestep by iterating over operators and adding their resulting force values to each particle's velocity channel.

Yep, but that's probably why the Particle Physics operator produces weird results, and it means that we'd have to add those channels and do our own back-of-the-napkin calculations for every Script operator. Especially because none of this is in seconds, so none of this is in Newtons, or ms^2 and so on.

Storing accelerations and then adding those to velocities simplifies all sorts of particle stuff, and makes it way more flexible, too. Highly recommended.

Slow by surface should be combined with follow surface

Not quite sure I understand this one...

Say you've got a great big space monster moving through a great big cloud of space stuff, and you want the particles near the monster to be dragged along with it. If you use Slow by surface, the particles just stop while the monster drifts past - they're being stopped in world space, rather than being dragged by the monster's velocity. Or say you want a passing car to disturb leaves on the ground.

A combined surface drag operator would just mix the velocity of nearby surfaces into particles' velocities (preferably as a force, so that it stays somewhat physical).

Air force... I'm not quite sure what you mean by this...some kind of grid that forces are fed into?

Not a grid, no. Just procedural, like the existing Force operator (or the Composite texture, for that matter), in that it takes a list of other forces as inputs, and at any given point in space it just sums the results of those inputs and returns a vector. But that vector wouldn't represent a force, per se - it'd tell you the speed of the air at that location, in metres per second (or miles per hour, or inches per frame, or knot-gallons per fathom-furlong, or whatever velocity units you end up with when you don't use the metric system).

Then particles that with little mass but lots of surface area (soap bubbles, for example) would almost immediately take on the velocity of the air around them, but denser particles like rocks wouldn't be blown around at all. The Air object would be an optional background - a light breeze, or a user-driven whirlwind, or a draft near an open window, or lightly churning water, or a Phoenix or Fume sim - and any event could just receive the local air-speed and accelerate its particles accordingly.

Currently, the built-in Wind will accelerate particles indefinitely, which is a non-physical carry-over from the legacy particle systems, which were written completely naively with regards to real-world physics. We could just use something like a Drag warp to cap that acceleration, like we've always had to do in Max, but that's a non-physical hack that we've been stuck with for 20 years, and it'd be nice to finally be able to make a proper wind with a proper wind-speed. Some of the old Blur spacewarps came closer to getting this right, but they were still just a slightly better hack.

A grid for advection would be neat (Stoke had that - whatever you fed it, it could store it in a grid and strip out the divergence) but it wouldn't be necessary just for making more plausible breezes. A grid just for visualising the air-speed would probably be handy, though.

Interparticle friction doesn't preserve angular momentum, scale with dot product.

I'm already doing this in my friction model, I think the artifacts you're noticing are simply the result of using a position-based integrator with high friction values. Although if you have a specific algorithm I could take a look at that you're referencing, I'm always open to improving things!

None of the viscosity models in SPH, PBF etc. exhibit that kind of angular damping, even with viscosity cranked all the way up to peanut butter levels, so any particle fluids papers would be good for reference. But of course they'll all assume you're buffering accelerations (or at least double-buffering velocity), and they'll all be based in physical units like seconds...  

With no forces acting on it, a particle system should produce no net motion

If you have a test case where the opposite is occurring, let me know - because it shouldn't be. The only time the solver itself introduces motion is if it's separating inter-penetrating particles.

The artifacts I'm seeing are the kind of artifacts you get when you try to manipulate interdependent values like velocity directly, instead of manipulating their derivatives or double-buffering them.

Test case attached. With gravity on, you'll see the kind of rotational damping I'm talking about (the lump can't be induced to tumble), and with gravity off, you'll see the spurious net motion, which changes direction when reseeded.

The scripting language is C#, which is an inherently case-sensitive language. The reason for capitalization in function names is because I'm adhering to the language standard (upper case for function names, lower case for variables).

Is there anything like #define or typedef for type aliases in C#? Can we #include? Then we could just paste a list of lower-case keywords into every script. Without autocomplete etc. it's going to be hard to remember to capitalise things that aren't usually capitalised in other languages, like crosses and dots and getters and setters.

I'll see if I can manage a Maxscript to filter all detected keywords into the correct case, but it's gonna be hacky.

I use a partitioning approach you can read about in a lot of recent literature. The accumulation approach you mention has the downside of requiring a huge number of steps to properly converge.

In my experience, storing the displacements in the constraints (and applying them, as a separate step in every solver iteration) requires the same number of steps to converge, because the math works out the same - but it avoids the race-conditions and doesn't require partitioning. Mine was set up with GPU in mind (or for lots & lots of threads), where partitioning has rapidly diminishing returns, anyway - but my most recent version was with XPBD, and I haven't tried it with regular PBD. It worked fine for Velocity Verlet and semi-implicit Euler integrators, though.

[tyFlow]

Well I still think the seconds-vs-frame critique you offer just boils down to personal preference. What tyFlow lacks in a traditional seconds-based timestep, it makes up for in not having to do napkin calculations for hundreds of force spinners across all of its operators. If you want a particle to travel a certain distance by the next frame, you just type it in. Since I find myself needing to dial in values on a per-frame basis more than a per-second basis for complicated effects, it's a no brainer.

If a complex sim does need to have its framerate changed for whatever reason, using the retiming spinner makes it a piece of cake. Same for slowmo.

Doing the to/from second conversions in 3rd party libraries/solvers I've integrated has been a simple conversion done at the start/end of the timestep, that the user doesn't have to worry about anyways so it has zero impact either way.

tyFlow's bind solver is position-based, unlike traditional MPM/SPH/etc, so they can't really be compared so well when it comes to the finer details. My friction model is straight out of the literature as well, and you can see similar artifacts with high-friction values in other implementations of similar solvers (see: Flex). These artifacts can usually be negated by just adding some position offset to particles when they're birthed, so they don't immediately form tall stacks.

Autocomplete for the script operator is a good idea...something I've thought about. Right now I have syntax highlighting but a little popup dropdown would add a lot of efficiency.

Your attachments didn't show up so I can't examine artifacts you mentioned unfortunately...I'm still trying to get attachments working properly, for some reason some people are having issues.

[AlexMcLeod]

What tyFlow lacks in a traditional seconds-based timestep, it makes up for in not having to do napkin calculations for hundreds of force spinners across all of its operators.

But... your way, anyone else who does their physics in seconds has to do those napkin calculations you're avoiding, but in the other direction. The rest of the world (physicists and simulation researchers and FX TDs in particular) has already standardised on the metric system, in which the unit of time is the second.
Using an arbitrary timestep also means we can't just use real-world values for acceleration, velocity etc. without always having to guess how they'll scale in tyFlow, or even what unit parameters like "strength" might represent (or even what order of unit, like is it displacement, velocity, acceleration or force?). It'll mean having to find fudge-factors for every setup - especially for scripts - which could be 24, or 1/30, or 4800, or the square root of 9.82, or just about any other number, and that's before we even have to consider spatial units. With the metric system, instead, the fudge factor would always be 1.0 or a power of ten. That's what makes the metric system so easy & reliable.
And while it's great that you've built in physics-like behaviours and stellar performance overall, it's still not-quite-physical, when with a little refactoring, it could be physically accurate (or at least state-of-the-art accurate), even more flexible, and with the same or better performance, because higher-order integrators allow larger timesteps without jiggling or exploding.
Anyway, the friction example I tried to post shows a wad of particles with Particle Physics friction that should be able to spin & tumble together, but instead it shows unnatural rotational damping. I've seen that before, when I messed up some friction calculations of my own. Using Particle Bind instead, the wad tumbles as expected (with only the minimal damping you get with PBD), which suggests you're only missing that dot-product friction scaling in the Particle Physics operator. If you treat interparticle forces as the summation of damped Hooke (or neo-Hookean) springs between particles, you can conserve angular momentum, and you can get all sorts of cool squishy behaviours to boot.
Don't get me wrong - you've done an amazing job putting this whole thing together, but you've copied a lot of the design flaws from PFlow (where "it sorta works" seems to have been Oleg's guiding design principle), and going forward, those decisions are only going to cause you and your users trouble. Straightening them out isn't complicated, but it'd break some existing setups, so... better done in the beta stage than after the 1.0 release?

[tyFlow]

I think I figured out what happened to your attachments....the way the forum code works, you have to browse for your attachment file and then click "add attachment" before clicking "post reply" or else it won't get attached. If you just press "post reply" the selected attachment is never uploaded. Kinda lame, I'll see if there's a way to adjust the way myBB works to fix that. In the meantime can you try uploading those examples again?

If you have a minute, maybe you could take a look at section 6.1 here, and see if it's working differently than how you imagine the friction algorithm should work. I don't know enough about various frictional models to understand the difference between the tangential vector/dot used in that algorithm and the one you've been describing. But at least then you'd know the method tyFlow is using, in order to compare.

[AlexMcLeod]

If you have a minute, maybe you could take a look at section 6.1 here, and see if it's working differently than how you imagine the friction algorithm should work.

At first glance, equation 23 in that paper should probably mean the frictional displacement is along the contact normal (same as the displacement that resolves interpenetration), and not perpendicular to it. So n would probably be the operative vector there, as it is in other particle methods going back to Baraff & Witkin. Unless NVidia have just found that it's easy & fun to make things pile up if they don't try to conserve angular momentum - but where would be the fun in a snowball that couldn't roll, or a sandcastle that couldn't topple?

I'll have a dig around in some other PBD/PBF implementations and see how they approach it.


Anyway, in the attached file (extra button pressed this time!) there are three copies of the same basic setup:
With Particle Bind, the lump of stuff hits the collider and bounces & tumbles as it should.
With Particle Physics, angular momentum is damped away entirely, allowing it to hang from the edge of the collider in a non-physical way. It should tumble something like the Particle Bind version, as should any breakaway groups.
The third one has no gravity and just shows how Particle Physics produces a net motion even when no other forces are acting on it, which just suggests your interparticle forces aren't quite symmetric.