notes/Projects/bachelor/RUNTIME_EXECUTOR.md

RUNTIME_EXECUTOR

This component receives a serialized node_graph and returns the output. I generally love the idea that a node is a pure function which just receives its own inputs and then calculates them. BUT! there is always a but, there are some things that the nodes needs that are not its own inputs. These are in my experience:

run_seed Some nodes generate random values, and to generate random values we generally need a seed value.

settings There are some values that are kind-of global inputs. That means they are inputs to the node, but each node receives the same input. For example for the plant nodes, they are the resolution of each leave.

After some thinking, my idea is to model these inputs as hidden inputs to the node. So they are not gonna be visible in the NODE_INTERFACE.

But they are gonna be passed to the nodes as normal inputs. This could be super useful to allow easy memoization of the nodes, because if none of the inputs changed, we can return the previously computed output.

Linking/Importing of Nodes

WebAssembly Components - only works in wasmtime / jco at the moment

For this the runtime_executor needs to be wasm aswell.

No linking + Very easy to implement, works everywhere - Probably not as fast as components

Handle multiple outputs

The general idea is to model the nodes as a visual representation of pure functions, that means the function only operates on it’s own inputs, doesn’t have any side effects and always produces the same output for the same input.

For very simple graph, such as the following this works very well.

The math/add node could be modelled very simple as the following pure function:

fn math_add(a: i32, b:i32) -> i32 {
  return a + b
}

The problem occurs if we try do adher to this principle for more complex use cases.

Problem Example 1

Users would probably expect the random node to output for different results. If we would model the random node as a pure function, it does not know how many results it should generate.

Problem Example 2

On the left you see a node setup for the procedural generation of geometry. On the right one of the probably outcomes a user would expect.

If we model the noise function as a pure function we would first have to figure out how to pass the result of the noise function as a pure value to the cylinder node. The noise node also has no idea how many outputs are connected to it.

Solution 1 (Power to the Nodes)

We could solve this by letting each node execute its parent node graph however it wants. This would allow the cylinder node to execute the noise however often it needs to create the cylinders.

// how this could look like in the node

{
  id: "cylinder",
  inputs: {
    "amount": { type: "integer", value: 1 },
    "height": { type: "float" },
    "radius": { type: "float" },
  },
  execute: ({ amount, height, width }, executeNodeGraph) => {

    const stemAmount = executeNodeGraph(amount);
    const ringAmount = 10;

    for(let i = 0; i < stemAmount; i++){
      for(let x = 0; x < ringAmount; x++){
        const radius = executeNodeGraph(width);
        // create a single ring
      }
    }
      
  }
}

Pros

• nodes become very flexible on how the work in their inputs

Cons

• Every Nodes needs access to the Projects/bachelor/RUNTIME_EXECUTOR
• It could become very unpredictable how each node behaves, because the each have a own way to handle inputs

Solution 2 (Parametrized nodes, embed execution)

We could change the output of the noise function to output a definition of noise like the following:

{
  "type": "noise",
  "size": 0.7,
  "strength": 1.5
}

Or the random node could output something like this:

{
  "type": "random",
  "min": 0,
  "max": 42
}

Then we could embed a function that nows how to turn these parametrized nodes into numbers inside the cylinder node. One problem with this is that this function needs to be recursive, because the inputs of the parametrized nodes could also be other parametrized nodes:

{
  "type": "noise",
  "size": 0.7,
  "strength": {
      "type": "random",
      "min": 0,
      "max": 42
    }
  }
}

Pros

• The overhead inside the cylinder node is smaller than the other solutions
• BIIIG Pro; the runtime knows nothing about that

Cons

• We basically have to embed the execution function of number nodes inside other nodes
• recursive execution of the nodes definitely needs memoization

Solution 3 (Partially invert the flow)

Normally the flow of information in this graph is only from the parents to the children nodes. One idea could be for the children nodes to send to their parents what they need, and then have the parents return that result to their children.

Solution 4 (Add execution information to parameters)

We could also add metadata to the node definition to tell the Projects/bachelor/RUNTIME_EXECUTOR. This could either be how it should execute the parent nodes of a node, or directly in the node how it should be executed.

For example in the first #Problem Example 1 it could look like this

const randomNode = {
    id: "random",
    inputs: {
      "value": { type: "float", value: 0.1, internal: true },
      "seed": { type:"runtime_seed" }
    },
    outputs: ["float"],
    execution: {
      // this is the important part
      distinctOutput: true
    },
    execute: ({ value, seed }) => { 
      return value
    }
}

This could tell the Projects/bachelor/RUNTIME_EXECUTOR to execute the randomNode for every of its child nodes individually.

---

Parameters Dimensionality

We could define how many dimensions a parameter can handle and which range it could handle. For example the radius parameter could be a 2D parameter, and we could map the x coordinate to the angle relative to the center of the stem and y to the height along the stem.

The noise node would output a 2D parameter. We sample it with 2 coordinates. But we could also only sample it with one coordinate.

If we would have a theoretical curve function, it would produce a 1D parameter which is sampled with a x coordinate from 0 to 1.

The random node outputs a 1D parameter, we sample it with the seed.

Then we would need a way to sample this parameter where we need the value.

const cylinderNode = {
  id: "cylinder",
  outputs: ["geometry"],
  inputs: {
    height: { type: "float" },
    radius: { type: "float" },
    amount: { type: "integer" },
    seed: { type: "runtime_seed" },
    xResolution: { type:"integer", setting:"resolution.x" },
    yResolution: { type:"integer", setting:"resolution.y" },
  },
  execute({ height, radius, amount, xResolution, yResolution, seed }){
    const coordinates: number[][] = [];

    const angleIncrement = (2 * Math.PI) / xResolution;

    const amount = sampleParameter1D(amount);

    for (let j = 0; j < amount; j++){
      const cylinder: number[] = []
      
      // here we sample a 1D Parameter
      const heightIncrement = sampleParameter1D(height, j) / yResolution;
      
      for (let i = 0; i < yResolution; i++) {
          const y = i * heightIncrement;
  
          for (let j = 0; j < xResolution; j++) {        
              const angle = j * angleIncrement;
              const radius = sampleParameter2D(radius, y, angle);
              const x = radius * Math.cos(angle);
              const z = radius * Math.sin(angle);
              coordinates.push( , y, z);
          }
      }
      coordinates.push(cylinder);
    }

    return coordinates;
  }
}