Let’s go back to the same data-transformation-graph as in the previous section:
We already assumed in the previous section that:
•The total available RAM memory inside the computer is 2GB (i.e. it’s a 32-bit computer).
•The memory used inside the MultiJoin Action to store the Slave Table is 1.7 GB.
Let’s do two more assumptions: Let’s assume that:
•The user already computed a HD cache at the following location:
This HD cache has been created by the user to help him during the interactive development of the data transformation graph.
•The “Memory Buffer Size” parameter of the Sort Action is 1GB.
When we run the graph by clicking the button on the toolbar (i.e. start without deleting any HD cache), Anatella re-uses the available HD cache and it can completely skip the (Multi)Join computation. This means that the amount of free RAM memory is 2GB. Since the Sort Action only requires 1GB of RAM memory, Anatella can easily sort the data.
When we run the graph by clicking the button on the toolbar (i.e. delete all HD caches and then start), Anatella must re-compute the MultiJoin Action (because the HD cache is not available anymore). This means that the amount of free RAM memory available for the sort is only 2GB-1.7GB=300MB. Since the Sort Action only requires 1GB of RAM memory, it fails.
For batch execution, Anatella never uses any available HD cache: it always re-computes everything from scratch (this is to ensure that we are not using old and inconsistent data). This means that, during batch execution, the Sort Action will also fail.
During interactive development, thanks to the usage of HD caches, the RAM memory consumption required to execute your data transformations is usually lower (because you can skip some computations such as the MultiJoin Action in the example here above) than in batch mode or “normal” execution mode. This means that some memory intensive Anatella graph might perfectly run in interactive mode but not in batch/production mode (because of the higher RAM memory consumption in batch/production mode).