SetupPrimitiveStorage is fully resident in all processes' memories
The general process of doing virtually anything in HyTeG starts like this.
const MeshInfo meshInfo = /*...*/;
const SetupPrimitiveStorage setupStorage(meshInfo, /*...*/);
std::shared_ptr< PrimitiveStorage > storage = std::make_shared< PrimitiveStorage >( setupStorage );
While the PrimitiveStorage
is fully distributed, the same is not the case for the MeshInfo
and the SetupPrimitiveStorage
.
To my limited experience the MeshInfo
is less of a problem but the SetupPrimitiveStorage
can get quite large (#177).
This limits the scalability of our code.
Let us consider an example.
Assume there is a system called "Hawk", which has 128 cores and 256GB per node, i.e. 2GB per process.
Let us say we know from smaller runs that this is enough to fit 2 level 7 cells per process with our chosen discretization.
We now try to scale our code to 2048 nodes and therefore construct a coarse mesh with 2048*128*2 = 524 288
macro cells.
The SetupPrimitiveStorage
now consumes more than 8GB which is far beyond the available 2GB per process.
It is therefore the limiting factor to scalability.
In principle there is no need for the SetupPrimitiveStorage
to be resident in each process.
Even if we do not distribute the SetupPrimitiveStorage
, allocating it only once would allow sizes close to 256GB.
Any thoughts on how to solve this issue are highly welcome.