PETSc CG solver enumerates DoFs in every solve
I just realized that the PETSc CG solver calls enumerate in every call to solve here: https://i10git.cs.fau.de/hyteg/hyteg/-/blob/master/src/hyteg/petsc/PETScCGSolver.hpp#L104.
Because that involves global communication (an expensive allGather
), this is obviously far from ideal.
I think that the enumeration should happen outside in a setup phase, such that it is done only once and can be timed independently from the solver.
To give you a perspective, in my recent scaling experiments on SNG Phase 2 using 192 nodes, 18% of the overall solve time (levels 0 to 7) is used for enumeration on the coarsest grid.
We must definitely check the other solvers, too.