That's pretty cool! I would like to add a reordering functionality in the NCMesh initialization, but haven't gotten to it yet. I would also appreciate any information on Gecko.

Thanks for sharing Dan, that's pretty cool :).

@acfisher had more experience with Gecko when he integrated it a while ago (see MFEM_USE_GECKO in the code).

Can you move step 5 earlier in your workflow? If the coarsest mesh is well-ordered, the NCMesh operations should produce reasonable-looking partitionings (you will also not need METIS).

@Dan2997925,

Parallel non-confoming meshes do not use the partitionings generated by Mesh::GeneratePartitioning - they always use Hilbert curve ordering - so what you're viewing is not really the parallel partitioning. The simplest way to do that in VisIt is to create an order 0 "L2" field and initialize it with the processor id.

I have a branch where I add more reordering methods, including the current reordering based on Gecko - I can push that work to GitHub if someone is interested to take a look. I have not worked on it for a while, so I don't remember how close it is to a version that can be merged into master.

Also, this is not obvious to me: is the mesh non-conforming from the beginning, i.e. step 1, or do you perform the non-conforming refinement later?

Hey,
I'm going to answer for Dan as he's out of town this week and Dan can correct me if I'm misunderstanding anything later :).

@v-dobrev

Also, this is not obvious to me: is the mesh non-conforming from the beginning, i.e. step 1, or do you perform the non-conforming refinement later?

The mesh can be non-conforming from the beginning as our software can take in a MFEM mesh format which supports non-conforming meshes.

@tzanio

Can you move step 5 earlier in your workflow? If the coarsest mesh is well-ordered, the NCMesh operations should produce reasonable-looking partitionings (you will also not need METIS).

I actually think for this problem moving step 5 earlier would work. I think what Dan is trying to do in steps 2 through 4 is to call METIS to re-order the elements before the NC code just splits the global element sequence into N parts (based on suggestions from issue #258) in case the original mesh is not well-ordered. However, since METIS makes no guarantee about making N equal parts, we end up getting reasonable partitions + sprinkles floating around when the splitting is performed.

METIS (used by Mesh::GeneratePartitioning()) should produce mostly contiguous blocks, so something is not quite right. In METIS 5, I think, the blocks are guaranteed to be contiguous - or at least there is an option to force this behavior.

I believe in this case METIS is generating contiguous partitions but these partitions are neither completely equal nor spatially next to each other in increasing rank (neither which are guarantees). I think because of this, chopping up the mesh into N parts based on global mesh numbering will result in mostly contiguous partitions that will have additional elements from the next or previous METIS partition which can be spatially located anywhere. I think that's why Dan's partitioning looks like it has some structure but with some sprinkles all over.

I think I should be able to show this on a much simple (and smaller case) - let me know if we're still in disagreement about the METIS strategy and I'll put that up on Monday.

If you are reading-in an AMR MFEM mesh then you have the whole refinement hierarchy and the default partitioning that MFEM will use when transitioning from Mesh to ParMesh should be good - it is based on a space filling curve. This will be true unless the coarsest mesh in the AMR hierarchy was not ordered nicely.

So obtaining a good partitioning for an AMR MFEM mesh relies on the coarsest starting mesh from which the AMR mesh is derived.

In the example above, is the starting mesh (before AMR) uniform, Cartesian, nx by ny, mesh?

In the example above, the starting mesh is uniform, Cartesian, nx by ny mesh. I agree that converting from Mesh toParMesh would simply work in this case as the grid is well ordered. For the general case though, I don't think the METIS based re-ordering strategy would work (as exemplified by Dan's image); it sounds like the Gecko re-ordering is the way to go?

Gecko is definitely the better choice for reordering. The METIS approach may work better when applied to the initial Cartesian mesh compared to applying it to the refined AMR mesh. Even with Gecko, it is probably better to apply the reordering to the starting coarse mesh.

I'm fairly certain that Dan is applying METIS to the initial mesh, re-ordering, creating a parallel mesh, and then adapting.

@Dan2997925 can you update us on the status of this issue?