Why NavMesh and formations need separate responsibilities
Navigation answers where an agent can travel. Formation logic answers where an agent should be relative to the group. Treating these as separate layers makes the system easier to reason about.
Generate world-space slot targets
Generate local formation offsets first, then transform them by the group anchor and orientation. Each resulting position becomes the desired target for navigation or steering.
Obstacles can break the shape
A perfect formation target may be unreachable because of a wall, narrow passage or moving obstacle. Allow individual paths to deviate temporarily, then give units a regrouping behaviour that brings them back toward their slots.
Regroup after a choke point
When a squad exits a narrow area, do not instantly force every unit into place. Let the formation expand gradually and use arrival thresholds to avoid oscillation.
Prevent target thrashing
If the formation recalculates every frame and changes slot targets aggressively, navigation agents can constantly restart paths. Cache targets and only rebuild them when the group moves far enough, rotates significantly or changes formation state.
Large groups
For large squads, calculate shared movement decisions at the group level where possible. Individual agents can perform lightweight final steering while the formation system handles the expensive structural work.
A practical architecture
Use four layers: formation generator, group anchor, navigation/movement adapter and local avoidance. This makes it possible to swap navigation strategies without rewriting the formation system.
Conclusion
The most reliable Unity workflows separate responsibilities, make expensive work predictable and turn repeated checks into repeatable systems. Start with a small implementation, measure the real bottleneck and expand only when the project needs it.