This speaks to me!
I’ve had the same concern and a backlinks section in the document itself indeed seems like an improvement. Haven’t tried it though.
Regarding performance, having optimized the org-id code myself, it is possible to make it fast. But all is relative I suppose. Have you done any profiling?
If its slower than org-id-update-id-locations (from master) then there should be low hanging fruits in the performance department.