I just thought of another way of operating this, that isn’t as precise but might give interesting results:
Just use the peak detector as a trigger. Treat that as your clock.
It will be all over the place, but close to following the live drumming groove. You can work with that.
I’d do a proof of concept if you want to share a sample of the drumming.
The drawback of using the peaks for a clock like that is that anything you trigger will play on top of your drum recording. I fix that by using an AS Signal Delay to delay the gate.
Even more fun, use the triggers from the envelope followers to clock a Hetrick Random Gates. Then you have 8 gate outputs to trigger different sounds.