I understood that by design when a record is deleted from Dynamics, the associated folder in SharePoint will remain.

D365 CE and SharePoint is a native integration and it is quite natural to expect associated SharePoint folders + files to be deleted on the deletion of dynamics records, the same way how they get created from the dynamics. It would be nice and helpful if you can have this as a feature in upcoming releases.