Unfortunately Microsoft does not support adding a security filter on an exclusion line in a permission set.

Thought it is practically possible to do so - it has NO effect. The exclude action takes all focus and ignores the security filter.

At the same time the Effective Permission page for that specific object line looks weird when doing so.

Since the theory of building permission sets encourages inclusion of several standard permission sets into a new custom one + excluding a few specific objects, it is not logical that you cannot exclude using a security filter.