If a field is set to 'Non searchable' it still appears when selecting a series or legend in the chart designer.
I believe this is a bug and will become problematic for deployments that have several fields that are either no longer used or have similarly named fields that may confuse the end user.