The SetLoadFields method will return some other fields from the table, not just the ones we implicitly asked for. If it turns out that some of those fields are from table extension, that will cause a JOIN in the SQL statement.
Because of that, we lose most of the performance benefits of SetLoadFields() which was supposed to give us in the first place.
Things go even further, so if, for example, you have BLOB fields that are obsoleted, the SetlLoadFields method will include those fields in the query, even if they have ObsoleteState set to Removed. That means that in such a scenario you lose the main benefit of retrieving partial records.