You ask a great question about the difference between ‘fair’ and ‘impartial’, and your surmise is along the right lines.
The difference is the standard of reference. The reference for ‘fair’ is what is culturally or commonly accepted. The reference for ‘impartial’ is other people or organizations.
So if I pay you $500 for damages to your car that may be ‘fair’ in that it is what is commonly paid for that type of damage.
If I pay one person $500 and another person $5,000 for the same damage, that is not impartial, in that personal attributes are likely to play a part in the decision to pay differential fees.
Impartiality implies that all persons are treated the same regardless of affiliation, race, color or creed.
Fair implies that the treatment met some ethically or culturally determined standard, such as ADA requirements, test speededness, test reliability, item authenticity, blueprint conformance, etc. Notice that to be fair you sometimes have to distinguish between people on some culturally or legally accepted principle.
Hope this helps…