Educators have devoted little attention to formal evaluation of educational administrative personnel. The authors surveyed the educational administrators working in McMaster University’s Department of Medicine residency program and found they felt they were receiving little useful feedback. The authors also surveyed the colleagues, residents, and administrative staff with whom the administrators worked and found they felt they had inadequate input into the administrators’ evaluation. In response to these results, a measurement instrument was developed based on existing job descriptions and feedback was obtained on administrators’ performance from relevant individuals. After three yearly evaluations, administrators and evaluators acknowledged much broader input into evaluation but saw little evaluation related improvement in performance. Of the administrators, 85% felt the process should continue as did 91% of the evaluators. An evaluation process may not alter perceived performance when it is already good but there may be other benefits to rigorous evaluation.