Alexander D'Amour

Kaleidoscope: Semantically-grounded, Context-specific ML Model Evaluation

Desired model behavior often differs across contexts (e.g., different geographies, communities, or institutions), but there is little infrastructure to facilitate context-specific evaluations key to deployment decisions and building trust. Here, we present Kaleidoscope, a system for evaluating models in terms of user-…