Does Multiple Criteria Decision Aiding (MCDA) improve the process of evaluating Machine Learning (ML) algorithms, when critical criteria of fairness are concurrently considered, beyond predictive power? To address this question, we employ several notions of fairness, such as Demographic Parity, Equalized Odds, and Lack of Disparate Mistreatment, and we appraise a set of supervised ML classifiers, under one of the most popular MCDA outranking methods, that is, the Preference Ranking Organization Method for Enrichment Evaluation (PROMETHEE) II. Moreover, to avoid any arbitrary choice in the importance attached to the criteria we apply the Stochastic Multicriteria Acceptability Analysis (SMAA), providing information in statistical terms through simulations. The empirical testing is processed over well-known databases, with several representative sub-datasets, securing variation in terms of observations’ volume. Overall, a series of ranking patterns that persists in the evaluation of the ML classifiers, across the utilized MCDA methodology and datasets, offers valuable relevant insights and documents specific useful interpretations. The obtained findings provide robust support that MCDA can be effectively exploited for the appraisal of ML classifiers, when aiming at the simultaneous consideration of critical fairness metrics, apart from the typical dimensions related to predictive power.