-
Notifications
You must be signed in to change notification settings - Fork 19
Description
Description of feature
It would be great if some additional voting methods were implemented to aggregate across classifiers apart from simple first-past-the-post voting. The basic idea would be to treat the classifier certainties for each cell as a ranked ballot, and compute the election results using some selection of methods, like Instant Run-off, Copeland's method, or Borda count (naturally, there's no expectation that one settles the question of which voting method is optimal – a small group of representative ones will do). There are existing Python packages, like pref_voting, that implement multiple methods and could potentially be used for this.
Additionally, even without aggregating the votes in multiple ways, treating the classifier certainties as rankings opens the possibility of computing finer-grained summary statistics for classifier agreement, like Kendall's W. If we only count up how many classifiers agree on their first choice, we leave a lot of information on the table.