 |
| Built by France in the wake of the First World War, the Maginot Line was an engineering marvel completely unsuited to the challenges of the Second World War. (Image via History.com) |
This is the seventh year that the E Pluribus Hugo (EPH) methodology of tabulating Hugo Award nominations has been in effect.
Since it was ratified at the business meeting in 2016, EPH has weighted nominating votes in an attempt to ensure that the shortlist is more representative of Worldcon fandom than it was in years past. By our count, the use of EPH has resulted in changes to the Hugo shortlists on 35 occasions. Over the past eight years, this system has removed some works from the shortlist in favour of other works that were nominated by a smaller (but hypothetically more representative) demographic.
Given that there have been almost 900 finalists across all Hugo (plus Lodestar and Astounding) categories since EPH went into effect, that means the new system has made about a four per cent difference to the shortlist.
In essence, EPH created noise around the edges of the data, to little benefit.
EPH was proposed in the wake of the 2015 Hugo Awards controversy, during which a co-ordinated minority of fans were able to overwhelm the nomination process. It was one of a variety of solutions proposed as a remedy to the problem of slate voting.
At the time, those involved with this blog were in support of the EPH proposal. Sure, sometimes it produced weird results like keeping Arkady Martine off the Astounding Award ballot in 2020 … but that seemed like a small price to pay to prevent another year like 2015, in which havoc raged and resulted in five categories resolving as “no award.”
 |
| It's worth noting who gets added and who gets removed from the shortlist due to EPH. (Image via Hugo Awards 2017 nominations) |
In the intervening years, EPH has not been faced with a significant challenge. From 2017 to 2022, nomination patterns among Worldcon members was as expected, with no “slate” that needed to be accounted for. If the data from this year is correct, however, the highly-correlated list of finalists that all received similarly inflated numbers of votes does more than just resemble a ‘slate.’ (This is not to imply malicious action on the part of those casting nominating ballots, but to say that clustered votes that are correlated due to a highly influential recommended reading list will be treated by the EPH system in a way that is similar to a slate of nominators.) And in the face of this trial by fire, EPH has failed.
EPH has also not lived up to the promise that it would ensure that different factions of fandom would be represented in the final ballot. Looking over the list of those who have been excluded from the Hugo Ballot because of EPH, you’ll find some excellent folks who have yet to receive their first nominations. If not for EPH in 2022, Black Nerd Problems would have become the first fanzine made by Black SFF fans to receive a Hugo nomination. If not for EPH in 2020, Priyanka Krishnan would have been the second-youngest editor ever shortlisted for a Hugo Award. Meanwhile, EPH has secured additional nominations for some of the folks who have been recognized the most often in the past. It was a solution that may have reinforced systems of power instead of mitigating their impact.
Another issue with EPH is that it can be gamed. Sufficient people nominating only one item in a category are likely to boost that one finalist through a process that’s been dubbed “bullet voting.” The effects of this can be extreme. In 2023, Oghenechovwe Donald Ekpeki’s short story Destiny Delayed was omitted from the Hugo Award ballot … despite receiving almost twice as many votes as the shortlisted work Resurrection by Ren Qing.
Equally if not more damning, EPH has created a barrier to the public understanding of how the Hugo Award nominees are selected. The integrity of the nominations process, and thus the awards themselves, is being questioned for a variety of reasons, and an arcane system of tabulation only adds to the problem. People are unlikely to trust a system that they don’t understand, and an obfuscatory system they are expected to participate in is anathema to public trust and participation.
EPH doesn’t offer better results, it simply picks different finalists in a way that seems to increase the democratic deficit in our community instead of removing it.
Fundamentally, we’ve seen that “E Pluribus Hugo” has not functioned as intended, produces a shortlist that less accurately reflects the will of the Worldcon community, and adds confusion to the process. It’s time to abandon it altogether. It’s time to craft tools appropriate for tomorrow’s awards.
All of the comments about 2023's data presume that the underlying numbers are legitimate, and given that the number of nomination points in Round 9 (the first one published in the longlist) literally add up to more than the number of ballots cast in four categories, I do not think that we can stipulate that the data is good. We can't necessarily say /what/ is wrong with it, just that something is pretty clearly off. [There are a few other categories where this number is "less than 100% but quite close". That's not the smoking gun of "over 100%" and on its own wouldn't be more than just being a little bit surprising, but taken with the over-100% categories it looks pretty bad for the validity of the data.]
ReplyDeleteHaving said that, /any/ system is vulnerable to swamping by a large enough number of voters. Even "partial block" voting can be overwhelmed if you have enough folks and efficient vote allocation/coordination.
The flip side of this argument is that EPH allowed Adrian Tchaikovsky to secure his first nomination in 2022 over a novella written by an author who was already a finalist in that category. The effect of EPH in Series that year was also to prefer a shortlist with six different authors to one with five. I think both of those are positive impacts on the ballot.
ReplyDeleteAs we've seen this year, EPH also permits individual voters (or small groups) to verify that the finalists were tabulated correctly in ways that are not possible under the old system.
I was very glad to see Adrian Tchaikovsky get his first Hugo nod. But there have been at least as many disappointments due to EPH as there have been positives.
DeleteThe problem is that attempts to influence the vote, be it bullet voting, slating or some other as yet undiscovered method, is not revealed until the nominations come in.
ReplyDeleteWe do an initial round of qualifying nominees based on the rules, but we've got no mechanism to address situations in which the system is hacked.
This is why I was in favor of "3SV" - Three Stage Voting - which was proposed as an addition to EPH, but not ratified when EPH was.
In essence: the top 15 nominees from the normal and customary initial round are published and the electorate is then asked to vote again, on only those top 15 nominees, as to whether or not they should be eligible to continue on to the final ballot.
Nominating stats are not included, and the list can be randomized to avoid weighting things. (Do you think this nominee should proceed to the final ballot? Yes/No)
The five nominees with the most initial nominations that aren't voted out then proceed to the finalist ballot.
The vulnerability is that those inclined to "slate and hate" will target a nominee(s) and get a work unjustly removed from consideration.
The original proposal for 3SV also suggested that only members of the current Worldcon (the one hosting the awards the ballots are for) are eligible in this second round as one way of mitigating against that.
Perhaps modifying 3SV in some way(s) might offer solutions to that.
For example, supposed that instead of addressing the entire (15 in each category) list of potential nominees, voters participating are limited in the total number of "NO" votes they are permitted to make. You can down-check 6 total, or ten total (out of a minimum of 285 potentials)...or maybe one per award category.
There are other possible "solutions", one that I'm personally kind of in favor of, though I believe the vast majority would object: no voting rights at all until you've got membership in two or three worldcons under your belt.
Reasoning behind that is - it takes time and money to do that and there's a good chance that someone who has is more interested in participating maturely, rather than doing so to screw with things. Or maybe you don't get to nominate, just vote in the final during your "probationary" period.
Or (need a budget increase and maybe some software for this one), we treat the nomination process as the final list. Any nominated work that is eligible under the rules, doesn't get withdrawn and which receives a minimum of X nominations goes on the final ballot.
Problematic for distributing free copies of everything. On the other hand, there's no real need to try and slate or fix.
Just blueskying, but clearly we need to do more work.
3SV could work, but yeah, I think randomizing the names on the "Fifteen list" would be mandatory, to avoid people just going for the highest nominated ones or even the ones in low alphabetical order.
ReplyDeleteThis post misrepresents as promises a number of facts about EPH. For example, there is no possible way you can make EPH deliver the finalists YOU would want. That's up to the voters. And would be without EPH. EPH is a means, in the face of attempted slating, to open some slots on the final ballot to other potential finalists. It does not screen the finalists. And you know that.
ReplyDeleteStrongly disagree that EPH should be dropped. The impossible nomination stats we've been given make it easy to see that the problems with the 2023 Hugo Administration go even deeper than wrongfully removing eligible finalists. Malfeasance like this would be much easier to hide under the old system.
ReplyDelete