1. 4

  2. 5

    The lord of men has two difficulties to face: If he appoints only worthy men to office, ministers will on the pretence of worthiness attempt to deceive their ruler; if he makes arbitrary promotions of officials, the state affairs will always be menaced. Similarly, if the lord of men loves worthiness, ministers will gloss over their defects in order to meet the ruler’s need. In consequence, no minister will show his true heart. If no minister shows his true heart, the lord of men will find no way to tell the worthy from the unworthy.

    For instance, because the King of Yue liked brave men, the people made light of death; because King Ling of Chu liked slender waists, the country became full of starvelings; because Duke Huan of Qi was by nature jealous and fond of women, Shu Diao castrated himself in order to administer the harem; because Duke Huan liked different tastes, Yiya steamed the head of his son and served Duke Huan with the rare taste; because Zikuai of Yan liked worthies, Zizhi pretended that he would not accept the state.

    Therefore, if the ruler reveals his hate, ministers will conceal their motives; if the ruler reveals his likes, ministers will pretend to talent; and if the ruler reveals his wants, ministers will have the opportunity to disguise their feelings and attitudes.

    That was the reason why Zizhi, by pretending to worthiness, usurped the ruler’s throne; and why Shu Diao and Yiya, by complying with their ruler’s wants, molested their ruler. Thus Zikuai died in consequence of a civil war and Duke Huan was left unburied until worms from his corpse crawled outdoors. What was the cause of these incidents? It was nothing but the calamity of the rulers’ revelation of true hearts to ministers. Every minister in his heart of hearts does not necessarily love the ruler. If he does, it is for the sake of his own great advantage.

    In these days, if the lord of men neither covers his feelings nor conceals his motives, and if he lets ministers have a chance to molest their master, the ministers will have no difficulty in following the examples of Zizhi and Tianchang. Hence the saying: “If the ruler’s likes and hate be concealed, the ministers’ true hearts will be revealed. If the ministers reveal their true hearts, the ruler never will be deluded.”

    – Han Feizi, Ch. Vii, “The Two Handles”, tr. W. K. Liao (with, of course, the Wade-Giles converted to Pinyin)

    1. 3

      To put this in more rationalist terminology, one of the ways around Goodhart’s law is that you find a measure… but keep it secret. Then, so long as no one else figures out the measure, you can target it without the measure being corrupted by people gaming the system.

      The problem is that the measure will inevitably become corrupted because people aren’t that dumb. I have an anecdote from a friend who worked at Subway. Subway implemented its own version of the lime equation - except instead of counting limes, they’d count bread. So you couldn’t get a free sandwich, because that’d make the bread count come up short at the end of the day. But, you could order a 6-inch veggie, and end up with a footlong deluxe and the tracking system would miss that.

      Another example is college admissions. Good colleges don’t publicize their “lime equations” determining how they determine who gets a spot and who doesn’t. But people reverse engineered the process anyway, simply by using other statistics about the student body that were published by the campuses. So, at this point, it doesn’t really do any good for colleges to keep their equations secret - all that does is disadvantage the people they’re trying to help the most.

      If a metric is high-stakes enough, it will be targeted, whether its secret or not, because no matter how secret the metric is, targeting it will have effects on the real world, and those effects, given time, will be observed and reverse-engineered.

      EDIT: The e-mail volume metric he mentions is exactly the sort of thing that’s easy for employees to game and corrupt. Given time, people will figure out that the people who send a lot of e-mails are the ones the boss favors. Then everyone will send lots of e-mails all the time, regardless of whether they’re getting any work done or not, and the advantage of the metric will disappear.

      1. 1

        This is essentially the objection I was going to post, so I entirely agree with it.

        Relevant to this is this discussion on LessWrong (1.0) about eigenkarma and Google’s PageRank algorithm. Some key quotes:

        Ilya Shpitser:

        This won’t work, for the same reason PageRank did not work, you can game it by collusion. Communities are excellent at collusion.

        Oliver Habryka:

        “This won’t work, for the same reason PageRank did not work”

        I am very confused by this. Google’s search vastly outperformed its competitors with PageRank and is still using a heavily tweaked version of PageRank to this day, delivering by far the best search on the market. It seems to me that PageRank should widely be considered to be the most successful reputation algorithm that has ever been invented, having demonstrated extraordinary real-world success. In what way does it make sense to say “PageRank did not work”?

        Ilya Shpitser:

        Google is using a much more complicated algorithm that is constantly tweaked, and is a trade secret – precisely because as soon as it became profitable to do so, the ecosystem proceeded to game the hell out of PageRank.

        Google hasn’t been using PageRank-as-in-the-paper for ages. The real secret sauce behind Google is not eigenvalues, it’s the fact that it’s effectively anti-inductive, because the algorithm isn’t open and there is an army of humans looking for attempts to game it, and modifying it as soon as such an attempt is found.

        Wei Dai:

        Given that, it seems equally valid to say “this will work, for the same reason that PageRank worked”, i.e., we can also tweak the reputation algorithm as people try to attack it. We don’t have as much resources as Google, but then we also don’t face as many attackers (with as strong incentives) as Google does.

        Ilya Shpitser:

        It’s not PageRank that worked, it’s anti-induction that worked. PageRank did not work, as soon as it faced resistance.

        1. 2

          Thinking more on this, it seems like this is the takeaway:

          Goodhart’s Law is not so easy to outsmart. There are no silver bullets. As usual, there are only a whole lot of lead bullets, and those bullets look like this:

          1. Be really clear on what you are actually trying to maximize.

          2. Find a good and non-obvious measure of that thing.

          3. Keep that measure secret.

          4. Even as you use that measure, never forget what actual thing you’re after.

          5. Keep looking for new, non-obvious measures of that thing.

          6. Abandon the old measure and switch to a new and different one as soon as you detect even a hint of gaming.

          7. Repeat steps 2–5 indefinitely, forever.

          This is hard. It is not a one-stop solution. You can’t ever stop working on this. It requires continuous effort, possibly even continuously increasing effort (as new measures are probably going to be harder and harder to find).

          But I don’t know that there is any alternative.

      2. 2

        One good example of this thing is the moderation heuristics I use for banning people on the #lesswrong IRC channel. I could put them in the guidelines, but I don’t because they consistently identify trolls long before anyone else calls it. I pretty much always call troll first, and I’m pretty much always right.

        Recent Comments