1. 4

On trying to figure out quantitative stuff and not getting it right. Doesn’t mention Fermi estimates, but it did remind me of them.

  1.  

  2. 2

    This is great.

    I was also reminded of Fermi estimates when I read this. This is an excellent example of why I am very skeptical of them, especially in domains where the estimator has no, or little, domain knowledge. There’s really no way to know whether the numbers you’re estimating are even remotely in the same ballpark of order of magnitude, and there’s no guarantee at all that you won’t get most of them basically right but then will be off by twelve OOM on one of them!

    Of course, my skepticism could be biasing me here. I wonder if it’s possible to give any sort of general account of in which kinds of problem domains high-OOM errors are possible and not exceedingly rare, and in which they are? This seems like it would be a useful approach to answering “what kinds of problem domains are likely to yield to Fermi estimates, and, conversely, what domains it is dangerous to attempt Fermi estimates in”? (And what affects the answer? Is it only subject-independent characteristics of problem domains, or do a subject’s own biases and background knowledge (or lack thereof) overshadow differences in domains? Or am I thinking about this all wrong?)

    1. 1

      Fermi estimates aren’t really estimates and should not be treated as such. They’re upper (or lower) bounds. Maybe we should call them Fermi bounds instead.

      For example, let’s try to figure an upper bound on the number of piano tuners in New York City. I know that New York City has somewhat less than 10 million people, so I assume that the population of NYC is 10 million. And of those 10 million, let’s say 1 in 10 people own a piano. Given my personal experience, far fewer than 1 in 10 people in real life own a piano, so even if the piano ownership rate in New York is far higher than the piano ownership rate on the west coast, 1 in 10 should cover it. So that gets us an upper bound of 1 million pianos. Now, let’s say that each piano has to be serviced once a week, and that each piano tuner can only service a single piano per day. Assuming that piano tuners work 5 days per week, this means that we have an upper bound of 200,000 piano tuners in NYC.

      Now, is this anywhere close to the real answer? Almost certainly not. But it does establish an upper bound. If someone were to claim, for example, that there were a million piano tuners in NYC, I’d be able to say that’s a pretty dubious estimate. But if someone said that there were 50,000 piano tuners in NYC, the Fermi bound doesn’t give me any data.

      Fermi bounds work well when you have a good sense of the absolute upper or lower bound of your data. I know that there are fewer than 10 million people in NYC. I know there are fewer than a trillion stars in the Milky Way. But I don’t have a good sense of what the upper bound is how many atoms of gold there are in a gold ring. Moreover, chemistry is one of those fields where things can vary by many orders of magnitude. Computer science is another such field. Fermi estimates tend to fail in those situations, simply because the amount of variance in the data overwhelms our ability to put bounds on it. Fermi estimates come from cosmology, where things only vary by four or five magnitudes, and you have pretty good handles on the upper bounds for values like, “number of stars in the Milky Way” or “number of galaxies in the observable universe”. They work less well in other fields that have greater variance in the magnitudes of their numbers.

      1. 1

        Yes, this is a good example of what I mean:

        so even if the piano ownership rate in New York is far higher than the piano ownership rate on the west coast, 1 in 10 should cover it

        What if some institutions have many pianos? Are pianos the kind of thing that a school or a concert hall might have dozens or hundreds of? I have no idea!

        let’s say that each piano has to be serviced once a week

        Why? What if it has to be serviced daily? Do I know that to be false? I do not!

        and that each piano tuner can only service a single piano per day

        Why? What if tuning a piano takes a week? Do I know that to be false? I do not!

        Now our upper bound is higher by at least an order of magnitude, maybe more; and a million piano tuners no longer seems weird.

        Of course, if there were a million piano tuners in NYC, I’d have met one, or heard of such people existing, at least, whereas in fact the only reason I even know there is such a profession as “piano tuner”[1] is from this exact silly archetypal interview question—but that sort of reasoning has nothing to do with Fermi estimates!

        [1] Actually, the way the question is phrased, it doesn’t exclude the possibility of “piano tuner” being a piece of equipment rather than a person, so maybe this whole approach is wrong! Why am I assuming a “piano tuner” is a person who tunes pianos? I don’t know anything about what “tuning” a piano even is, or means. Heck, maybe a “piano tuner” has nothing at all to do with pianos, and the name is coincidental or something.


        My point is that “the numbers in this field have great variance” is not the only failure condition of Fermi estimates; “minimal domain knowledge” is another, because while the numbers in a field may have (relatively) low variance, subjectively the variance may nonetheless be arbitrarily high, because you personally have no idea what range of numbers to expect (on account of your lack of domain knowledge).

    Recent Comments