(Cross posted at Decision Science News.)

Do you know what the #$*! your social media strategy is? Perhaps it’s “to facilitate audience conversations and drive engagement with social currency”? Or maybe “to amplify word of mouth by motivating influencers”? Well, given all the lies and damned lies being told about social, fellow yahoo Dan Goldstein and I decided to enter the fray with statistics. We measured the extent to which your friends’ behavior predicts your own, and found that in several consumer domains the effect is substantial, complementing traditional demographic and behavioral predictors.

That friends are similar along a variety of dimensions is a long-observed empirical regularity—a pattern sociologists call homophily. As McPherson et al. write in their canonical review on the subject, “homophily limits people’s social worlds in a way that has powerful implications for the information they receive, the attitudes they form, and the interactions they experience.” Turning this statement around, where there is homophily, one can in principle predict an individual’s behavior based on the attributes and actions of his or her associates.

To assess the quality of such network-based predictions, we merged a large social network (based on email and IM exchanges) with offline sales data at an upscale, national department store chain. Thus, for each of over one million users, we had their past purchase amounts in dollars, and had the same information for each of their network contacts. Think about this for a minute: we not only know how much these individuals themselves spent at an offline retailer, but also how much their social contacts spent, a testament to how profoundly the Internet is changing the way we study human behavior. (Despite bolstering social science research, these newfound tools raise serious privacy issues. We left the matching to a third party that specializes in doing this securely, so neither we nor the department store had access to the other’s complete customer database.)

The plot below summarizes our findings. First, as indicated by the top line, consumers whose friends spent a lot, also spent a lot themselves, consistent with the hypothesis that homophily extends to consumer behavior. When friends (alters) on average spent $400 during the six-month observation period, the consumer herself (ego) spent nearly $600, more than twice the typical consumer (indicated by the dotted line). As our aim is prediction, however, the relevant question is not just whether friends are similar in their purchasing behavior, but rather how much information is conveyed by social ties relative to other attributes. One might conjecture that ties simply indicate demographic (i.e., age and sex) similarity, that those who spend a lot are more likely to be middle-aged women—the primary market segment for this department store—and that friends of middle-aged women tend also to be middle-aged women. To test this hypothesis, we first paired each individual with a randomly chosen consumer of identical age and sex. The bottom line shows that this demographically matched group is, perhaps surprisingly, pretty ordinary. In other words, looking only at age and sex, you can’t identify consumers whose friends spend a lot (and who we know spend a lot themselves).

Though it’s standard marketing practice to target consumers based on their demographics, it’s an admittedly noisy profiling technique. So, to put social through the wringer, we next took the “socially select” group—consumers whose friends spent a lot—and matched them to random consumers with identical age, sex, and past purchases. Each social candidate, that is, was matched to a consumer not only of the same age and sex, but one who spent approximately the same amount as the social candidate during the previous six months. Even relative to this formidable baseline, social cues still provide considerable information. As the middle line indicates, knowing a consumer’s age, sex, and past purchases, but not that their friends are shopaholics, one would still underestimate their future sales.^[1]

We repeated this analysis for two other domains—examining signups for Yahoo! Fantasy Football, and clicks on ten online banner ads for movies, apparel, government programs, and beyond—again finding that the predictive power of social persists even after adjusting for age, sex, and past behavior. Lest you run off to rejigger your social strategy, I should mention a couple of caveats. First, we have shown that consumers with big-spending friends tend to spend a lot themselves—more, in fact, than their demographics or past purchases alone would suggest. But since most people, even premium customers, don’t have shopaholic friends, social cues do not substantially boost average predictive performance. Second, though social signals help predict how much consumers spend, they don’t always help identify which consumers will spend the most. Those who recently spent fifty grand on sartorial elegance are likely to be habitual top spenders, regardless of what you know about their friends.

Assessing the value of social, as with most things, is a messy affair. On the one hand, network ties convey information not captured by the usual egocentric metrics, a conclusion that at the very least I find scientifically interesting. On the other hand, it’s not immediately obvious how to use that knowledge to take over the world. Well, rest assured that an army of social strategy gurus are waiting in the wings with a game-changing, technology-disrupting way to, you know, leverage the social graph to deliver personalized experiences or something.

N.B. Thanks to Randall Lewis and David Reiley for acquiring the sales data, Jake Hofman for assembling the email data, and Duncan Watts and Dan Reeves for comments. The plot above was generated with ggplot2. For related work in the telecom domain, check out the paper, “Network-Based Marketing: Identifying Likely Adopters via Consumer Networks,” by Shawndra Hill, Foster Provost, and Chris Volinsky.

Illustration by Kelly Savage

Footnotes

[1] It’s perhaps tempting to conclude from these results that shopping is contagious (i.e., to assert causation where only correlation has been shown). Though there is probably some truth to that claim, establishing such is neither our objective nor justified from our analysis.

I love this quote from security expert Bruce Schneier:

Remember, if it’s in the news don’t worry about it. The very definition of news is “something that almost never happens.” When something is so common that it’s no longer news — car crashes, domestic violence — that’s when you should worry about it.

The truth of that^[1] hit home recently when I saw a news feature on the abduction of a four-year-old girl from her front yard in Missouri. Candlelight vigils, nation-wide amber alert, police blockades where every single car was stopped and questioned, FBI agents swarming the house. I think the expected reaction from parents is “oh my god, I need to be so vigilant, even in my own front yard!” My reaction was the opposite: Wow, this sort of thing really does essentially never happen. Let the kids run free!

I’m saying this without a drop of irony. Stranger abductions just don’t happen. You should worry more about your baby boy dying from complications of circumcision (true). What you should actually worry about are the real killers of kids, like driving and drowning. Speaking of which, here’s a valuable public service announcement about what drowning looks like. (It doesn’t look like drowning.)

By the way, if you didn’t hear the news story about the four-year-old girl, it had a happy ending. She turned up unharmed.

The Opposite Story: Things That Do Happen

But speaking of child abductions and drowning, I have to bring up a brutally ironic news story from several years ago.

In summary, a toddler got separated from her caregivers and wandered off, where a passerby saw her. Seeing no one else around, his first instinct was to scoop her up and take her with him. But he decided that that was a big risk to himself — what would people think? — and continued on, reporting it later.

The girl then fell in a pond and drowned.

Just to spell out the heart-wrenching irony: Even though the man made the wrong decision (and should’ve known it at the time) he was correct that helping that girl was dangerous to him. He could’ve been accused of abducting the girl and the accusation, sadly, and insanely, would have been taken very seriously. That needs to change. Remember, child abductions by strangers don’t happen! Picking up a lost child should not be regarded with any suspicion! Arrgh!

This is also why I think these common guidelines parents give their young kids about which kinds of adults to seek help from if they’re lost (store employees, other moms) are misguided and dangerous. The best person to get assistance from is the first adult they find. That adult simply is not going to happen to be a child predator! Put that absurd thought out of your mind.

By the way, the above story happened in the UK which, strangely, seems to be the only place worse than the US with this out-of-control paranoia.

But Think of the Children!

Q: Isn’t some amount of paranoia justified when it comes to children? Shouldn’t they at least be taught the difference between good strangers and bad strangers? Shouldn’t we be vigilant about any possible danger?

A: I disagree that children need to learn about “bad strangers”. I teach my kids that strangers are good, period. This is hard to talk about but it’s true: a scary number of kids really do get abused and it’s rarely by strangers. Your kids need to know about inappropriate touching but please don’t muddy that with talk of strangers. (Can you imagine, “it’s ok, I’m not a stranger…” Shudder!)

As for vigilance, it’s a limited resource: focus your energy on the real dangers.

At risk of getting repetitive I’ll re-assert my strong belief: No stranger will ever harm your child. It’s a risk (and, fine, if you want to get technical, it’s a risk, with some positive probability) that you can completely ignore. In fact, I’ll go further and say that maximum vigilance requires that you actively dismiss the idea that stangers can be dangerous. Doing so could prevent, for example, something like the tragically ironic drowning of the girl in that news story.

Q: I heard that there were 5 attempted abductions in Central Park last week. What do you say to that? (This was a real response when this came up on a parent list I’m on, though I doubt the veracity of the claim. The rest of these questions were real too, though most weren’t phrased as questions.)

A: It’s hard to even imagine this number but Central Park gets 25 million visitors every year. With those numbers, impossibly unlikely things will have happened to someone but it doesn’t necessarily mean you have to worry about them happening to you. Like when you hear about a meteor crashing through the roof of someone’s house (that’s for real, by the way).

Sticking with Central Park, here’s another real story from this summer: A woman and her baby were posing for a photo under a tree and a branch somehow fell off and killed the child. So that’s a 1 in 25 million chance of getting killed by a tree if you visit Central Park. (The appliances in your house are more dangerous than that.)

More to the point though, what’s an attempted abduction? Is it like this? A stranger talking to a child and the parents freaking out? Even more to the point, if you hear official numbers on child abductions they sound horrifying, until you learn that they’re mostly divorced parents in custody battles. Which should give you pause if you’re a divorced parent, but again we’re back to: know where to focus your vigilance. Think about who has repeated contact with your child, not the stranger in the park.

Q: What if child abductions are so rare precisely because parents are so vigilant?

A: I’m not advocating any kind of extreme disregard for children’s safety. In fact, I’m basically advocating raising kids more like how kids were raised a couple generations ago.

Q: But that was a different time!

A: Yes, it’s actually safer now. You wouldn’t think so from watching the news but crime has been going down for a couple decades and is currently at 1960s levels in the US.

Q: Fine, but what’s the harm in being cautious?

A: Here’s how Lenore Skenazy of Free-Range Kids (highly recommended) puts it:

Children, like chickens, deserve a life outside the cage. The overprotected life is stunting and stifling, not to mention boring for all concerned.

Straw Exit Poll

Thanks to Sharad Goel, Lenore Skenazy, Andrew Reeves, Laurie Reeves, David Pennock, and many parents at Harlem4Kids for reading drafts of this. If you liked this article you may like my twitter stream for parents: twitter.com/parentips

Image by Kelly Savage

Footnotes

[1] Sharad points out that Schneier’s quote is a little too glib. The more correct way to say it would be something like “prominence in the news does not necessarily correlate with the magnitude of risk to you”. As xkcd points out, frequency of occurrence in the news overall probably does correlate with actual risk. Sharad adds that there are actually prominent news features you should worry (though this is maybe a different sense of “worry”) about, for example, war and political/corporate corruption.

My email is dysfunctional. I keep things in my inbox because I can’t afford for them to go out of sight, out of mind — but then that’s exactly what happens. They get buried deeper and deeper in my inbox by all the other messages I delusionally think I’m going to deal with. It’s a given that some things will fall through the cracks. I just need to make sure I have some control over which things those are.

Let’s start with a straw poll:

Consider the reasons a message sits in your inbox:

• The To-do Excuse. You’re going to need to deal with it but you’re not ready to yet. (Some of this is procrastination, some is legit.)
• The Waiting Excuse. It’s something you’ve replied to but you don’t want to forget about it in case the recipient drops the ball. I.e., you might need to deal with it later.
• The Reference Excuse. It has information that you’ll want to refer to. (Different than maybe needing to deal with it later in that there’s nothing for you to remember to do.)
• The Laziness Excuse. It’s one of many messages that has outlived its usefulness in your inbox but you haven’t yet slogged through and archived them all.

In other words, it’s either serving as a to-do item, a might-need-to-do item, an item for future reference, or there’s no reason for it to be there at all. Or, of course, you haven’t read it yet. Which brings us to…

Inbox Zero

Inbox Zero is a simple system devised by Merlin Mann for keeping your email inbox empty. The idea is that your inbox should really only be for messages you haven’t actually read. So every time you check your email, do something with every single message. Specifically, do one of the following:

• Archive: just get rid of it (originally Delete^[1]).
• Delegate: get someone else to deal with it.
• Respond: if it takes just a few minutes, fire off a reply right then.^[2]
• Defer: put it on a to-do list, archive it, and deal with it later.
• Do: handle whatever the email actually needs you to do, then archive it.

The first one, Archive/Delete, is huge and completely destroys the Reference Excuse for a message to sit in your inbox. If it’s ok for a message to go out of sight, out of mind — like because you’ll go looking for it when you need it — then archive it immediately. Modern email clients make it easier to search among a million messages than to scroll and scan through a thousand. But even if you don’t have decent search, you should still get it out of your inbox. Just put it in a folder called Reference and hunt through that when you need to find it.^[3]

The last one, Do, is dandy. It helps with the To-do Excuse. The procrastination part anyway.

It’s the middle three — Delegate, Respond, Defer — that aren’t actually so helpful for keeping your inbox small. None of them address very well the Waiting Excuse, the Laziness Excuse, or the other half of the To-do Excuse (legitimate need to defer dealing with an email). First, Delegate and Respond don’t get things out of your inbox if you need to make sure they get dealt with and follow up if not. And Defer also can’t get a message out of your inbox without some system for transferring it to a to-do list, which is often too much overhead. However easy it is to put something on your to-do list, you’ve still got to retrieve the email again when it’s time to deal with it. Chances are you’ll be inclined to just leave it in your inbox because you’ll be dealing with it Really Soon.

So of the four excuses for a message to sit in your inbox, only the Reference Excuse has a satisfactory solution. What can be done about the others?

Proposed Email Features

What if you had a snooze button for email — a way to easily make a message disappear from your inbox for 24 hours (or any number of days)? The To-do Excuse: eliminated.

And what if, when you sent an email or replied to one, you could have your own message re-ping you, i.e., appear in your inbox after a certain amount of time if the recipient failed to follow up? This, by the way, is one of the Getting Things Done (GTD) principles: have systems you can fully trust to remind you of things so you can put them out of your mind in the meantime. With luck the other person will deal with your message or respond in some way that helps move things forward. If not, the conversation will reappear in your inbox and you’re back where you started. The Waiting Excuse: eliminated.

Finally, what if you had a simple way (just a couple keystrokes) to, while reading a message, give it an expiration date, i.e., specify a number of days after which it should automatically leave your inbox and archive itself? The Laziness Excuse: if not eliminated, at least alleviated.

Nitty Gritty

Even if I’ve convinced you in the abstract that these are good ideas, they hinge on the interface being very quick and easy. So how would they actually work?

Snooze (eliminates the To-do Excuse; makes it easier to Defer)

For any message, type “sN” where N is a number and the message will disappear from your inbox and reappear in N days. If N is omitted the message is snoozed for 24 hours. (Fractional days are allowed, eg, “s.5” snoozes for 12 hours; “s.1” for 2.4 hours; etc.) This is also nice for reminders: send yourself an email and then snooze it.

Anti-procrastination tweak: It’s possible you’ll just keep snoozing a message indefinitely. Maybe that’s ok — at least it keeps being brought back to your attention, solving the out of sight, out of mind problem. But if it’s a problem, the snooze button could force you to double the snooze length each time you snooze a message. That way you will at some point not be able to snooze it again without admitting that you’re essentially never going to deal with it.

Re-Ping (eliminates the Waiting Excuse; makes it possible to Archive/Delete immediately after a Delegate or Respond)

Have an additional email header along with To, Cc, Bcc, and Subject for outgoing mail:

  Re-Ping: N

In N days, if there’s no reply yet, your own email shows up back in your inbox, and you can reply to it to nudge the person with a “just making sure you got this”. This could also be treated as just another form of snooze: “Put this conversation back in my inbox in N days if no new replies show up before then.” Again, when you get an email on an important topic, relating to a work project perhaps, you tend not to archive it even after replying because you need to make sure the thread doesn’t get dropped. By specifying a re-ping you can confidently archive the email after replying since the thread is now guaranteed not to end up out of sight, out of mind.

I propose this as a header because I think it should be part of composing an email and should work identically whether it’s a reply in an existing thread or a brand new message.

Auto-Expire (alleviates the Laziness Excuse; less inbox slogging)

Type “eN” and the message will be automatically archived in N days. This is roughly the inverse of Snooze. Alarm clocks sometimes call this a Sleep feature where you can have the radio play for a certain amount of time and then automatically turn off. Here, if you get an email about some event in a couple weeks you can say “auto-expire 14” and it will stay in your inbox for 2 weeks, and then silently, automatically go away. It’s not that it takes much more time to clean up no-longer-relevant emails from your inbox once in a while. But when you get the email is when it has your attention. If you decide to leave it in your inbox you might as well schedule the archiving you’ll have to do in a couple weeks anyway.

It might even be valuable to cultivate the habit of setting an auto-expire for every message as you read it.^[4] Auto-expire 0 is the special case of archiving right now and auto-expire 999 (i.e., almost 3 years) would be the equivalent of choosing to just leave it in your inbox. Most messages probably have a much shorter horizon of possible relevance. If done right, you might never have to do any painful inbox housekeeping.

Conclusion

These features — Snooze, Re-Ping, and Auto-Expire — offer a quick way to confidently get stuff out of your inbox. Archive+search is a huge step in that direction, but many of us clearly need more than that. You might complain that Snooze and Re-Ping only temporarily remove things from your inbox, but it’s better than that. First, if you snooze everything till closer to when you’ll be acting on it, that alone shrinks the average size of your inbox a lot. Second, it will often be the case that when the message reappears it will be much easier to decide to just archive it. Maybe you delegated something and it actually got done. Or there was some event coming up but now that it’s closer you know that you have other plans.

The Snooze and Re-Ping features allow you to almost instantly get a message out of your inbox and out of your mind, while still having full faith that it will re-appear in time for you to take necessary action. Auto-Expire spreads out the work of inbox housekeeping. It helps keep down the size of your inbox to the extent that you put off the task of archiving messages that are no longer relevant. Whether this actually increases efficiency is an empirical question but the rationale is that it saves having to open up and think about a message twice. While you’re reading an email about a conference coming up in a couple months you can set it to auto-expire in 90 days. You’ve thus ensured that soon after the conference is over, that email will no longer be cluttering your inbox — with zero further action from you.

The only one of these I’ve actually tried so far is Snooze, which I hacked up in Pine (that’s right, I mainly use Pine for my personal email). I do sometimes find myself resnoozing indefinitely, so I probably need the anti-procrastination tweak mentioned above. But I believe (hope) that by adding Re-Ping and Auto-Expire I will finally achieve Inbox Zero Nirvana.

Thanks to Melanie Reeves, David Yang, Michael Tiffany, Patrick Jordan, Dave Pennock, Kevin Lochner, Mohammad Mahdian, Ed Fu, Sharad Goel, and Samson Yeung for reading drafts of this. I’ve also gotten some great comments on Hacker News. Thank you!

Image by Kelly Savage.

PS: Since writing this I’ve learned of two relevant startups: Etacts.com is trying to solve the Re-Ping problem, among other things. And HitMeLater.com implements email snooze. I would prefer these to be simple client-side features though.

Appendix: Notes on How to Hack Snooze, Re-Ping, and Auto-Expire into Email Clients

Snooze

When you hit sN, where N is a real number, a client-side script applies the label “snooze-N” (or puts it in a folder called such) to the message and removes it from the inbox. A daemon checks all such labeled emails every so often and removes the label and puts it back in the inbox (applies the “Inbox” label) when the email’s date is >= N days in the past. Note that this means a proliferation of labels/folders of the form “snooze-N”. An open question is whether, when putting something back in the inbox, to also mark it unread. If you’re succeeding in keeping your inbox very small then this would be unnecessary.

Re-Ping

There are two parts to the hackery here: creating the additional field, “Re-Ping:”, in the compose headers, and keeping track of whether there has been a reply. It should be fine if the Re-Ping header is included in the actual outgoing message. For the actual re-pinging, a partial solution in Gmail is easy due to its thread-centricness. Don’t actually check if there was a reply, just apply the Inbox label after the specified amount of time. If a reply is already in the inbox, fine. This works because a whole thread (conversation) is either in the inbox or not. In fact, all labels are applied to threads, not individual messages.

The only thing missing with the above hack is that you may have archived the conversation after a reply came in but before the re-ping time. In that case you don’t actually want the conversation to reappear in your inbox. So to do this right you have to actually keep track of each outgoing message with a re-ping set and do the re-ping conditional on no reply.

Auto-Expire

Implementation-wise this is basically the inverse of Snooze. Label the email “auto-archive-N” and remove it from the inbox at the appropriate time.

Footnotes

[1] I’ll always archive and never delete a message, simply because the cognitive cost of deciding that I’m certain that I’ll never need to look at it again is too high. Consider the economics: How much time will you save, cumulatively, by just always hitting “archive” without thinking about whether you might ever need to look at it again? And how much will it cost you to have your archive folder full of junk? The additional storage probably costs nothing, and if your email client’s search feature is decent it won’t slow down your searches either. In other words, there’s only upside. And that doesn’t even factor in the chance of accidentally deleting something important!

[2] If you sometimes delay responses so as not to seem overeager, or because you want your reply to arrive during business hours, or any number of other reasons, check out this clever idea. I think my proposed Snooze feature also partly solves this problem.

[3] I’m anti-folders for the same reason I never delete anything. I don’t want to waste time deciding what to do with a message that I might never look at again. Everything goes in one folder and that’s where I go to find anything.

[4] It would be even better if people set expiration dates on their own outgoing mail, but I can’t think of a way to get that convention to catch on.

How well do you know your friends’ political views? According to recent work by Winter Mason, Duncan Watts, and myself, you probably don’t know them as well as you think. In particular, we found that when friends disagree on a political issue, they are unaware of that disagreement about 60% of the time. Even close friends who discuss politics are typically unaware of their differences in opinions.

Homophily, the tendency for individuals to associate with similar others, is one of the more consistently observed phenomena of the social world. While reports of homophily have focused on socio-demographic attributes—like age, gender, and race—it’s reasonable that homophily would extend to attitudes as well, in part because individuals likely seek out those who agree with them, and in part because social pressure often breeds conformity. However, measuring the extent to which friends agree with one another is a tough empirical problem, and quantifying the differences between actual and perceived homophily was, until recently, prohibitively difficult.

Enter Facebook, the social science experimentation platform with a subject pool of 400 million strong (and growing). To examine the differences between real and perceived attitude agreement, we built a Facebook game where players both answered a series of questions about their own attitudes, and attempted to guess how particular friends would respond. We collected 12,000+ responses in which we had a player’s answer to a given question, their friend’s answer to that same question, and the player’s guess as to what their friend would say.

The figure below shows that people consistently overestimate the likelihood that their friends agree with them on political issues. Notably, even though close friends (so-called strong ties^[1]) are in reality more likely to agree with one another than distant friends, people do not appropriately adjust their perceptions. In other words, though we think close and distant friends are about equally likely to agree with us on political issues, in reality we are much more likely to agree with close friends.

Upon reflection, it’s perhaps not so surprising that we often don’t know where our friends stand on hot-button political issues. How many times have you actually asked your friends how they feel about abortion, capital punishment, or affirmative action? Inferring friends’ attitudes is further complicated by the fact that political positions tend to be only weakly correlated across issues. For example, according to the 2008 General Social Survey, 65% of people who support abortion rights also support capital punishment, compared to 68% among those who oppose abortions. That is, someone’s position on abortion tells you nearly nothing about their view on capital punishment.

I think society would benefit from friends and neighbors engaging in more active political deliberation, discussions that at the very least leave participants aware of each other’s views. On the other hand, if your friends really knew what you thought, you might have a lot fewer friends.

N.B. For more details, check out our paper.

Illustration by Kelly Savage

Footnotes

[1] We define the tie strength between a pair of individuals by considering the number of mutual friends that they have, and whether or not the pair discusses politics.

The idea of rot13 is to obscure text, for example, to prevent spoilers. It’s not meant to be cryptographically secure but simply to make sure that only people who are sure they want to read something will read it.

We, being pretty extreme nerds, sometimes find ourselves wanting to do something similar for numbers. This is typically in the context of a sealed-bid auction where you want to submit a bid that the other party can unseal whenever they’re ready, with no further coordination needed. Most recently, Sharad and I had to decide who would be presenting our Prediction Without Markets paper at the upcoming ACM Ecommerce conference. To determine who wanted to present more (i.e., who minded least), I sent him my sealed bid (how much I would pay him, at most, to present) and told him to unseal it when he had committed to his own bid. (Eliminating the need for further coordination was especially useful here because Sharad was about to get on a plane. It’s similarly handy when coordinating by text message when one or both parties are about to get on the subway.)

The outcome in that case was that Sharad bid lower and so I paid him that lower bid to be the one to present our paper.

In general what we want is a way for me to send you a number and trust you to pick your own number, uninfluenced by mine. And then you should be able to reveal mine when you’re ready, without requiring further input from me or any third party. Note the assumption that the recipient is being trusted not to cheat.

So how do we do that? It’s not as simple as rot13 because certain numbers, like 1 and 2, will recur often enough that you might remember that, say, “gjb” is really 2.

To be more specific, we seek a function, seal(), that maps a real number to a real number (or a string). It should not be deterministic — seal(7) should not map to the same thing every time. But the corresponding function, reveal(), should be deterministic — reveal(seal(x)) should equal x for all x.

For example:

   > seal(7)              # Some random-seeming
   429964                 # number or string.

   > seal(7)              # A completely different
   749932                 # random-seeming number.

   > reveal(seal(7))      # Revealing always uncovers
   7                      # the original number.

Can you think of a way to implement seal() and reveal()?

How to Seal and Reveal a Secret Number

There are many ways, it turns out, to accomplish this but here, in lay terms, is an elegant scheme that can be carried out easily with a pocket calculator:

Pick a random number (from your head is fine) between 9 and 99 and multiply it by 9999, then add your secret number. This will yield a 5 or 6-digit number that encodes your secret number. To decode it, divide by 9999, subtract the part to the left of the decimal point, then multiply by 9999. (This is known to children and mathematicians as “finding the remainder when dividing by 9999” and “mod’ing by 9999”, respectively.)

This works for nonnegative numbers less than 9999 (if that’s not enough, use 99999 or as many nines as you want). If you want to allow negative numbers, then the magic 9999 number needs to be twice the biggest possible number. And when decoding, if the result is greater than half of 9999, i.e., 5000 or more, then subtract 9999 to get the actual (negative) number.

Again, this is not a true commitment scheme. But for cases where the honor system suffices and the only goal is to not be influenced by each other’s number, we find this quite handy.

Pseudocode

   M = 9999  # Numbers between -M/2 and M/2 can be sealed.

   seal(x): M * randInt(9, 99) + x

   reveal(x): m = mod(x, M);
              if m > M/2 return m - M else return m

Image by Jeff Moser.

Thanks to Svante v. Erichsen for the idea behind this scheme, and to a dozen other Stack Overflow users for alternative schemes.

How do you estimate the prevalence of HIV among drug addicts? In a perfect (though somewhat Orwellian) world you’d have a giant list of all drug users, and you’d randomly select a subset of them to test for HIV. Alas, the world is messy, and we’re left to less rigorous methods, for example time-location sampling, where surveyors recruit drug users from the local “shooting gallery” (heroin users’ version of a crack house) at randomly selected times.

A few years ago, sociologists Doug Heckathorn and Matthew Salganik proposed a new approach to surveying these so-called hard-to-reach or hidden populations: respondent-driven sampling. RDS is a variant of snowball sampling—where existing study participants recruit the next wave of participants, usually in exchange for payment—with a few clever tweaks. Matt and Doug realized that popular drug users (i.e., those with a lot of friends) are disproportionately likely to be in the final snowball sample, for the simple reason that popular users know more people who could potentially recruit them. In RDS, recruits are thus down-weighted by the number of people they know. In just a few short years, RDS has already been applied in more than 20 countries, and it is currently used by the Centers for Disease Control and Prevention (CDC) to help track the HIV epidemic in the United States. Now for the bad news: Matt and I just wrote a paper showing that RDS may not be suitable for key aspects of public health surveillance where it is now extensively applied.

Our critique of RDS boils down to a simple empirical reality: outcomes are typically not typical. To give an example, while the mean household income in the United States is around $60K, half of households earn either less than $30K or more than $100K. In other words, households typically have incomes that are far from typical. So what does this have to do with RDS? The trick of down-weighting popular drug users guarantees—albeit under strong assumptions—that RDS will on average yield the true infection level. (In statistical parlance, RDS is an asymptotically unbiased estimator.) In the RDS community, this theoretical property was widely interpreted as meaning that RDS is a generally reliable method of estimation. What we show is that even in situations where RDS estimates are correct on average, these estimates are typically so far from the truth that meaningful statistical inference is difficult. To clarify the point, suppose we estimate the mean age in New York by sampling one person uniformly at random. Even though our estimate would be perfect on average (i.e., unbiased), it could hardly be called accurate. Since RDS participants tend to know, and hence recruit, people who are similar to themselves, this caricatured example is actually not so far from the truth about why RDS estimates are so volatile.

I suspect that this tendency to neglect the variance also plays a role in discriminatory attitudes. Consider the following statistic, culled from the 2008 General Social Survey: white Americans on average watch 2.8 hours of television daily, compared to 4.4 hours watched on average by black Americans. The difference in group means is large, both on a relative scale—blacks on average watch nearly 60% more television than whites—and on an absolute scale—1.6 hours a day. (Citing a similar statistic, conservative radio host, and Rush Limbaugh protege, Michael Medved has pontificated on the subject.) Now, if you’re walking down the street and you see a random black person and a random white person, how likely is it that the black person watches more TV? Given the large group differences, you might be tempted to conclude it’s pretty likely. As it turns out, however, individual variation is so large that it’s basically a coin flip (about 60%) as to who watches more TV—in other words, race is a poor cue. The plot below makes this phenomenon visually apparent: despite the substantial difference in group means, whites and blacks have distributions of TV watching that are tough to distinguish. Just as in RDS, the means are misleading.

NB: For more details on RDS, check out our paper.

Illustration by Kelly Savage: A stylized depiction of an RDS recruitment tree, where nodes correspond to sample members and links indicate who recruited whom. Based on an RDS study of drug users in New York City, presented in Abdul-Quader, et al, 2006.

218 of you took our calibration quiz, not counting the 10% of submissions that had to be thrown out for not being complete or giving ranges with the min greater than the max or other sanity check failures. (Here’s the raw data.)

The bad news is that you’re terrible at making 90% confidence intervals. For example, not a single person had all 10 of their intervals contain the true answer, which, if everyone were perfectly calibrated, should’ve happened by chance to 35% of you. Getting less than 6 good intervals should, statistically, not have happened to anyone. How many actually had 5 or fewer good intervals? 76% of you.

Here’s a histogram of the number of good intervals you got, out of 10:

The overlaid phantom histogram is what it would look like if it were really the case that every interval people gave had a 90% chance of containing the true answer. In other words, you should’ve made your intervals much wider. When we ask for a 90% confidence interval there’s in fact only a 41% chance that your interval contains the true answer.

We ran this quiz on Mechanical Turk as well and you marginally outperformed the turkers. The histogram of turkers’ good intervals is indicated by the red dots in the above graph. They failed our sanity checks at almost twice the rate (19%) of Messy Matters readers and of the remaining responses, the mean number of good intervals was 3.5 out of 10.

The more we’ve thought about (and read the literature on — or rather, consulted endlessly with Dan Goldstein, who knows the literature on) these kinds of overconfidence results, however, the less clear it is that the moral of this quiz is simply “people are overconfident”. For one thing, overconfidence depends on the question. The fraction of good intervals in your responses ranged from 23% (the length of the Nile and the gestation period of an Asian elephant) to 75% (number of OPEC countries). Of course, even 75% is not the 90% that was asked for.

More interestingly, in an ongoing follow-up study on Mechanical Turk we’re finding that after you get people’s intervals, more than half of them realize in retrospect that too few of their intervals are good. This suggests that people can learn to perform much better at this task.

Obligatory Wisdom of Crowds Demonstration

It’s not a fair demonstration since people weren’t asked for their best guesses, but here’s a table of median lower bounds, upper bounds, and midpoints of everyone’s ranges. Interestingly, people’s upper bounds are overall most accurate.

Thanks to Sharad Goel, Dan Goldstein, Bethany Soule, Dan Kaminsky, and Michael J.J. Tiffany.

Image: Kelly Savage

We shall now find out if Messy Matters readers are smarter than Mechanical Turkers. For each of the questions below, provide a numerical range that you are 90% sure contains the correct answer. In particular, if you have “no idea” then give a very wide range; and if you happen to be quite certain then give a narrow range.

If you are perfectly calibrated then 90% of your intervals (no more, no less) should contain the right answers.

Please try your best, but don’t look up the answers!

(If the quiz doesn’t show up in your news reader, click through to the post on messymatters.com.)

We’ll discuss the results in the next post.

PS: Here are the results.

Quiz from Decision Traps by Russo and Shoemaker.
Image by Kelly Savage.

In the 2008 Summer Olympics Usain Bolt ran 100 meters in 9.69 seconds, earning the gold medal and garnering the international attention that comes with being the “fastest man in the world.” While Bolt became a household name, his competitors didn’t fare nearly as well: far fewer people know that Richard Thompson and Walter Dix received silver and bronze, and I suspect that 8th place Darvis Patton is practically unknown outside the sprinting world. The 340 milliseconds that separated Bolt from Patton—the duration of a blink of the eye—was the difference between celebrity and obscurity. While a fascination with rank is perhaps justifiable for sports, such focus—let’s call it the Gold Medal Syndrome—is often problematic in statistical analysis.

Consider the case of prediction markets. In these markets, participants buy and sell securities that realize a value based on the occurrence of some future outcome, such as the result of an election, the box office revenue of an upcoming film, or the market share of a new product. For example, the day before the 2008 U.S. presidential election you could have paid $0.92 for a contract in the Iowa Electronic Markets that yielded $1 when Barack Obama won, implying a 92% market-estimated probability that Obama would win. There are compelling theoretical reasons to expect prediction markets to outperform all other available forecasting methods. As formalized by the efficient-market hypothesis, if there were some way to beat the market then at least one savvy trader would presumably exploit that advantage to make money; hence, market prices should update to eliminate any performance disparity.

Inspired by such theoretical arguments, and also by a growing body of empirical findings that show markets beat alternatives, several researchers have called for widespread application of prediction markets to real-world business strategy and policy development problems. In a 2007 Wall Street Journal op-ed, economists Robert Hahn and Paul Tetlock write:

“Imagine the president had a crystal ball to predict more accurately the impact of broader prescription coverage on the Medicare budget, the effect of more frequent audits on tax compliance—or even the consequences of a political settlement in Iraq on oil prices. Now, stop imagining: Such crystal balls [prediction markets] are within our grasp.”

The work on which these appeals are based, however, primarily addresses the relative ranking of prediction methods. By contrast, the magnitude of the differences in question has received much less attention, and as such, it remains unclear whether the performance improvement associated with prediction markets is meaningful from a practical perspective.

In a new study, Daniel Reeves, Duncan Watts, Dave Pennock and I compare the performance of prediction markets to conventional means of forecasting, namely polls and statistical models. Examining thousands of sporting and movie events, we find that the relative advantage of prediction markets is remarkably small. For example, the Las Vegas market for professional football is only 3% more accurate in predicting final game scores than a simple, three parameter statistical model, and the market is only 1% better than a poll of football enthusiasts. The plot below shows how the three methods perform on the complementary task of estimating the probability the home team wins.

Given that sports and entertainment markets are among the most mature and successful, our results challenge the view that prediction markets are substantively superior to alternative forecasting mechanisms. Nevertheless, it is certainly possible that there are forecasting applications where either the relative advantage of markets is larger, or that such differences in performance are consequential. Thus, while prediction markets may yet prove to be useful, it would seem the enthusiasm for their predictive prowess has outpaced the evidence.

NB: Check out our paper for more details.

Illustration by Kelly Savage

As a poser economist (as Jeff Ely calls me), I love to complain about the social inefficiency of gift giving. It’s a terrible idea, guys! We waste 13 billion dollars a year on it! ^[1] But I do appreciate that I have a tendency to pooh-pooh social conventions far too blithely. Nonetheless, social conventions change. Drastically so, given enough time. As Al Roth points out, wedding registries, gift receipts, and, more regrettably, gift cards represent a shift towards cash as gifts, and thus greater efficiency.

But as much as I applaud this trend and agree that holiday gift giving makes us all poorer and generally worse off, I’m sure Joel Waldfogel’s estimate (that the value we get from gifts is 25 cents on the dollar) is overblown. So, in the spirit of the holiday season, here are some reasons why gift giving isn’t quite so bad as it seems:

1. Ulterior Motives. When put that way it doesn’t exactly sound holiday spirited but I think ulterior motives is the most overlooked and perhaps most legitimate justification for gift giving. If I give you a ticket to see a show with me because I really want to see it with you, then it really doesn’t matter how much you would’ve paid for that ticket yourself. I wanted your company and this is a socially acceptable way for me to pay for it. ^[2] Homer Simpson giving Marge a bowling ball (or a power screwdriver or somesuch) is another example. Nothing’s wasted if Homer gets enough use out of those gifts himself! A more common example is giving one’s girlfriend lingerie. But the ulterior motives need not be selfish, exactly. It might well be socially efficient for me to buy you a book you wouldn’t have bought, if I have a lot of value for discussing it with you.
2. Signaling Value. I don’t pretend to understand the intricacies of this, but presumably it’s socially useful to prove our thoughtfulness. We may also find it socially valuable to be mutually indebted to each other (and in ways hard to quantify).
3. The Ritual. The process itself may have value to us, including the process of figuring out each other’s desires well enough to buy gifts.
4. Extrospection. In rare cases you might actually know the recipient’s utility function better than they do. Usually this is ill-advised, from an efficiency standpoint. Why not just recommend your new favorite music to your friend and let them decide whether to buy it? The one case where extrospection is a slam dunk, of course, is giving gifts to small children (all the other excuses for gift giving — except perhaps number 2 — also apply in this case).
5. Transactional Efficiency. Related to number 4, it may be easier for you to get your friend something than it is for them to get it. Maybe you’re buying a book for yourself and it’s just as easy to get a second copy. Or you may have access to something the recipient doesn’t — for example, if you’re visiting a foreign country.

The last two don’t really count, of course, since Waldfogel’s analysis accounts for those. But the first three may put a legitimate dent in that 13 billion dollar deadweight loss.

Added: Vincent Conitzer adds a great one in the comments:

6. Plausible Deniability. There are some things that you just have embarrassingly high utility for. And “hey, it was a gift!” can be a nice airtight excuse.

Addendum: Other Inefficiencies of the Season

Sharad pointed out that “after-Christmas sale” is a euphemism for “pre-Christmas price-gouging.” This is not an inefficiency if you include the utility of the retailers but from consumers’ perspective, it’s inefficient to concentrate demand around a single day of the year. Bah humbug!

I recently encountered an inefficiency related to holiday travel. Many outbound NYC flights were canceled the weekend before Christmas due to a snowstorm. People were generally rebooked a few days later. But on Monday everything was back to normal. I know because I flew out of New York on Monday. My aunt was supposed to fly out on Saturday but got stranded until Tuesday. She clearly wanted to depart on Monday more than we did. It’s a shame that there’s essentially no secondary market for airline tickets! (Though there is in a very limited sense: You can volunteer to be bumped and the airline will compensate you with travel vouchers, worth deceptively little. But naturally the airlines are not bumping willing travelers in order to rebook people sooner when their original flights were canceled.)

Footnotes

[1] That’s not a claim that we spend $13B/year on gifts. We spend 17 billion, 13 billion of which is pure waste. That’s Joel Waldfogel’s estimate of the difference between what we spend on holiday gifts and what the recipients would have been willing to spend to buy those items themselves.

[2] Alright, it sounds much ickier when you put it that way!

Thanks to Alex Strehl, Eleanor Strehl, David Reiley, Jill Sazama, Bethany Soule, and Sharad Goel for helpful discussion.

Illustration by Kelly Savage