I hate wasting time writing about libertarians, other than to mock their terrible ideas (Exhibit A: Ayn Rand), but since they have billioinaires like the Kochs carrying their banner and we increasingly live in a plutonomy where such individuals matter much much more than millions of others, well it has to be done. Liberals need to understand libertarians enough to at least write them off as right wingers and stop waiting for the social issue fairy to bring them on board to support us for anything more than the most rote and prosaic single-issue half-hearted support. The Terri Schiavo mess wasn't enough to drive the libertarians out of the GOP, Obama's opposition to DOMA damn well isn't going to draw them over either.
To start the answer to this, we need to address the widespread misunderstanding of how the ideological map of humanity looks. You've probably seen something like this (taken from Political Compass for discussion):
I would prefer to draw it like this:
I tried to keep the axes titles value neutral/positive such that people in those quadrants would be comfortable with the terms, but you can substitute other terms if you like. Likely if you've spent any time on ideology, you've encountered such things before. This does propose a solution for how to think about libertarians and would seem to account for their general agreement with liberals on social questions, and with conservatives on economic ones. One serious problem is what to put in that fourth quadrant? What do we call people who are economically in favour of equality, but socially restrictive or hierarchical?
The usual answer for this quadrant is to call them "communitarians." The only group I can think of that might fit are the Amish, and being no expert on them, I could be totally wrong on that. If you go looking, you can find some evidence that actual communitarians exist, but outside of a political science or philosophy class, I can't think of any time in which they've managed to achieve political relevance in wider society.
I think this is suggestive of what the problem is for libertarians. See, they're plenty noisy and it isn't difficult to find people calling themselves libertarians, particularly on the internet, but they remain inconsequential in polling and voting. There just aren't very many of them and if it weren't for a few of them being very rich and using that money to exert disproportionate influence, I doubt we would ever spend time discussing them.
This is why I still generally favour the simple left-right line for most political analysis. You can draw a political space in 2 dimensions with 4 quadrants, maybe even one in 3 dimensions with 8 octants. Great theory. But what do most actual humans believe in? My sense is that if you gave a large cross section of adults a detailed political ideology test that would place them on the above chart, you would get something like this (each dot a person):
Stressing that isn't real data, but what it might look like that would explain what we can plainly see in the absence of communitarians and large disparity in the numbers of liberals and conservatives compared to libertarians. Yes, there are four equal quadrants but I don't think homo sapiens are equally or even normally (in a statistical sense) distributed across them. So if we try and draw the line through the bulk of the populace, you might get something like:
I have intentially shifted the line a little bit up from the midpoint of the axes to reflect the possibility that humans are a little biased toward unrestricted economics, but also a little disposed toward social permissiveness. Naturally this is all just my theory, but it would explain the available data. It also allows for outliers, those individuals that some theorists like to throw out as proof the simple spectrum-line cannot be correct. Sure, so Glenn Greenwald isn't a conventional liberal on some issues, or maybe you have an uncle who wants higher taxes on the rich but protests outside abortion clinics. Such people probably do exist, but still the left-right line would still be able to describe the vast majority of the populace closely enough without needing to get bogged down in the differences between libertarian and "libertarianism" to explain a few exceptional individuals.
I won't go to the wall for the distribution I've proposed, only the general point that I think there is no reason to assume that humans fall uniformly across whatever ideological chart you want to propose. Some other stuff is going on whether from evolutionary biology or predominant cultural norms that compresses the field into something that can be described pretty well with a line.
So where does this leave us with libertarians defaulting to support conservatives whenever it comes down to choosing?
First, it should make it obvious that libertarians who want to influence policy will mostly conclude they have to choose one side or the other, the way most highly liberal people remain with the Democratic party even if they would rather something like the NDP existed as a viable political force. That said, why would they almost always side with their economic interests over social ones?
One possibility (quite likely) is that economic and social issues are not equally weighted for most voters. This is another problem with these multi-dimensional ideological graphs, they don't provide any means to portray that voters might value the X axis more than the Y (or Z) if forced to choose. They imply equal priority.
The other idea linked to this is that "economic" and "social" isues are not truly independent. The drive people have to attain status, importance, respect can easily lead them to pursue either social betterment or economic betterment. If you could straight up buy a Nobel prize, and be equally respected as those who earn one, a great many wealthy people would want to buy one. As suggested above, it's possible that most people would choose the Nobel prize money over a Nobel prize if they had to choose just one, but then it can't be denied that money buys a lot of respect, even if nothing more than flying first class and the like. Similarly, the social rules imposed by traditional values almost always work in favour of the already rich and powerful. The great social deference and respect expected of lower status people to their "betters" under traditional norms can obviously also be of great financial value in any number of situations. Libertarians are given a lot of credit for typically being pro-legalization of marijuana and pro-same sex marriage equality. But yet I notice that the last several (at least 3) Libertarian Party USA Presidential nominees have been personally anti-choice. Further, Ron Paul is hardly in favour of gay social equality seeing that he thinks Lawrence v. Texas was wrongly decided.
More to Ideology than Ideas
The last broad idea I want to propose has to do with another aspect that isn't captured on these ideological graphs. I might call it "meta-ideology" - a sort of second order ideology that is not related to the specific political ideas people hold, but how they think and reason. In this regard I think you find a lot more similarity between libertarians and conservatives. I have discussed this before obliquely back while explaining why libertarians today should not be confused for the classic liberals of the 19th century even as they advocate very similar ideas:
I was kinder to them back then, but we must consider the way in which libertarians actually ape the conservative penchant for deciding on an idea, and supporting it no matter what the empirical record says. The obvious example here is the very high levels of climate change denial among libertarians (Koch most definitely included). We could also discuss their support for supply side economics, and baffling love of the gold standard despite many obvious glaring logical and practical flaws with the idea, and many real world failures of it, flying in the face of their claims that it would safeproof us against crashes or inflation.Now we're ready to arrive at our answer for the libertarians: Yes, in many ways their policy preferences today map very well to the policies pushed by liberals like John Bright and Richard Cobden in the UK, Jefferson and Madison in the US or William Lyon MacKenzie in Canada. The classic liberals. However the difference is that those men did not have the extra 150 years of experience with the reality of capitalism. Libertarians have stuck to a set of beliefs that liberals abandoned because they weren't serving the true goals of liberalism. Rather than assume libertarian thinkers are unaware of this history, we must conclude that they either do not share the same goals as liberals, or lack the rational capacity to reach the correct conclusions about the empirical policy record.
There's a famous criticism of communism which went "nice idea, wrong species" and it really applies quite neatly to libertarians too. Specifically their devotion to the rational actor individual, in contravention to vast reams of psychology, sociology and even common sense. Just look at modern advertising's reliance on sexual and emotional appeals to move inferior or overpriced products. People just aren't very rational and a large part of the real economy relies on the ways they aren't. The market itself has decided that people are irrational and has developed a whole industry devoted to profiting from that! The book title "Predictably Irrational" (which I have keep meaning to read) perfectly captures this idea. Yet libertarians persist in the belief that perfect information and rationality will allow reputation to adequately regulate bad actors and socially harmful yet privately profitable behaviour in the absence of government action.
Returning to Climate Change, there's nothing about on-paper libertarian theory that should make one prone to deny that the planet has environmental limits, and our activity breaching those limits has deleterious consequences for the living things that rely on the environmental status quo (which includes us). A good theoretical libertarian should be as amenable to the evidence record as any liberal, yet simply favour different policy responses (well, maybe not, seeing as pricing carbon is about as pro-market as you can get, and if we talk about relying on "voluntary action" we're back into libertarian reliance on fantasy humans rather than real humans, where things like free rider problems evaporate and everyone voluntarily does the right thing without any coordinating mechanism via government). Yet libertarians broadly are climate deniers just like conservatives. Why? It's how they think, how they approach information that matters, not their ideology, and in this regard they're evidently just like conservatives because they reach the same conclusions (hoax, Al Gore fat etc).
Obviously libertarians aren't going to be particularly fond of how I'm characterizing them, but this bull headed devotion to an empirically dubious theory of human behaviour and motivations is something they really share with conservatives, who have a different theory but it's equally wrong and they're at least as devoted to it.
I'm a big fan of multi-factor explanations so I won't try and pick just one of the possibilities I've offered here. All of them play a role, but the sum total is that libertarians will reliably fall on the right side of the left-right spectrum most every time they're required to choose. Maybe we can postulate some alternate reality where liberalism is vanquished and the primary political debate is between libertarians and conservatives over drug laws and abortion choice, and maybe there the bizzarro-Koch brothers are devoting tens of millions of dollars to their drug legalization think tanks because there's already a mega low flat tax and no social safety net. That isn't the reality we have and so the two groups very rarely divide in politically salient ways so long as there are liberals under the bed waiting to leap out with nightmare policies like taxes on second yachts or limiting amount of mercury that can be added to baby products.