Wednesday, August 3, 2022

On Rome and America

I recently finished listening to Mike Duncan’s excellent “History of Rome” podcast series. It left me with a confusing swirl of thoughts that I wanted to put down on paper before they gradually evaporated.

One of the themes I found myself reflecting on was the Moldbug point that a government is just a corporation. Moldbug used this argument to note that the difference between how the average westerner thinks governments should optimally be structured (with democracy and separation of powers and an independent judiciary), is radically different than how they think private corporations should be structured (with dispersed shareholders who vote for a board, who appoints and supervises an all-powerful CEO). Which either means that you need to explain what exactly it is about their different tasks that justifies the different form, or  if there's no fundamental difference in tasks, at least one of the two is suboptimal

When you see the various governing forms that the Roman Empire took over time, it brings to mind the more basic question – if a government is just a special case of a firm, well, what exactly is a firm, anyway? And Ronald Coase (and Jensen and Meckling) has an answer for us here. A government, like a firm, is a nexus of contracts. That is, a government is a way for various people to coordinate their behavior for some overall purpose. Some of those people are coordinated against their will, which is an odd form of contract, but not a fundamental obstacle to the idea. And secondly, most contracts are incomplete. We either can’t perfectly commit to future actions, or can’t observe other people’s information sets enough to know if they’re screwing us (is their lousy performance because got a bad draw, or because they stopped putting in effort?) or a variety of other problems in contract theory.

And in this lens, to say that contracts are always partially incomplete is another way of saying that power is always somewhat informal. The org chart can specify who answers to whom over what questions, but unless you can actually enforce action the whole way down the chain of command, there’s always a “run it up the flagpole and see” aspect to any given order. 

So in this sense, the simplest version that solves this incomplete contracts problem is the early Roman Republic – when “the Romans were like brothers, in the brave days of old”, in Macauley’s phrase. With enough mutual love and regard, the incomplete contracts problem gets largely solved, through sacrifice, shared understanding, and desire to not maximally exploit everyone else. Government as the expression of self-determination for a well-defined ethnic group with mutual fellow-feeling thus makes sense as a reliable way of solving the contracting problem. 

This is like the highly successful family business. But a firm where you only hire family members is limited in its capital and ability to expand, unless you’re having bazillions of children each generation. This creates incentives supporting the tendency to hire outsiders, but keep control and cash flow rights to the family. So now you go from the family-only operation, to a family-owned firm employing outsiders. This can be quite stable and allow for a lot of growth. But a lot of firms find out that if you really want to expand to take advantage of profitable opportunities, you may need to dilute control by selling shares in capital markets, and hiring competent external managers instead of just your eldest son, or even your most competent son. 

At each step, the challenge is figuring out how to coordinate the increasingly complex nexus of incomplete contracts that holds the whole thing together. And for the case of governments, this is an order of magnitude harder than for private firms, as there’s no larger governing apparatus or courts to appeal to in order to resolve disputes. You can get some progress due to personal loyalty, or traditions being a Schelling Point for behavior, or displays of force making things collapse to a single Nash equilibrium. But if people disagree enough on what the contract requires in a given high stakes situation, maybe the nexus itself breaks, reverting to smaller groups with less ambitious aims who can still agree on things. Assuming, that is, that those groups can actually operate independently of the broken off group. Complexity, like entropy, is easier to add than it is to get rid of again.

And this sounds a lot like the early history of the Rome. In the beginning, the Romans were literally the citizens of Rome, who were also the soldiers. And their enemies were other nearby cities – the Samnites, the Latins, and the rest. But once defeated, these become subject cities, then friendly allies, and then finally, Italians, the inner core of the expanded polity. The concept of who is “us”, at the core of the high trust solution to the incomplete contracts problem, gets expanded. And you see the genuine aspect of ethnogenesis – what exactly makes a people, anyway? This concept has been so abused by blank slate progressives advocating for open borders (where "there is some flexibility here" gets substituted for "there is thus infinite flexibility here"), that it’s easy to miss that there are genuinely multiple ways to construct identity. And the surest way you can tell this is that the old group names just stop appearing in the stories at a certain point. We stop hearing about the Samnites, then we stop hearing about Hispania, then finally we stop hearing about “the Gauls” (as a people) and just hear about “Gaul” (as a province). 

The flip side of this expansion, however, is that it becomes harder and harder to articulate an operating principle of what and whom the whole thing is for, especially in terms of the question of who gets to be in charge and who gets to enjoy the benefits. The Roman consuls first have to be Patricians, but then that gets discarded, and they can then be wealthy Plebs too, but they still have to be Roman. Then once it moves to emperors, they eventually can be Roman families from the provinces, like Trajan, but still generally of Roman stock. These expansions prove successful, then you get the series of great Ilyrian emperors, like Diocletian. But then you get peasant emperors like Maximinus Thrax, and then you get Gothic generals who wield lots of power over puppet emperors, but still do good service to the empire (like Stilicho). But then this gives way to barbarian generals like Ricimer, who notionally still serve the decaying empire successfully, but historians start to argue whether he’s actually just a King. And finally Odoacer decides to make our lives easy by actually sending the purple cloak back to the Eastern Roman Emperor and saying he’s done with the whole thing, and now is just the King. Which is lucky for us, only because it makes the whole narrative simpler for “when did they meaningfully stop being Roman Emperors”. The Senate kept meeting until the 7th century, even as it was totally useless and a joke. 

And it’s not just the leaders – by the end, when the western Roman empire is fighting off increasing barbarian invasions, the legions doing the fighting are mostly barbarian too, and there’s not always a lot in the legitimacy stakes to make you prefer the notional emperor(s) from the usurper. It’s certainly not obviously the extent to which they represent “the Roman people” in a Wilsonian sense. 

If all this sounds like it becomes a confusing mess of an org chart by the end, it is. Part of this is because when we refer to this as a nexus of contracts, there really aren’t any actual formal contracts anywhere once the simple ties of ethnic kin aren’t the main driver. The closest we get is clear rules of succession, whose importance becomes very obvious. And when these get ignored, we’re left to the coordination game version of the Melian dialogue – the strong (and their nexus of contracts) do what they will, and the weak either join the winning team, or get killed. This often happens surprisingly fast, and a lot things that seem like they’re going to turn into massive civil wars end up as a total collapse by one side without a fight, or after a small initial battle, where all the supporters of one side abandon their leader to join the opposing faction, and the losing general either gets killed by his former supporters or commits suicide. Say what you will about the late Romans, but they played politics for keeps. 

If democracy has any virtues, it’s certainly not the wisdom of the people, or even the legitimacy that elections confer, but probably just the well understood rules of succession, whereby power is transferred on schedule, and the losers generally get to live out their lives in peace without being thrown in jail or killed. Then again, this was true in Republican Rome too, until one day it wasn’t. In the early parts of the story, you hear mostly about cities and structures, with the individual Romans playing less of a central role (with occasional exceptions like Scipio Africanus). By the time you’re routinely hearing about individual personalities at high frequency, you’re well on the way to the collapse of the Republic. And at first this takes the form of utilizing factions of Romans violently against other Romans, then it becomes using foreign auxiliaries to overcome Roman legions (who are themselves mixed), then it ends with the foreigners just running the show. 

In this regard, what you’re trying to coordinate with the early Roman Republic is a form of group selection, whereby members of a group all end up benefitting in evolutionary fitness terms because of cooperative action. Like ethnogenesis, progressives love this idea, and abuse the concept mightily to argue for all sorts of nonsense. But it can exist – not only in the case of a termite mound, but also in terms of the cells in your body. They engage in all sorts of group coordination so that their DNA gets transferred on. For a while, so did the Romans, and then later on, so did the Italians (at least in our metaphorical sense, though perhaps in a literal one too). But even inside your body, defection is often locally profitable without strong enforcement mechanisms. When one section of cells starts growing uncontrollably at the expense of the rest, this is called cancer, and has a lot of parallels to a rebellion in civil society. But the metaphor is imperfect, because the cancer of rebellion doesn’t always kill the host. Rather, at some point it starts to look less like the body with cancer and more like a cancer in charge of what used to be the body. The polity, unlike your body, can theoretically absorb outsiders. 

Aside from these curious aspects about the nature of government and its change and decay, the other part I found myself wondering about were the parts of the story being left out. Not as a conspiracy or anything, but just which are the noticeably important aspects that don’t seem to get much emphasis in the Duncan retelling? 

There are two big ones. The first of these is birthrates. Obviously he’s not recounting a continuous census of fertility. But in the days of the Republic, the recurring theme is always that whenever a Roman legion gets wiped out, they just raise another one and keep fighting, refusing to give in. You can see this an indomitable spirit and resolution, but it’s worth pondering the sheer logistics too – there must have been a large excess of tough military age males sitting around at any time, ready to be brought into military service. Where those males all went by the end of the empire is not well explained. The standard view is that they were unwilling to fight, and were kept from service by rich land owners who didn't want to sacrifice their workers. But it also seems quite likely that they might just not have been around in the same numbers. Either way, it’s puzzling. 

The other piece that supports the latter interpretation is that by the time you get to the Empire (where there is much more focus on individuals, so we know their life stories better), you can’t help but notice just how few children the Roman emperors seemed to have. If you’re not constrained by resources, why not have 12 children, or 20? How on earth do you reach the stage that you don’t have enough sons to continue your dynasty, when this is the single most important aspect of succession? Or only one son who's a total muppet? Weirdly, having no children at all is more understandable than only having two, as the former might just be general unlucky infertility, but the latter suggests it's possible, just not done much. I had always viewed the overwhelming knock on Marcus Aurelius as being that he broke the tradition of the “Five Good Emperors” of appointing as successor the most competent man, rather than their blood offspring. This is true, but the part that Duncan notes (which I hadn’t known) is that none of the others actually had any blood sons! Which is bizarre, when you think about it. It’s not even like they’re Henry VIII, producing unlucky daughter after daughter. Aurelius appointed Commodus, who was a disaster, but as Duncan notes, he was rather in a bind. Either he probably had to kill him, or make him successor, since letting him just hang around was likely to lead to civil war, which is much worse. Even Augustus, the most powerful man in the empire, ran out of sons. Like, just get some mistresses! Or bang your wife more! I have a feeling something very odd was going on with Roman birthrates, which I want to understand more.

The other even less remarked on piece is Roman engineering and technology. Occasionally you get snippets of the story like Trajan building the longest bridge for the next thousand years, or Caesar shocking the Germanic tribes by building a bridge to transport his whole army across the Rhine before the barbarians knew it had happened. Or the general impressiveness of aqueducts, which managed to slope ever so gradually over miles and miles to deliver water from one place to the other. 

Nobody else was doing that. So how did Roman technology advance? How late in the story did that stop happening (surely an important question to those who judge present civilisational health by the existence of iPhones). And why couldn’t anyone else match it in the same way? You realize that you have such a tiny number of primary sources, and if it wasn’t a subject that people bothered to record the history of, you’re going to have a lot of guesswork.  

Finally, comes the applications to the present. One aspect that stands out is that the ability to forecast the lifespan of the empire is complicated by the fact that battles at that time were still largely tactical and not strategic. That is, there often might be only one or a couple of large encounters that decide who comes off the best in a war between nations. And because these often turn on small decisions on the battlefield, smaller armies can frequently rout and destroy much larger ones, or a small number of surprise defeats can threaten the entire nation (like in the Punic Wars with Carthage). This means that at a lot of places, something going differently could have ended the whole Roman experiment much sooner – the Punic Wars, for instance, or the Crisis of the Third Century. In the age of strategic warfare, you might actually have a better chance at forecasting which structural forces will prove decisive if the two sides just grind it out long enough. Then again, maybe not – there’s been no shortage of American defeats to notionally weaker foes that might not have seemed structurally obvious ahead of time. 

The last one is the surprisingly low correlation between the level of domestic misery at any given time and the actual instability of the whole project. The late Roman Republic was an utterly miserable time for its citizens compared with the previous centuries, with the advent of massive Roman civil wars, proscriptions and deaths of political opponents and their supporters, and general chaos and uncertainty. But all the protagonists at this point are still Roman. You never really get a sense that there’s an Alexander of Macedon equivalent waiting in the wings to crush all the war-weary parties. Instead, Rome still seemed pretty secure from external enemies. By contrast, there are lots of periods during the Empire where the barbarians seemed to be both stronger and more coordinated, and could tip the whole thing over. Even here, it’s not an inexorable decline. By the early 5th century, things actually seem to be looking slightly brighter for a while. Yet the striking thing at this stage is that even a competent Emperor like Aetius isn’t able to actually fully reverse the losses, only slow them. It’s not long then before things go off the rails for good, at least in the west. 

If we look at the level of America’s foreign predation, we see useless and counterproductive wars, and endless third world migration, but there is nothing like barbarian armies forcibly peeling off territory, or menacing the homeland. This calculus is made a little more difficult by the fact that we are highly dishonest about what the American empire actually is, and thus what it would mean to be losing territory if you only rule indirectly. If you defined the American empire crudely as “everywhere that had a George Floyd protest in 2020” (including places like Berlin, London and Melbourne) then maybe the calculus gets a little harder. But still – in physical fights, America loses, but it loses in other people’s backyards, when the stakes for the domestic citizens are mostly pretty low. 

And in a pinch, I find myself thinking that our recent developments seem to map more closely to the late Roman Republic than the late Roman Empire. If I had to guess who Trump most closely matches too, it’s probably the Gracchi brothers. They started using populism, and in their case threats of mob violence, to get their legislation passed. This isn’t exactly Trump, but the sense of tapping into populist anger to circumvent the normal political process seemed like a good description of his nomination and victory in 2016. And the establishment backlash to unseat him, both from dubious Russiagate investigations from the civil service and Democratic party, periodic deaths and violence in political confrontations and protests becoming more normal, unusual levels of election fraud in 2020, and Trump’s final temper tantrum protest on January 6th, all fit the same pattern. A gradual erosion of previous norms, a gradual escalation by each side. The Gracchi brothers both got killed, which I don’t think will be Trump’s fate (though blog.jim still is betting on this). But the important parallel I see is that they showed a path to a certain kind of escalation that later men like Marius, then Sulla, then finally Caesar and Augustus, would both increase and take to its logical conclusion. At some point, people just realize that the Republic is dead and not coming back. It takes a long time, and a lot of denial, before that point is reached.

Sometimes, that’s just your lot, and there’s not much you can do. To modify Brad Pitt’s character in Inglourious Basterds – you don’t gotta be Niccolo Machiavelli to know that you don’t want to live through “the Year of the Four Emperors”.

But America as the late Roman Republic is actually the optimistic scenario. Not because it won’t get worse – if the metaphor holds, it definitely will. But because it might eventually get better. 

Monday, May 30, 2022

America, December 31st, 2021

[Editorial Note: I wanted to risk trying something unusual for this blog. This poem was written as my submission for Lomez's excellent Passage Prize, which I ordered, and you should too. It made the short list for the finals (yay!), but not the final prizes (boo!), nor the second round selections for the print edition (double boo!). So you might describe this as being among the worst of the worst of the best, which sounds about right to me.

The following was my introduction to the submission, which I'm not sure if I should get straight to the point and delete, but while I am a confident essayist, I am a nervous poet, so forgive the endless self-effacement:

"Hey Curtis,

Let me begin with an apology of sorts - it has been twenty years since I last wrote a poem, and I never really understood free verse. I kind of think of poetry as divided into either a) regular forms with rhyme and meter, or b) unusually personal imagery-heavy essays broken up to look visually appealing and emphasize certain pauses. I don't know if the latter is your view of free verse, however. To write the following in regular poetic form is probably outside my skill level, would take a very long time and probably would end up worse. Hence the result below. When you describe the meter of your poems, it mostly doesn't register with me, as I just breeze past this and read the sentences, which I really like. All of which is to say – I’m not sure whether this should be a poem at all, or an essay. But it doesn't seem to fit the prompt for the literary non-fiction version. So I figured I'd submit it anyway, if for no other reason than that I very much enjoyed the writing prompt to write something personal that risked being cringe." ]


America, December 31st, 2021



America, that land

That drew me in so long ago,

Is caught, pincer-like,

Between the two great forces

Of decaying empires.

The Scylla,

Of the great deal of ruin in a nation,

And the Charybdis,

That that which cannot continue, will cease.

 

I remember, when I first arrived,

Having occasion to observe,

With some regularity,

That this was a great country.

At billboards advertising

"Twenty Chicken McNuggets for $6.99".

Partly in jest, but mostly serious,

I used to remark:

"These should have the national anthem

Blaring on repeat,

Flag flying in the breeze.

This country believes in value!".

At redneck engineering videos,

Of homemade trebuchets.

At old universities,

Taking classes for free,

With famous and brilliant professors.

At the wonder that every band I loved,

Would just turn up in my town,

And play live every year or so.

At college girls that would find

My accent just cute enough,

For that 5-10% boost,

To come back to my apartment.

 

America, you have been very good to me.

 

I had one such moment recently,

At seeing the winning entries,

In a giant pumpkin contest,

At a small-town country fair.

A two-thousand-pound pumpkin!

Grown simply for the je ne sais quois!

There is still greatness,

Wonder and weirdness,

In odd corners you can stumble on.

But the next thought I had

Was realizing

Just how many years it was

Since last I had that thought.

Partly, the desensitization

Of repeated exposure.

Partly, the ingrate foreigner,

Now successful and dismissive,

Of those that helped him.

But partly, I think,

The decline that is all around us.

 

I used to joke that America

Seemed to be experiencing

The Soviet time-line in reverse,

Except it was crumbling, not strengthening.

Then some mental reflex noted

The multiplying number of epicycles,

And I wondered how sure I was,

That things didn't better match,

To the Roman empire,

Or the Roman republic,

Or the French revolution,

Or the Byzantines,

Or to many others

Of which I knew less.

(The amateur historian,

Confident in his theory,

Would do well to count

How many Chinese dynasties

He can name at all,

Then exclude those where

The mental association maps

Only to a diad,

With a name, and a phrase

Like "vase" or "pottery army".)

The confusing pattern-matching,

Where every peg is a meteorite,

Fractally weird and irregular,

And endlessly able to be rotated,

And every hole is an impact crater,

Blasted into the earth,

Chaos where a neat outline should be.

Beware the advice

Of the reactionary

Who only knows one history

Of decline and fall.

 

Decline, in one form or another,

Is on the lips of almost

Everyone these days.

A Democrat-voting work friend

Asks me if I plan

To home-school my children,

With "Yes" his obvious answer.

I responded that,

I had thought about this, and

Concluded that if there

Is not a nearby school,

Either public or private,

That I would trust to

Educate my child,

Is this actually still

The country I should be living in?

 

The answer, unspoken, lingers in the air.

 

The obvious follow-on question,

Also unspoken, is:

"If not here, then where?"

This one has no easy answer,

As everywhere turns into America.

 

But we have sailed

Very close to Charybdis,

And fail to tack towards

Scylla at our peril.

Is it really about to collapse?

Or is this just

The Twitter talking?

The outrage-bait machine,

Using my brain as

A meat puppet?

The glowing square is

Hypnotic and smooth,

And out of it pours

Misery and anxiety.

The view out my window

Is the same as ever.

 

The conditional is easy to tell.

Fussell understood it well,

Describing the prospect of death

For a soldier in wartime.

If the porridge hits the propeller:

"It is going to happen to me,

And only my not being here

Will prevent it."

This realization,

Fussell thought,

Was what drove them mad.

 

As a foreigner, I can tell you,

Woodrow Wilson was right

About us hyphenated-Americans

(For the first generation at least).

The man who would leave

His wife for a mistress,

Will abandon her, too, in turn,

When the deal's gone sour.

When it is your country,

You will fight.

When you are a stranger,

You will leave.

That is, if you can

Figure it out in time.

 

One day, just like

Niall Ferguson's bond investors,

On the eve of WW1,

You may wake up and find out

That the great deal

Of ruin in a nation

Has finally been exhausted.

 

It is not going to happen, probably,

This week, month, or year, though.

The mean decline is still slow.

The variance is alarming.

 

The young man who once left

His home, carelessly,

Not even really sure

Quite what the plan was,

Finds this an overwhelming question

Now in middle age.

 

So what to do in the meantime?

If now is not, in fact, the right time?

 

Relative to the Soviets,

Our mangled and mismatched metaphor,

We have one great advantage.

We also have an NKVD,

But no one is in charge of it.

Its ad hoc structure gives

Only loose coordination,

And since the only payoff comes

In the debased coin of status,

Our own era's commissars

Simply cannot wait

To announce themselves publicly.

"In this house we believe..."

Solzhenitsyn would have dreamed

Of foes this blatant.

 

I suspect that as things get

Inexorably worse,

The skill that soon,

Will matter most

Is knowing whom to trust,

And whom you can

Speak freely to.

 

I have found just two

Rules of thumb worth relaying.

If you have a

Sense of humor,

And if you can

Debate a point

And not take it personally,

I can likely talk about

Almost anything with you,

If I choose my words correctly.

At least today.

Maybe one day,

The backwards-winding clock

Will strike 1937,

And then everyone will be,

Guilty of something.

 

Wish as we might,

We cannot live

In any era but our own.

One must always try

To avoid the uselessness

And self-pity in

"The whining, the pleas of a coward".

 

So, what can you do with all this?

 

Propaganda succeeds, in part,

When those who disagree with it,

Are afraid to say so.

 

Dissenting openly and publicly,

Especially in the written word,

Is courting great trouble.

It is for the bold,

If you have a heart and spirit,

As firm as Solzhenitsyn,

An old testament prophet,

In this post testament world.

 

But speaking up in private,

To those you can trust,

Builds camaraderie and friendship,

The basis of all bonds that form

Incipient organizations,

Upon which revival may depend.

 

Perhaps this adds

Small brick on brick,

To the start of something new,

The Empire that grows,

From the ashes of the Republic.

 

Perhaps it serves only,

As the intellectual companionship,

Of knowing one is not alone,

In these dispirited times.

That the Soviet mental asylum,

We dissidents are placed in,

Is actually filled with the sane.

 

I may not live boldly

In many things,

But I believe in backing one's judgment

In estimations of character.

Learn how to read people,

Judiciously and carefully,

To figure out whom you can trust.

 

But to let them know

That they can trust you,

To break the higher order uncertainty,

Someone generally has to have

The courage to say something

Out of sync with modernity.

 

Might you get it wrong?

Of course you might get it wrong!

I have got it badly wrong

Exactly three times, so far.

None were fatal, thankfully.

Did you really think that there

Was some option, in this morass,

That didn't come with risk?

 

Reader, you would go

A fair way towards

Being conditionally trusted,

Knowing not much more

Than that you stumbled across this poem. 

Thursday, May 12, 2022

Living History Forwards

I have recently been working my way through the excellent History of Rome podcast series. I had been meaning to do this for some time as my previous knowledge had been mostly from another excellent series, The Fall of Rome , which I've written some thoughts from before. I realized in hindsight that I was doing things in a mistaken order. Perhaps due to my own slightly pessimistic nature and preoccupation with the current political situation, the narrative of decline and why it happened was very appealing. But before you can delve into the big picture forces (which Wyman does well in the Fall of Rome), it's helpful to understand the basic sequence of what happened when. Not only that, but you can't really hope to understand the fall of Rome unless you also understand its rise, and all the times when it could have fallen, but didn't. Going through the story made me want to go back and read all sorts of things again with a better knowledge of the events, from the Asterix comics, to Horatius at the Bridge by Macauley, to The God Abandons Antony by Cavafy, to Blog.Jim's posts on normality bias.

One of the interesting challenges when listening to the narrative is to try to invert the ex post story back into the ex ante perspective at the time - what people would have or should have thought, knowing only what they knew at the time. History is told backwards, but must be lived forwards. The simplest version of history tells things as a story, describing the important events that happened. But quickly students of history want to move from what happened, to why it happened. In the language of statistics, this means fitting the right ex-post model to the data, so you can understand what variation drove what outcome. Even this is hard to do - you might overfit the model, or select the wrong variables (and you don't get to re-run things to find out if you're right.) The causes you identify are probably there, and perhaps even contributed, but are they actually the important ones? This is a lot of the challenge of historians. But from a statistical point of view, the next step is the ex-ante one. If you'd run this same model using only the data you'd observed up to that point, even if you'd thought of the same variables, what relationship would you have estimated? Price to dividend ratios predict market returns reasonably well at long horizons ex-post, if you believe Campbell and Shiller. If you run it out of sample, Goyal and Welch say they don't.

The statistician's version of this is quite easy - just run the same model on less data, and see what it produces. So why is it so hard as a historian to do the same? Because you're not really running models, you're evaluating things according to your own judgment. This isn't a knock on the field, per se. Some bits of history lend themselves to quantification, like Robert Fogel did, but others (including a number that you really care about) simply don't. When you form your own judgment of things, it's hard not to fall victim to the curse of knowledge. That is, when you know something, it is very difficult to credibly put yourself in the position of someone who doesn't know the thing. It will always seem like things that you know after the fact should have been easy to forecast at the time, but they often aren't. 

In history, we always know how the story ends, so identifying what counts as a major event, or a turning point, or a transition, is always made with the benefit of hindsight, so as to give the most informative narrative. This leads people to make a significant mistake when translating history into their own lives. They assume that when some major shift occurs, there will be lots of signs to indicate this fact. But there might not be. Maybe what's important won't be obvious until much later. Reading about the last days of the Roman republic, one of the interesting aspects is that what in hindsight seem like important turning points. When the Gracchi brothers started using mobs of plebians as threats to get their political will, it might not have seemed that shocking. But it draws a line to Marius becoming consul seven times and leading an army into Rome to institute a reign of terror, and then Sulla being declared dictator for life. Except Sulla stepped down, and attempted to restore the Republic. You can imagine that things might have seemed back to normal then. But instead, this is described as more steps towards empire. Even Caesar Augustus, who consolidated power single-handedly more than anyone since Tarquin, kept a lot of the forms of the Republic, and only changed his status quite gradually. There was still a senate, and consuls, and praetors. To someone wanting to convince themselves that things weren't actually that different, it was probably easier than you might think. Indeed, one narrative of Julius Caesar's downfall is that he attempted to shift power to himself too quickly, and got stabbed to death by the Senate for his troubles, even after all his triumphs. This seems to suggest that the prudent strategy is probably to maintain the old forms, and pretend like they're still in operation, even as they're gradually undermined.

Which perhaps should make you wonder - has this... happened in America? Almost certainly. As Moldbug describes, America has gone through at least four versions of the Republic since its founding. The main reason people don't notice this is that they all swear fealty to the same piece of paper. But look around! Does the paper actually describe the government? If it does, why is the government so radically different, even as the paper is the same? Try explaining the CIA to George Washington, or the modern interpretation of the Commerce Clause to Thomas Jefferson. (As a party joke, I enjoy asking law students to list as many hypothetical pieces of legislation as they can that they're sure would not be justified by the Commerce clause. There's, uh, gun-free school zones? And ... hmmm, did I already mention gun-free school zones?). 

Is FDR delivering his inaugural address, which basically demanded absolute authority from Congress under threat he'd just take it anyway, and threatening to pack the Supreme Court when he didn't get his desired judgments, and serving an unprecedented four consecutive terms, breaching the 150 year old norm of only two, basically equivalent to a less violent form of a Caesar? The case is at least arguable, but you can be damn sure that you won't read this argument in your high school civics class. Was Nixon a corrupt figure that was justly impeached, or was he stitched up in a deep state coup? Also at least arguable.   

Or, to take one that's not yet a fait accompli in where it will end. You can also observe a gradual breaking down of existing norms and compromises that served to keep the parties' relationships with each other civil. The Democrats breach the previous norm that presidents basically get their Supreme Court nominations, by filibustering the eminently qualified Robert Bork. Republicans targeted Clinton, first with special prosecutors empowered to go on endless fishing expeditions (like starting out looking into dodgy Arkansas land deals and ending up looking into semen-stained dresses), and impeachment over purely process crimes like perjury when the underlying events were not actually criminal. Or the FBI illegally wiretapping the Trump campaign and Carter Page. Or Trump calling mobs to the capital to protest what he (and I) saw as election fraud. I happened to think that the January 6th mob was obviously going to be useless for anything other than theater with no coherent plan. But still, it is a notable shift from previous norms. Just like the Gracchi brothers. Maybe this is one more step towards perdition. Maybe it's just rumblings that will eventually settle down, like the secession of the plebs.  

In other words, we expect changes of government to look like America and Russia turning up in Berlin in 1945 - the game is over, and everyone knows it. But even the collapse of the western Roman empire doesn't quite work like this. One might think that when Rome gets sacked, that's basically the end. But Rome got sacked by the Gauls in 390 BC and bounced back. It got sacked by the Visigoths in 410 AD which was bad, but things still limped along. It got sacked again by the Vandals in 455 AD, by which time things were looking pretty dire indeed, but Odoacer declaring himself King was still twenty years away.

The challenge, in other words, is to be able to estimate the versions of history that could have happened but didn't, and the probabilities one should have attached to them. 

And when people imagine the idealised version of what this could be like if done well, those with a sci-fi bent will immediately think of Isaac Asimov's psychohistory. Imagine a fully worked out statistical model of psychology, sociology, and economics. Asimov's idea was that the perfect version of the social sciences should operate akin to the gas laws. The behavior of any one person is random, just like the movement of any one gas molecule. But the behavior of quadrillions of gas molecules or people is highly regular, and thus can be predicted quite well.

I am a huge Asimov fan, and found his writing highly influential in my teenage years. But the more I've pondered it, the more I think the idea of psychohistory has a tendency to lead people badly astray as to what ought to be possible, even in theory.

The version of psychohistory in the first foundation novel starts out with a version that presents the science as statistical, in the sense of assigning probabilities. 

Gaal said, "Indeed? In that case, if Dr. Seldon can predict the history of Trantor three hundred years into the future -"

"He can predict it fifteen hundred years into the future."

"Let it be fifteen thousand. Why couldn’t he yesterday have predicted the events of this morning and warned me. -No, I’m sorry." Gaal sat down and rested his head in one sweating palm, "I quite understand that psychohistory is a statistical science and cannot predict the future of a single man with any accuracy. You’ll understand that I’m upset."

"But you are wrong. Dr. Seldon was of the opinion that you would be arrested this morning."

"What!"

"It is unfortunate, but true. The Commission has been more and more hostile to his activities. New members joining the group have been interfered with to an increasing extent. The graphs showed that for our purposes, matters might best be brought to a climax now. The Commission of itself was moving somewhat slowly so Dr. Seldon visited you yesterday for the purpose of forcing their hand. No other reason."

Gaal caught his breath, "I resent -"

"Please. It was necessary. You were not picked for any personal reasons. You must realize that Dr. Seldon’s plans, which are laid out with the developed mathematics of over eighteen years include all eventualities with significant probabilities. This is one of them. I’ve been sent here for no other purpose than to assure you that you need not fear. It will end well; almost certainly so for the project; and with reasonable probability for you."

"What are the figures?" demanded Gaal.

"For the project, over 99.9%."

"And for myself?"

"I am instructed that this probability is 77.2%."


In other words, psychohistory predicts a range of outcomes, and their associated probabilities. This sounds like something I can imagine in the hyper-competent social sciences. But you can already see the tension in the paragraph - how is it that the death of a major figure has a 77.2% probability (three significant figures!), but the model also predicts events in 1500 years without the error bars blowing up to infinity? Admittedly, the project itself was predicted to succeed with 99.9% probability, and that was (in the book) the more important driver, so maybe it's not totally inconsistent, but still.

As the books go on, the mention of probabilities barely rates a mention again. Instead, the recurring narrative of the book is how Hari Seldon, the father of psychohistory, has recorded hologram messages for people hundreds of years into the future, explaining to them that the dramatic events that just happened were all foreseen and were part of the plan. The initial tension between probability and horizon gets resolved into the more satisfying plot device of the perfect forecast.

Asimov understands the idea of model risk. In one of the plot twists (I won't give much in the way of spoilers), eventually there appears the character of the Mule - a random structural break that couldn't have possibly been foreseen. But the general pattern is that the model works almost perfectly well in forecasting at very long horizons, right up to the point that the world has a dramatic and one-off shift.

Asimov later said that he should have actually called his science "psychosociology", not "psychohistory". I actually think this is a very revealing admission, and gets to the heart of the matter. History, in the popular version, is about predicting the ex-post path of exactly what happened. When you conceive of the task as being to predict history, it suggests knowing the precise series of events that the historian could narrate. Sociology, even when it works at all, is much more uncertain in its predictions - the 77.2% chance of death version, not the precise predictions in 500 years time version. The wrong choice of name was not innocuous - it showed an ambivalence, if not confusion, about the scope of the task.

Because Asimov's version of psychohistory is fatally flawed for two reasons. One of them he should have known at the time, the other one he probably couldn't have. 

Where Asimov should have known better is that he reveals himself to be a great storyteller, and an excellent scientist, but a poor statistician. In his conception of the gas laws, he emphasized the importance of there being millions of planets in the galaxy, in order to get a sufficiently large number of humans that he felt his statistical concept would work. But he also saw that democracy simply won't scale at that level, so he imagined the existence of an emperor.

The problem is that Asimov misunderstood the statistics behind the gas laws. The crucial factor that enables prediction is not that you have Avogadro's number of molecules. Rather, the crucial thing is that the molecules are essentially independent. This, not sheer number, is the crucial thing that makes the individual noise cancel out. If things aren't independent, you can keep adding more and more observations, and it won't help you. If one molecule is the emperor, it doesn't matter how many subjects you add.

And human beings simply aren't independent. Indeed, his own conception reveals this. If the Emperor has any actual power at all, then they're susceptible to being laid low by a bacterial infection, or killed from falling down stairs, or having a bad night's sleep due to some weird dream on a crucial day, or a thousand other random and idiosyncratic events that no model of psychohistory will ever be able to capture. The only way it can work is if the emperor is in fact not an emperor at all, and 100% of his choices, literally every single thing that matters, are already pre-determined by impersonal forces. If he dies, he will be replaced by someone else who will then do the same.

As long as the great man theory of history has even a kernel of truth, which it surely does, your chances of making eerily accurate hologram images for 500 years' time goes roughly to zero. You knowledge will only ever be probabilistic, and its accuracy will decline with time, like almost every statistical model. In many ways, this is the big danger of inferring things from fictional evidence - if your premises are subtly incorrect, you'll still write the whole story as if they were right. 

All of which might make you wonder - why wasn't this mistake obvious to Asimov at the time? 

I think this gets to the second part, which Asimov probably couldn't have known. Specifically, he was writing in the 1940s and 50s, before the age of cheap computing power and large, easily available datasets. That is to say, Asimov almost certainly had no experience actually constructing and testing statistical models. He couldn't have! Unless he was inverting the matrixes for the OLS estimator by hand. 

And as a result, he missed out on the single most important lesson you get from actually testing quantitative models. Namely, you find out how often your intuition about the world is just completely wrong. Or the effects are kinda-sorta there but much weaker than you thought. Or you start to worry about which of the many variations on some predictor variable you should be coding up, or whether there might be data errors, or whether linear models really are the right choice here, or whether there's reverse causality going on, and a thousand other things. You learn, in other words, that predicting almost anything you actually care about is surprisingly hard. And the work of doing so doesn't look at all like psychohistory, where one mathematical genius comes along, and suddenly you've got perfect predictions. Rather, it's about the slow grind of finding one new variable to improve the R-squared, or some new estimation technique to get the mean-squared error down. Sometimes a new discovery improves things a fair bit. But you never have a sense that you'll get the R-Squared to one, until your number of predictors equals the number of observations, at which point you've played yourself, as the Beastie Boys put it. Indeed, you quickly hit the law of diminishing returns on this type of thing. Initially, you add the large, big picture effects that have the most predictive power. But then what's left over is increasingly random and noise-driven, coming from outside forces and quirks mostly orthogonal to what you're interested in studying. Like Zeno's paradox, you might get closer and closer, but there's no hope of getting to the final goal. 

Even this is in the idealized version! Often you come away just convinced of your own deep epistemic uncertainty about the universe. You'll never really even perfectly explain what happened in a dataset, let alone forecast out of sample, because the world is just a shockingly complicated place. And all the bits you leave out end up in your residual term.

Nobody who has ever run a regression could really believe in psychohistory, if they thought about it hard. But many, (like me until very recently), suspended their disbelief in the face of the wonderful story, and just didn't think about it. But when you do, you realize it's just not how prediction works. Not in practice, and without independence, not even in theory. Models don't fail just because the Mule comes along. They fail because the task itself can only be statistical and uncertain. 

But if you never actually run these tests, you'll evaluate your theories of history on a heuristic basis, and make all sorts of kludges and exceptions, and be surprised when the world doesn't work out with as much certainty as a history book. 

What does genuine, significant, high stakes predictions actually look like, in the heat of the moment? Before you actually know how it's going to go down? 

When Russia first invaded Ukraine I spent an inordinate amount of time trying to figure out what the probability was that this would turn into a nuclear exchange. Because if the probability were non-trivial, it was time to get out of America, for at least weeks, to see how it was going to pan out. The important question is not "was it going to happen". Rather, the better question was, and is, "what would be the trigger events that I could observe at the time that would indicate a significant increase in the probability of dramatic escalation?" 

My guess was that the highest probability path to large scale war between Russia and America (and thus potentially going nuclear) was the US imposing a no-fly zone. Which is to say, shooting down Russian jets. The chances that this might spiral out of control, and quite fast, seems decent. Large overt NATO troop present in Ukraine would also be in the category. Other wild-card events like Poland unilaterally sending troops might also could, but it's harder to know where that goes. 

As it turns out, thankfully none of this seems likely any more. Whether it was ever likely is a separate question, but the identification of trigger events doesn't hinge on this question hugely, except for the question of whether the mental exercise is worth your time. Fortunately, the Biden administration repeatedly said early on that it wasn't interested in a no-fly zone, something we can all be very grateful for. But even if he had declared one, I suspect you probably would have had time to get on a plane out of America within 24 hours if you acted immediately, as things probably don't go nuclear at the first downed plane. On the other hand, it seems highly likely to me that most people wouldn't act, and would just sit there. Which is lucky really - the plane capacity to leave America each day is only a tiny, tiny fraction of the population. The plan only works if nobody else acts. But this in turn means that you need to act, and quickly, exactly at the point when everyone else thinks you're weird and paranoid. How else could it work? If you wait until the air raid alerts are being sent out, you will probably just die getting incinerated in your car, stuck in the biggest traffic jam in the soon-to-be-concluded history of the city. 

This is why forecasting things usefully ex-ante is hard. People expect there to be a big glaring sign that everyone will see. But there probably won't, at least until the historians write about you in 200 years' time.