Parties have deadlocked at nominating conventions before.
The Democratic National Convention of 1860 easily gave Stephen Douglas a majority, but party rules required a 2/3 vote for the nomination. The vote went on to 57 ballots, always with Douglas holding a comfortable majority but the southern runner-up serving as a spoiler. At this point the convention broke up into separate Northern and Southern conventions, nominating Douglas and John Breckinridge, respectively. The rest, as they say, is history.
The Democratic National Convention of 1924 split between the Protestant and Prohibitionist William McAdoo, who had the endorsement of the Ku Klux Klan and refused to repudiate it, and the Catholic, anti-Prohibition, anti-Klan Al Smith. The convention split over the issue of whether to condemn the Klan and went to a whopping 103 ballots, with neither McAdoo nor Smith able to garner a majority. The party, exhausted, eventually settled for a compromise candidate and, unsurprisingly, were thoroughly trounced at the polls by Calvin Coolidge.
The Democratic National Convention of 1968 was divided between pro- and anti-war factions. Party leaders arranged the nomination of the pro-war Hubert Humphrey, who had not even run in the primaries at all. This time, instead of an extended ballot deadlock, there were riots in the street. No doubt the sight of a party unable to maintain order at its own convention played a part in Nixon's victory (albeit narrow). The last seriously contested Democratic Convention took place in 1972 and was followed by a devastating defeat. The last convention in which anything of any significance happened was the Republican National Convention of 1976, in which Ronald Reagan mounted a serious challenge to Gerald Ford and was defeated. Ford went on to lose in the general.
Do you see a pattern here? First of all, split conventions are mostly a Democratic disease, although I suppose there is no real reason Republicans should be immune. Second, they are almost always followed by defeat in the general election. Apparently the sight of a political party committing ceremonial self-disembowelment on the national stage, though entertaining in a sick sort of way, does not inspire most people to want to entrust the country to them.
That said, messy conventions are a symptom, not the disease itself. The underlying disease in all cases is a split in the party, which is never much good for its prospects. This can mean not just its immediate prospects in the upcoming election, but its long-term prospects in its current condition. It can signal a serious electoral realignment.
In 1860 the Democrats were hopelessly split between their northern wing, which was as opposed as its southern wing to using the federal government to keep slavery out of the territories, but dead set against using federal power to impose slavery on a territory against the wishes of the inhabitants, and their southern wing, which was demanding a federal slave code in all territories, regardless of the wishes of the inhabitants. The difference proved irreconcilable and led (as we all know) to Civil War and a long period of Republican domination.
|The 1924 Presidential election|
|1928 Presidential election|
The later two examples were less damaging. In 1968, the immediate underlying issue was the Vietnam War, with the Democratic Party split between pro- and anti-war factions. This split could be relieved (at least in part) by ending the war. But there were other issues seething below the surface -- racial equality, crime, and culture war issues. The Democrats narrowly lost to Nixon in 1968. In 1972 they fully embraced the counter-culture and got an even worse drubbing than in 1924 and 1928. Richard Nixon then inexplicably assisted the Democrats by resorting to dirty tricks when he was so far ahead that he could easily have taken the high road and coasted to victory nonetheless, and they proceeded to win the mid-terms and the next election.
As for the 1976 convention, it signaled a split between the moderate and conservative wings of the Republican Party. Though defeated in 1976, the conservatives easily won in 1980 and have completely dominated the Republican Party ever since, to the extent that many of the things Reagan did would now be seen as apostasy.
The Republicans are once again experiencing a split that has been a long time in the making, this one not so much between moderates and conservatives as between the donor class and the activist class. These two factions stand united on some issues. Both are enraged at the thought of a Democrat (any Democrat) in the White House. Both hate Obamacare and any other government spending perceived to favor the poor, the young, or minorities. Both are opposed to diplomacy and compromise in foreign policy and dead set against any agreement with a hostile power short of unconditional surrender. But donors see the assault on Obamacare as simply a prelude to assault on Medicaid and Medicare; activists see the assault on Obamacare as a way of shoring up healthcare to seniors and veterans. Donors want to respond to every foreign policy crisis with war; activists prefer disengagement. Donors are moderately pro-immigration; activists are strongly anti-immigration. Donors want to roll back the New Deal; activists are its leading beneficiaries. Donors oppose all government spending except on the military; activist say they oppose government spending but make exceptions for most of the budget. Donors' top priority is to cut taxes and gut regulations; activists don't care all that much about either issue.* At the same time, activists have been dead set against any sort of negotiation or compromise in domestic politics, while donors have seen them as necessary evils. Or so it seemed, until Trump suddenly started talking about what great deals he would negotiate. Under these circumstances, it is hard to see how some sort of split could be avoided. It is equally hard to see how the split would play out. Joining the Democrats is not option for either faction.** Forming a third party is rarely more than a vehicle for handing power over to the other party.
But a brokered convention remains a disastrous option, worse than any convention split before, not merely a symptom of a party split, but a certain cause of disaster as well. The reason for this that the prior conventions, messy and ruinous as they were, were at least not violating basic political norms. It was fully accepted and, indeed, expected practice at the time that party conventions would choose the party's nominee for President. That the candidate should be chosen in the primaries was not even a concept in 1860 or 1924. Things had changed by 1968, so at least part of the anger was over the pro-war Humphrey becoming party nominee despite not even running in a single primary. But the basic norm that the nominee should be chosen entirely through election of the delegates and the nominating convention a mere formality was not established at the time. Today it is.
It has long been remarked that our political system is governed not only by laws, but also by unwritten (and often unspoken) norms. As polarization grows, politicians are increasingly doing thing that do not break any laws, but do violate accepted norms. Up till now, this has mostly been in the parties' contests with each other. A brokered convention would mean a party violating its own norms in its dealings with itself. If Trump wins a plurality but not a majority of delegates and the convention seeks to deny him the nomination, they would not be breaking any laws. Indeed, the convention could deny Trump the nomination even if he wins a majority of delegates and still not technically break any laws. That is because there are no laws on how presidential candidates are nominated. Yes, there are state laws on how to conduct primaries and choose delegates to the convention. But first, there is no obligation for parties to actually follow these laws. A party can opt out of the primaries and hold a party caucus instead if it so chooses. Caucuses can take any form the party wishes; a caucus is simply any system of nomination run by the party instead of the state. (That is why some parties have one party hold primaries and the other party holding caucuses). And once delegates are chosen for the party convention, there are no laws governing what the convention is required to do.
If the delegates to the Republican Convention change to rules to deny Trump the nomination even if he wins a majority of delegates, they will not be breaking any laws, so there will be no legal recourse to stop them. But they will be breaking well-established political norms, and the party faithful will quite justifiably respond with outrage. And the nation will, once again, be treated to the sight of a political party committing ceremonial self-disembowelment on the national stage. Only this time such things will be contrary to political norms in a way that they never were before, and the party's prospects for recovery will be -- well, no one can say what the party's prospects for recovery will be because there is no precedent for such a thing happening since the norm of nomination by primary was established.
All of which leads me to believe that Republicans will not be so insane as to make such an attempt.
*Donors and activists also differ on abortion and same sex marriage, but these do not appear to be the issues driving the split or the Trump coalition. Trans-gender issues do not really appear to have appeared on either group's radar screen, although the more trans-gender activists keep insisting on access to the bathrooms of their choice or (worse yet, an end to separate bathrooms altogether), the more attention the issue is going to get.
**Bernie Sanders is trying to bring back the white non-southern working class vote (exactly where Trump is strongest) by focusing on bread-and-butter issues, but immigration will probably be a fatal stumbling block there.