How to predict UK by-elections - and why the Tories are in trouble

Parliamentary by-elections are a chance for voters to give the government of the day a firm kicking without ejecting them from office with immediate effect, making these unscheduled local ballots a different beast to general elections.

The relatively low-stakes nature of by-elections means governing parties usually perform worse than the national polls would indicate, with opposition voters disproportionately inclined to show up and inflict as much embarrassment on the government as possible.

Grouping historic results based on whether the incumbent party was also the main party of government, we can see that when modelled against a Uniform National Swing, governing parties fare about nine points worse on average than their opposition counterparts.

Likewise, opposition parties function differently in by-elections. The Liberal Democrats, for example, win by-elections seats at more than three times their general election rate. Historically, the Lib Dems have sat in the ideological space between the two major parties, making them the perfect vehicle for disaffected voters of the governing party who want to voice their frustrations without opting for their usually-favoured party’s mortal enemy.

For these reasons, UNS is not a great predictor for by-election results - especially when it comes to governing parties - so a different approach is needed.

I’ve crunched the numbers going back to 1979 and created a model which aims to forecast the incumbent party’s vote share and chance of winning based on a number of variables. They include the previous election result at the constituency and national levels; their current standing in the national polls, and whether they currently head the UK government. The model performs better than both a Uniform National Swing and proportional swing for governing and non-governing parties alike.

With four upcoming by-elections and a fifth up in the air, the graphic below shows how the different models would compare if those elections were held today.

While UNS points towards the Tories standing a good chance of keeping at least two of the four seats they currently hold (depending on how the opposition votes fall), my by-election model makes the governing party considerable outsiders in all four instances. 

As it stands, the model does not distinguish between opposition parties, so it won’t be able to tell you who the most likely anti-Tory candidate is in each seat, but based on local campaigning and news reports you can probably figure that one out for yourself.

Looking back at some of the historic odds for recent by-elections, I noticed that the Conservatives started out as very heavy favourites in Chesham and Amersham and North Shropshire, both of which they went on to lose. It seems like time and time again the betting markets and conventional pundit wisdom have a tendency to conflate general election and by-election environments and thus overrate the prospects of the governing party. I wonder if four losses in one fell swoop would be enough to change some minds.