The unprecedented banishment of Prince Andrew to the outer reaches of the royal firm, without a useful future role or official engagements, free – presumably – to spend more time with his golf clubs, comes at a critical time for the family.

After the calamities of the 1990s and the patient rebuilding of reputation that followed, all seemed set fair only 18 months ago. The Queen was still soldiering stoically on into her 90s, working on her red boxes of official papers every day as she had done for the previous seven decades and attending 293 engagements during the year. Prince Harry had just married Meghan Markle to popular acclaim, seemingly inaugurating a new era for the royal family. And the succession was secure to three generations, if they played their cards right.

How easily destabilising uncertainties can intrude. From the Duke of Edinburgh’s car crash in January, through to Harry and Meghan’s evident unhappiness with their royal role and their fissure with Prince William and Duchess Kate, the great hopes of the royal family, it has been a difficult year, if not yet another annus horribilis like 1992. On top of that came Andrew’s self-inflicted crisis, which has angered other members of the family, especially Prince Charles, reopening old sores and throwing the smooth running of the institution into chaos.

Andrew’s fall from grace has had a slow motion inevitability. His relationship with the paedophile American financier Jeffrey Epstein had been known since the pair were pictured strolling together in Central Park in 2011, but was ignored, especially once a Florida judge ruled that the allegations involving the prince were unnecessary to know. But following the latest complaints about Epstein and his suicide in prison in August, Andrew’s association with him was bound to float into view once more, even though no criminal allegations have been laid against him and he denies any personal wrongdoing.

His ill-judged television interview last weekend precipitated his fall, highlighting as it did well documented flaws: his sense of entitlement, freeloading, obtuseness and insensitivity. The British public can bear a lot from its royal family but resent extravagance and arrogance, especially over the expenditure of money and particularly from someone whose dedication to the common good has been sometimes hard to detect.

Prince Andrew at a British trade fair in Kuwait in 2000.
Prince Andrew at a British trade fair in Kuwait in 2000. Photograph: Yasser Al-Zayyat/AFP via Getty Images

The prince’s timing was also at fault: distracting attention from the general election (which royals are not supposed to do) and, from Charles’s point of view, deflecting more positive publicity from the Waleses’ visit to New Zealand – not that that was generating much excitement anyway.

Making matters worse: Andrew did not have the excuse that the interview was sprung on him. It was his decision to take part, evidently without proper briefing, rehearsal or forethought. Perhaps he believed he could wing it, despite his ill-concealed disdain for the media and the warnings of his newly appointed communications adviser Jason Stein, who lasted four weeks before leaving for the City PR firm Finsbury. The future of his private secretary, Amanda Thirsk, who lobbied in favour of the interview as a clear-the-air exercise must now be in doubt.

The royal family operates to a complex grid system, not only in terms of official visits (“whose turn is it to go to the west country? ... when was the last visit to Leicester?”) but also in their association with particular interests: William and Kate with the arts, Camilla, duchess of Cornwall with health and literacy issues.

Andrew’s turf was business. For a time he relished the title of British trade ambassador, giving him scope to travel in some comfort and no little expense by chartered jet to often agreeable parts of the world, a role he continued to carry out until this month, though he lost the title some time ago because of his association with Epstein. If the trade results were nebulous – did foreigners really sign deals because Prince Andrew had turned up at a British embassy reception? – the contacts did him no harm at all. One such is Timur Kulibayev, the son-in-law of the former dictator of Kazakhstan, who bought the Yorks’ derelict former marital home near Ascot for £15m – £3m above the asking price – in 2008.

Years of the media ridiculing the cost of his trips – last year’s best effort was £22,208 for a three-day visit to Hungary – had no effect whatsoever. “The trade role was perfect for him because he felt he was having an impact,” a former palace official told me. “The problem with Yorkie is he can be very arrogant and petulant. I think that’s down to insecurity.”

Beginnings

Andrew is reported as having been introduced to Jeffrey Epstein through mutual friend Ghislaine Maxwell, daughter of newspaper tycoon Robert Maxwell. Epstein reportedly visits the Queen’s private Scottish retreat in Aberdeenshire

Sightings

Andrew, Maxwell and Epstein are seen together at Donald Trump’s Mar-a-Lago Club in Florida. Later in the year, Epstein and Maxwell attend a joint birthday party at Windsor castle hosted by the Queen

Teenager claims

Virginia Giuffre, then aged 17, claims to have had sex with Andrew in Maxwell’s Belgravia home. Giuffre, whose surname was Roberts at the time of the alleged incidents, says she slept with Andrew on two further occasions, at Epstein’s New York home and at an “orgy” on his private island in the Caribbean

Epstein jailed

Epstein is jailed for 18 months after pleading guilty to prostituting minors

Epstein freed

Soon after his release from jail, Epstein is visited by Andrew in New York. The pair are photographed together in Central Park. Footage emerges years later, reportedly shot on 6 December, that appears to show Andrew inside Epstein’s Manhattan mansion waving goodbye to a woman from behind a large door

Backlash

Andrew quits his role as UK trade envoy following a furore over the Central Park photos

Claims emerge

Allegations that Andrew had sex with Giuffre emerge in US court documents related to Epstein. The papers, filed in Florida, state that she was forced to have sex with Andrew when she was 17, which is under the age of consent in the state. Buckingham Palace denies the allegations.

The claims against Andrew are later struck from US civil court records following a federal judge’s ruling

More claims

Andrew is accused of sexual impropriety by a second alleged Epstein victim, Johanna Sjoberg. She claims he touched her breast inside the US billionaire’s Manhattan apartment in 2001. Buckingham Palace said the allegations were “categorically untrue”.

On 10 August, Epstein is found dead in his jail cell, having killed himself after being charged with sex trafficking.

A pilot on Epstein’s private jet later that month claims Andrew was a passenger on past flights with the financier and Roberts

Prince Andrew announces he is to step back from public duties 'for foreseeable future'.

The writing was really on the wall for Andrew’s future when companies started questioning their sponsorship of his Pitch@Palace initiative, a sort of Dragon’s Den with chandeliers, where young entrepreneurs once a year attend Buckingham Palace and present their ideas to business leaders.

They would scarcely want to be associated with someone tarred by friendship with a notorious sex offender. Andrew also has links to more than 200 charities, 33 of which are involved with education and young people. He won’t immediately shed his honorary military titles, as an admiral (except perhaps as admiral of the Sea Cadet Corps), honorary colonel of several regiments and an air commodore in the RAF, but they won’t like the taint either and he may have to stand down for the good of the services. Talk that Andrew’s purdah will only last a few months is wishful thinking: Epstein’s ghost will always hover behind him now.

Andrew’s absence will leave a hole in the royal grid: who will fulfil the 394 engagements he carried out last year? Who will step in to his role with business, particularly with Harry and Meghan also taking a back seat? The ambition of Prince Charles and some advisers to slim down the working membership of the family to a smaller active core is being achieved by natural wastage, not design.

Age is also having its effect: the 98 year-old Duke of Edinburgh has retired, the Queen herself is slowing down at 93, Charles is 71 and even William is nearly 40: “He’s just another bald bloke in a suit,” one royal photographer lamented. The latest poll this week, YouGov for the Times, was ominous: 54% supporting the sovereign, 20% not and 25% indifferent. The strategy will need a rethink if the royals are, in the Queen’s words, needing to be seen to be believed.

Stephen Bates is the author of Royalty Inc: Britain’s Best-Known Brand (Aurum)