Sebastian Barry was once in the basement of the Brown and Thomas clothing store in Dublin, buying an Armani suit. In a corner, he saw that other literary excavator of Ireland’s historic secrets and lies, Frank McCourt. “He looked so miserable. But I knew he had just won the Pulitzer for Angela’s Ashes, and the book had sold 1.6m copies. So I went over and said, ‘Are you OK, Frank?’ And he said, ‘No. She’s upstairs buying a hat. A hat!’ He meant, you know, his wife. For me that was very wonderful and very emblematic of the writer’s life.”

Barry tells me this story the morning after receiving the £25,000 cheque for the Costa book of the year award, for his novel The Secret Scripture. The 53-year-old novelist, playwright, poet, sometime singer (and charming interviewee) seems happy enough, though his wife hasn’t yet had time to case the hat department. Barry didn’t expect to win the Costa – he expected, rather, to become a three-time loser of British literary prizes. In 2005, his novel A Long, Long Way was Booker shortlisted, but lost to John Banville’s The Sea. Last October, he lost the Booker again, this time to Aravind Adiga and his debut novel, White Tiger.

Even on Tuesday night, as Barry sat dudded up to the nines with his wife Ali at the prize-giving dinner, he didn’t expect to win. True, The Secret Scripture was, with Adam Fould’s poetry volume The Broken Word, neck and neck when it came to the judges’ affections, and yes, William Hill had made Barry 2-1 favourite. But, halfway through the soirée, he leaned over to Ali and said: “I’m not going to win. Let’s just enjoy the chocolates.”

Why? Perhaps he shared the judges’ misgivings about his book. Justifying the decision, Matthew Parris, chair of the panel, said: “They agreed that it was flawed, and almost no one liked the ­ending, which was almost fatal to its success.” Barry laughs: “Only after he’d devastatingly criticised the novel did he reveal that it had won the prize!”

After winning, Barry spent a sleepless night. Partly, he was thinking about Parris’s words. “Without that ending, which some have found melodramatic, there wouldn’t have been a novel. I aimed at it while writing the book. When Parris complained about the ending, I was reminded of Cavafy’s poem [Ithaca]. Cavafy said that if you’re disappointed in Ithaca, you shouldn’t be – because it was Ithaca that gave you the journey.”

The journey of Barry’s heroine Roseanne McNulty takes us through decades of Irish history, from the civil war to the bombing of Belfast by the Luftwaffe, and the shameful moment when Ireland’s postwar politicians imported British hangman Albert Pierrepoint to execute IRA men. It also features the ancestral wrongs within Irish families, the dubious social role of the Catholic church in the ­formation of the new Irish republic, the lunacy of institutional mental care – plus rape, murder, suicide and betrayal.

Throughout, as Barry wrote in an earlier novel, The Whereabouts of Eneas McNulty, his concern is with “scraps of people, blown off the road of life by “history’s hungry breezes”. One scrap, Roseanne McNulty, is nearing 100 as she records the story of her life.

The novel was catalysed 10 years ago by his mother. “We were driving through Sligo, and my mother pointed out a hut and told me that was where my great uncle’s first wife had lived before being put into a lunatic asylum by the family. She knew nothing more, except that she was beautiful.”

In his story, the great aunt mutates from a Catholic woman to a Presbyterian, though one who retains the misfortune – as Barry calls it – of being beautiful in Ireland. The change in religion allows him to explore some historic wrongs for southern Ireland’s protestants. “Wolfe Tone dreamed of a free Ireland that could be home to Catholic, Protestant, Jew, whoever. That dream was mangled in the aftermath of the civil war.”

Barry admits to plundering his family history for fiction: “I can’t seem to do anything else”. While he was at work on the book, his mother, a famous Irish actor, became terminally ill. This family drama fed into the novel. There’s a scene at the end when Dr Grene, the psychiatrist charged with assessing Roseanne, goes back to the mental hospital not so much to visit her as to “go to see if she was alive. I was pulled up short with this elemental sense of a mother and a son that I hadn’t felt before. We had had a difficult relationship, not speaking to each other for a year at a time.”

Barry’s seeming miserablism has exasperated some English critics (those unremittingly sunny people) who ­perhaps yearn for Irish fiction to be like a mini-break to Dublin, all craic and no downside. But Barry refuses to cater to them. His first agent, one Sophia Sackville-West, told him at the outset not to write about Ireland. “She said there’s no market for Irish stuff, write about England.” Why did you not take her advice? “I couldn’t. It’s like salmon fishing. It’s so hard to catch a book in the nets of time. Hard enough catching an Irish one. I don’t think I could catch an English one”.