Brian O’Nolan was born 100 years ago on Wednesday. As the man behind Flann O’Brien and Myles na gCopaleen, he could have been a celebrated national treasure - but he was far too radical for that

In The Poor Mouth, Flann O’Brien’s devastating parody of Gaelic-language autobiographical peasant narratives, the hero is alone at night on the seashore when he hears a terrible, unrecognisable sound. He is then assailed by “an ancient smell of putridity which set the skin of my nose humming and dancing”. He eventually sees a huge black quadruped like a giant hairy seal with legs. The following day he tries to describe the beast to his grandfather, who asks him to sketch it. The contours of the terrible creature, called the Sea-cat, appear in the text of the novel.

It is a map of Ireland turned on its side, the four major peninsulas acting as legs, the bulbous sweep of the north-eastern shoreline forming the head. In a footnote, the “editor” of the cod-memoir tells us that “it is not without importance that the Sea-cat and Ireland bear the same shape and that both of them have all the same bad destiny, hard times and ill-luck attending them.”

The ancient smell of putridity that emanates from this half-comic, half-terrifying embodiment of Ireland is not unrelated to the stink of “history’s ancient faeces” that, according to the narrator of Samuel Beckett’s First Love (written five years after The Poor Mouth in 1946) largely constitutes “the charm of our country”. If Beckett and O’Brien shared a great deal besides their belief that something was rotten in the state of Ireland, the overwhelming difference between them is that Beckett, like the majority of their literary contemporaries, managed to flee from the Sea-cat. O’Brien, almost alone among the great writers of 20th century Ireland, fell into its clutches. He stayed in Ireland and paid a fearful price in frustration and neglect.

Flann O’Brien was born into a culture of lingering, post-revolutionary dissolution. As with Beckett, his genius was to find energy, both comic and grotesque, in that entropy. The great ferment of change in the early years of the 20th century had resulted rather anti-climactically in a small, impoverished state, culturally philistine and sexually repressed, its energies drained by exhaustion and mass emigration. WB Yeats died in 1939, a month before the 27-year-old Brian O’Nolan, using the pseudonym Flann O’Brien, published his astonishing first novel, At Swim-Two-Birds. James Joyce’s last work, Finnegans Wake was published in the same year.

Frank O’Connor, writing in 1942, claimed that it was impossible to write a normal social novel in Ireland: “the moment a writer raises his eyes from the slums and cabins, he finds nothing but a vicious and ignorant middle-class, and for aristocracy the remnants of an English garrison, alien in religion and education. From such material he finds it almost impossible to create a picture of life ... a realistic literature is clearly impossible.”

If realistic literature is impossible, the writer’s options are stark: stay quiet or invent something new. O’Brien and Beckett chose the option of making a new literature of lingering dissolution.

At Swim-Two-Birds has a strong claim to be one of the founding texts of literary postmodernism. All the markers of that baggy but indispensible cultural category - the deconstruction of narrative, the replacement of nature by culture, an ahistoric sensibility in which tropes and genres from different eras can be mixed and matched promiscuously, the prominence of pastiche, the notion of language itself as the real author of the work - are openly declared in At Swim.

This is a book that begins by questioning why a book should have just one opening, and proceeds to give us three. It is a book by a man (Brian O’Nolan) who invents an author (Flann O’Brien) who is writing a book about an unnamed student narrator who is writing a book about a man (Dermot Trellis) who is writing a book. The narrator openly declares that “a satisfactory novel should be a self-evident sham” and that “the modern novel should be largely a work of reference”, since virtually all characters have already been invented. Its governing caprice is that fictional characters do in fact already exist, have independent lives and are capable of revolting against the author. The novel is a treasure house of brilliant pastiches of everything from Gaelic sagas and Irish folkloric narratives to the Bible, Victorian encyclopedias, scholasticism, pub poets, cowboy novels and trashy thrillers.

Frederic Jameson suggests that “postmodernism is what you have when the modernisation process is complete and nature is gone for good”.

There could be no better illustration of this condition than O’Brien’s other major novel, The Third Policeman. Its narrator is trapped in an entirely literary Hell, from which nature is indeed gone for good, and the world is a large machine controlled with levers and knobs by strange policemen. Even the purest of natural phenomena, light, is not natural at all but “light of a kind rarely seen in this country and ... . possibly manufactured with raw materials from abroad.”

Yet - and this may account for his relative critical neglect - O’Brien does not sit easily with postmodern theory. His ideas and idioms cannot be explained, as such theory would like to suggest, as responses to the conditions of “late capitalism”. O’Brien was not responding to the completion of the project of industrial modernity, but to its failure.

He lived and worked in a largely agricultural country that was struggling to impose an ideal of cultural and economic self-sufficiency that cut it off from the mainstream of capitalist development. He poses a critical dilemma that can be resolved only by seeing his dazzling novels as paradoxical products of the conditions of mid-20th century Ireland. The paradox is that what made those conditions so strangely fruitful was the collapse of any notion that a novel could be a direct representation of the society in which it was written.

§

If The Third Policeman gives us O’Brien’s vision of Hell, what is his Heaven? It is the state of lying alone in one’s own bedroom, cut off from the social world except for the occasional visit of a like-minded butty. “What is wrong with most people,” says the dilettante intellectual Byrne in At-Swim, “is that they do not spend sufficient time in bed” - a version of Blaise Pascal’s statement, used as an epigraph for the late, minor O’Brien novel The Hard Life, that all the trouble of the world comes from not staying alone in one’s room. And what does one do in bed? In a peculiar triumph for the puritanical literary censorship that deformed Irish culture during his lifetime, the bedroom in O’Brien is the locus, not of sex, but of writing. Secret and unbridled instincts are played out, not in the flesh, but in the word.

In this, O’Brien turns to advantage the great agony of being an experimental writer in post-revolutionary Ireland: indifference. In another culture, Brian O’Nolan would have made a perfect Establishment intellectual. He was a government official of relatively conservative disposition. His family background was fully in tune with the new State’s major cultural project: the revival of Irish, which he spoke and wrote superbly. O’Brien was steeped in Gaelic legend and folklore.

O’Brien ought to have been a treasured mainstream figure in nationalist Ireland, a dazzling writer, working within the State apparatus, who could synthesise Gaelic and English, ancient lore and contemporary Modernism.

Yet he was an extraordinarily marginal figure. His journalistic alter ego, Myles na gCopaleen, was celebrated in intellectual circles, but both his official and literary careers were disastrous. A combination of his gradually deepening alcoholism and his habit of making derogatory remarks about senior politicians in his newspaper columns led to his forced retirement from the civil service in 1953. (He departed, recalled a colleague, “in a final fanfare of f***s”.) More significantly, Irish literary culture, constrained by censorship, had little place for his staggeringly original novels.

§

O’Brien was deeply disillusioned by the philistinism of the official nationalist culture. The Gaelic language revival is unmercifully burlesqued in The Poor Mouth. A German scholar receives a PhD in Berlin for his recordings of what he thinks is a native speaker, but is in fact a pig. The tendency of Gaelic writers to give themselves flowery pen-names is parodied in the nom-de-plumes of the writers the narrator encounters, among them The Bandy Ulsterman, The Sod of Turf, The Gluttonous Rabbit and Popeye the Sailor.

The Puritanism and narrowness of the official culture meant not just that O’Brien could not embrace it, but that it could not embrace him.

His scorn for the purists who saw in Gaelic and in traditional customs a barrier against modernity was boundless. “I do not think”, he wrote “that there is any real ground for regarding Irish dancing as a sovereign spiritual and nationalistic prophylactic.” He was too utterly Irish to be easily appreciated abroad and too contemptuous of official forms of Irishness to be comfortably placed at home.

Of his three important works, At Swim met with the enthusiastic approval of Graham Greene and James Joyce - it was the last novel he ever read - but got largely puzzled reviews, sold poorly and was swallowed up by the outbreak of the second World War. The Poor Mouth was published in Irish, a language with fewer readers, and was appreciated largely as a brilliant in-joke. And The Third Policeman was rejected in 1940 by the publishers, Longman’s, who explained that they wanted O’Brien to become less fantastic and instead he had become more so. Humiliated, O’Brien put about the story that the manuscript of the novel had been lost. This was, at least metaphorically, true: the novel was not published until 1967, after O’Brien’s death, by which time he had cannibalised it for the vastly inferior The Dalkey Archive.

Yet, if the conditions of post-revolutionary Ireland doomed O’Brien to neglect, they also forced him into fabulous invention. Sometimes, to take the most direct example, O’Brien’s jokes are a direct burlesque of the official censorship that disallowed any mention of sex. In At Swim, the narrator mentions student societies at his university: “Some were devoted to English letters, some to Irish letters, and some to the study and advancement of French language” - the final comic circumlocution arising from the inadmissibility of “French letters”, the colloquial term for condoms.

There is, in The Third Policeman, a parody of the kind of trashy sex scene that would undoubtedly have fallen foul of the censors, were it not for the fact that the object of desire is not a woman but a bicycle. The narrator slavers over “the prefect proportion of its parts ... How desirable her seat was, how charming the invitation of her slim encircling handle-arms, how unaccountably competent and reassuring her pump resting warmly against her rear thigh!”

The banning of almost every serious Irish contemporary novel also created the strange literary culture in which O’Brien revelled, one in which officially approved reading was narrowed to theological reflections, Gaelic sagas and peasant narratives while the thirst for contemporary stories was slaked by imported cowboy stories and cheap crime thrillers.

O’Brien’s main novels draw much of their humour from the absurd conjunctions implicit in this unlikely mix. At Swim sets heroic and folkloric figures (Finn MacCool, Sweeny, The Good Fairy, The Pooka MacPhellimey) literally alongside the cowboys Slug and Shorty. The Third Policeman draws on detective stories and science fiction as well as Catholic theology and mediaeval Gaelic literature.

More importantly, O’Brien’s novels draw their dark energy from the sexual repression that lay behind the censorship. They are remarkable for the almost complete absence of either the nuclear family or healthy sexuality. Instead of being merely desolate, however, this absence of family and sexual fulfilment is linked to O’Brien’s great conceit in At Swim - that of literary creation as the male substitute for giving birth.

Writing is sex for an all-male, sex-averse society. Its children are conceived without all the bother and awkwardness of having to deal with women. In the bedroom that is the world of his narrators, congress with oneself generates the only life that is available - the life of words and stories.