The far-off island that we can see from our roof terrace is Berlenga. It is a shadow on the horizon right where the sun sets; misty, distant, often hidden. Over time it has worked its way into our imagination: it is a fantasy place, an enigma. The kids are sure there are pirates there. We catch ourselves gazing at it – particularly Menna, when she’s in one of those wistful moods.

We head there by boat one day, after some early morning haggling on the Peniche quay. We travel in a rib to be precise, light and buoyant but with a ton of horsepower. With its inflatable hull and lack of keel, it feels like the wrong sort of vessel to tackle the unexpectedly large and brutish waves that greet us when we swing round the headland. The captain seems to know what he is doing though and we smash our way bluntly over everything that the sea throws at us, airborne at some points with engines whining, before skidding down into dark troughs where the sun disappears and the spray feels very cold. Matilda squeezes my hand throughout the journey, and lets out keening cries like a little storm petrel. There are no other signs of life upon the ocean.

After an hour the vague shape of Berlenga solidifies into a rocky mass in front of us. Cliffs soar, the sun steadies, and the grey of the ocean lightens into a post-card aquamarine. We putter into a bay where a ramshackle cluster of white cottages and a taverna are strung up the cliff road and the rocks are draped with fishing nets, lobster pots and buoys. We clamber onto the dock all damp and unsteady and lurch off to explore the island.

We’ve done this trip the Nicholl way – which is to scorn all of the organised tours but then to forget to do any of the research ourselves – so there is a period on the quayside where we march backwards and forwards waving our phones in the air to find signal and load a map of the island. In the end we give up and stride out to navigate by visual clues. The standard island symbols are all present here: a fort, a lighthouse, some ruins, the ubiquitous clifftop church. This much we can see without Google’s help. It turns out moreover that there is also one of those tourist information boards halfway up the mountain road. Peregrine falcons nest in the cliffs it says, and there is a huge cavern in the Northern rock face – a Catedral. Best of all though… pirates! It turns out actual pirates were here, ransacking the monastery, butchering monks and generally having fun.

We cover the island in a couple of hours, snacking on salt crackers and pomegranate seeds like true buccaneers. Berlenga is a windswept and rugged place, handsome from some angles but very forlorn. It is how I imagine the Falklands to be, only without the puffins. There is very little vegetation though there are plenty of epic rock formations. We see a lot of bird skeletons and Arthur is able to scavenge various interesting bones for his collection. We find the cathedral cave, which is truly awesome, but we do not sight any peregrines.

The highlight of the island is the fort, where you can read stirring accounts of the Spartan-esque heroics of twenty Portuguese soldiers who held off fifteen Spanish warships and a combined force of two thousand men in 1666 (though a little later digging shows that they simply surrendered once they ran out of ammunition). It is the Portuguese Thermopylae and we are honoured to be there. The Portuguese don’t have a lot of military victories to celebrate, but like the Scots, they savour their heroic defeats.

The fort is an imposing structure, jutting out to sea but joined to the mainland by a stone walkway that zig zags over some very clear turquoise waters. We picnic here and spend some time snorkelling. We see lots of fish, some crazy underwater geology and then Menna spots a huge octopus. We follow him along, pointing and making excited whale-like snorkel sounds to each other, while he changes colours, squirts ink and finally folds himself away into a crack in the rocks to be rid of us.

With rash bravado I have opted not to bring my wetsuit and despite the sun I get very cold floating in the Atlantic waters. I drag the family out of the sea and march them back over the cliffs to catch the afternoon boat home.

A bare-chested chap in Hawaiian shorts is the Berlenga harbourmaster. He greets incoming boats, ties mooring ropes and bosses around a pair of barefoot urchins. His belly taut and rounded and he has a fine black moustache which gives him authority. We watch him explaining some important unloading procedure to one of his underlings when he astonishes us by throwing himself off the pontoon mid-sentence. It is a neat dive and he enters the water with barely a splash. Some seconds later he re-emerges on the slipway at the far end of the jetty, walking slowly and majestically up the ramp and it is like a messiah is rising up out of the sea. Best of all, he resumes his conversation as if nothing has happened, although he is now quite some distance away and has to shout. Arthur and I are deeply impressed.

On the way back the waves are with us and the ride is noticeably smoother. The boat is pumping out house music. We plane along the big rollers until we reach the port at Peniche. Then our captain decides to give us all a virtuoso finale. As soon as we are level with the ‘4 Knots. Speed Restriction’ sign on the harbour wall, he slams his throttle forward, revs up to maximum and slaloms his way through the breakwater at high speed, tipping the boat hard up on its side through several sharp curves. We are horizontal at points, thrown one way then the other and only our seatbelts stop us being ejected into the water. Everyone screams and an elderly Dutch couple look like they’re going to have a heart attack. It is a bizarre manoeuvre and I’m not sure if it is intended to generate tips, demonstrate his helming skill or as a anti-disestablishment ‘fuck-you’ salute to the port authorities.

Once inside the marina the captain kills his speed and glides smoothly into his pontoon slot. He cuts the engine, turns off the music and says nothing. We all turn to look at him but he is inscrutable behind his sunglasses.

We disembark and head off into Peniche to look around the prison, but it is shut on Tuesdays.