"Goldsmiths CCA is an art gallery not a squat, and it is unseemly for it to pretend to be one"

Assemble's approach to transforming a public bathhouse into an art centre is refreshing, says Owen Hatherley, but the results feels a little too much like a celebration of poverty.

Nineteen years ago, I got to explore a normal area of London for the first time. In September 1999, I was enrolling at Goldsmiths College, where I would study for the next six years, as undergraduate and postgraduate. I've never moved further than a bus ride away since.

New Cross is a district of late Georgian, early Victorian classical streets. At that time some were still coated in soot, with ornate, often squatted pubs, Caribbean takeaways with pop-art signs – Cummin Up is still there, and probably ought to be listed – and the huge interwar faience picture palace of The Venue, then painted entirely black.

The college itself was never somewhere you'd go just to look at buildings. It consisted of Reginald Blomfield's clipped, bright red neoclassical main building, supplemented by a dull brutalist tower (since clad), an early Allies and Morrison library (which seemed quite excitingly modern in 1999), barrack-like halls of residence and a succession of prefab buildings. These served as the studios where the irritatingly flippant conceptual art, for which the place was famous at the time, was produced. It was the temporary buildings that were the real hive of innovation, as Cedric Price always said they would be.

But what really caught my eye then was the former Laurie Grove Public Baths, a fabulous red and white jacobethan palace. Through an archway, the fanciful, florid detail led through to filthy stock brick, a water tower, bins and more prefabs. Opposite this was a Victorian shopping street, with the Edwardian and flamboyantly camp Deptford Town Hall on one side, and a dull brown 1950s block on the site of one of the second world war's worst rocket attacks on the other.

That's the sort of a place this was and still is – a mess of illusions and a historical pile-up. I'm glad it was my introduction to London.

For Goldsmiths to pick Assemble was a smart choice. There are few designers that could do this organised chaos justice

For Goldsmiths to pick Assemble as architects for its Centre for Contemporary Art (CCA), carved out of the Laurie Grove Baths, was a smart choice. There are few designers that could do this organised chaos justice without smoothing it over and making it neat.

I walked to it through the archway that is part of the baths' facade, which seemed the right thing to do – Goldsmiths proposes to demolish some prefabs to create a frontal, formal approach, which is a terrible idea. Taking this route, the CCA emerges organically, up mock-industrial steps, out of the alleyway. It's a hulk of stock brick, with an angular chimney and a water tank, emphasised Smithsons-style as a found object. Next to it is a reminder of the last time Goldsmiths tried to do a landmark building – a multi-storey block by Will Alsop, with sheet metal siding and a spaghetti "sculpture" at the top, because this is an art school, yeah?

What Assemble have done here represents a very different approach. It isn't designed to impress, and any artiness is all about space and surface rather than gesture. There's a huge shift in British architectural fashion in the difference between these two adjacent buildings.

Assemble have turned the Baths' back end into a frontage, of an odd sort, a Bernd and Hilla Becher industrial vignette, asymmetrical and eerie.

Inside, there's the sort of montage you'd expect from the redesigners of the Four Streets in Granby, Liverpool, where the destructive effects of a botched clearance scheme were incorporated into the renovated houses. Thankfully, this time we don't have to pretend that the buildings are solving the housing crisis.

There are fragments of white tiles, soggy bricks, liquefying metal, and unexpected views outwards, into the painting studios in the main baths, and downwards, into a double-height empty space below the ground floor. This creates a tightly packed, illogical maze.

It isn't designed to impress, and any artiness is all about space and surface rather than gesture

To see what is strange and exciting about Assemble's building, one could compare it to 6a's just-opened renovation of a Camberwell fire station into an expansion of the South London Gallery, just a couple of miles up the road. Again, it's a rough Victorian building that the architects have refused to prettify, but the approach is so much more polite and sensible, with none of the fetishism and weirdness Assemble have gone in for.

I loved the results, but it's easy to imagine someone finding it all exceptionally irritating and mannered; the Goldsmiths CCA is a high-end art gallery, not a squat, and it is perhaps unseemly for it to pretend to be one, in a city where that sort of non-commercial activity has been ruthlessly marginalised. Nineteen years ago, there were several actual squats around the corner from here, but that sort of thing isn't allowed to happen in London anymore. Here, at least, the feel of that strange city is retained, but how much of its spirit?

When the gallery opened, protesters, drawing attention to an ongoing dispute over the pay and conditions of cleaners at Goldsmiths, unfurled a banner reading Who Keeps the White Cube White? One side of the double-meaning doesn't entirely work – one of the most refreshing things about Assemble's building is its refusal of the tedious white cube idea – but the other side does.

One of the most refreshing things about Assemble's building is its refusal of the tedious white cube idea

Part of the reason why I went here, two decades ago, was that I wanted to study in London, not in a campus in London, and Goldsmiths was one of the places where you could do that. You were embedded in an ordinary residential inner city area, with no gates, lodges or quadrangles. Not only was this the opposite of Oxbridge, is was also the opposite of the main University of London campus in Bloomsbury. One of the things I would come to realise about this place was that in London, closeness can be deceptive.

Proximity to people different from you is something London does well, much better than most European capitals; but it isn't the same thing as equality. This intriguing building opens at a time when inequality in the capital has been stretched to breaking point, with the staggeringly rich making even very ordinary places like New Cross as expensive as Manhattan – and we all know what role artists have had as the advance guard for that process.

I enjoyed Assemble's building a great deal, because of how it refuses to tidy up London; but in today's London, a refusal to make nice can feel like a simulation of poverty.