The New Museum for Western Australia will be nearly four times bigger than the now-closed Western Australian Museum, which it will replace.
The building will provide 7,000 square meters of exhibition space and a huge new public square.
It will be located in the heart of the Perth Cultural Centre, a district that also includes the Art Gallery of Western Australia, the State Library and the Perth Institute of Contemporary Arts.
The proposal is to insert a top-heavy, glazed museum building between the surrounding structures, which date back to the late 19th century and have formerly housed an assortment of museums, galleries and libraries.
A public square will sit between these restored heritage structures – known as the Old Gaol, Jubilee, Beaufort and Hackett buildings – and the new museum, creating a grand entrance and events space.
"Our vision for the design was to create spaces that promote engagement and collaboration, responding to the needs of the museum and the community," explained Hassell principal Mark Loughnan and OMA managing partner David Gianotten.
"We want it to create a civic place for everyone, an interesting mix of heritage and contemporary architecture, that helps revitalise the Perth Cultural Centre while celebrating the culture of Western Australia on the world stage."
Inside, galleries for both temporary exhibitions and the museum's permanent collection will be organised around a pair of looping circulation routes.
Among the collections will be the popular Blue Whale skeleton, which featured in the Western Australian Museum when it first opened in 1972.
The museum will also include "learning studios", as well as shop and cafe spaces.
"The design is based on the intersection of a horizontal and vertical loop, creating large possibilities of curatorial strategies for both temporary and fixed exhibitions," added the architects.
Construction of the £245 million museum is expected to start next year and complete in 2020.
It could become the first completed project in Australia for Rem Koolhaas' OMA.
The firm also developed a masterplan for the Gold Coast in 2013, and previously teamed up with Hassell on another project – a winning bid for the redevelopment of Sydney's convention, exhibition and entertainment precinct at Darling Harbour.
OMA recently also completed a museum project in Quebec, an extension to the Musée National des Beaux-Arts.