Nouvel's lawyers William Bourdon and Vincent Brengarth said the fee was "totally disproportionate" and "unprecedented in the world of architecture", reported the Guardian.
They argued that the €170.6 million (£147 million) fine would be tantamount to a death sentence for his architecture studio. To pay it, Ateliers Jean Nouvel would reportedly need to go into liquidation.
The Philharmonie de Paris infamously opened two years later than planned and three times over budget.
Finally opening its doors in 2015, it cost €386 million (£333 million) – significantly more than the €118 million (£102 million) initial budget.
Clad in interlocking aluminium tiles in the shape of birds in flight, the main concert hall seats 2,400 people, while the rooftop features a park for picnicking.
In 2013, the French government considered abandoning the project when it made cuts to arts funding, but construction was too far along and the concert hall was spared the guillotine.
It's not the first time that the Philharmonie de Paris has been the subject of a legal battle with its architect.
In 2015, Nouvel applied for a court order to distance himself from the project, citing the finished concert hall as "non-compliant" with the original designs he drew up. Nouvel insisted his name be removed from the project until 26 modifications be made to bring the building in line with his plans.
He also snubbed the opening night of the Philharmonie de Paris, telling Dezeen at the time that the building was opening "too early" and that he considered it unfinished.
The architect has maintained he is not responsible for the cost and time overruns, accusing the French government – who commissioned the cultural building – of "fakery and sabotage" in the French press.
Nouvel, a Pritzker prize-winning architect, has had budget issues with projects before. In 2003, the Carnegie Museums of Pittsburgh announced it was scrapping plans for Ateliers Jean Nouvel to build a $90 million (£70 million) extension to the Carnegie Science Centre over a "dramatic difference" between the budget and projected cost. The museum was reported to have considered litigation.
Dezeen has approached Ateliers Jean Nouvel for a statement.
Photography is by Danica O Kus.