Roth, 61, has been director of the museum since 2011 and will leave this Autumn to return to Germany.
He had told German broadcaster DW that the Brexit vote was a sign of a brutal "me-first mentality" that was spreading through Europe.
He said that the campaign had been won using "war rhetoric" that was aggressive and "untypically English".
Roth said he had no idea how the vote would impact the cultural sector, but that a cut in European funds would have a direct impact on research. However he said the real impact was more ideological than economical.
"Until now, we had assumed we were all working together towards a common platform," he said. "Now that platform has been smashed."
"I can understand that people ask: how should we deal with Brussels?" he added. "But it seems crazy to me that this question should lead to smashing everything else. It is destroying the future of our children."
In an official statement, the V&A said Roth was leaving to spend more time working with international cultural consultancies, and with his wife and children.
He has overseen some of the institution's most successful exhibitions in its 150-year history, including last year's record-breaking David Bowie Is and the blockbuster Alexander McQueen: Savage Beauty – an expanded version of a show originally staged at the Metropolitan Museum of Art in New York.
Under his directorship, the museum has also refurbished a number of its galleries and hosted ambitious temporary installations – most recently a robotically woven carbon-fibre pavilion developed by a team from the University of Stuttgart, which formed part of a programme of engineering events related to an exhibition on Ove Arup.
He also spearheaded the museum's new Exhibition Road entrance, designed by Amanda Levete and due to open next year, and its expansion with V&A East in Stratford, east London, and the Kengo Kuma-designed V&A Dundee in Scotland.
"It's been an enormous privilege and tremendously exciting to lead this great museum, with its outstanding staff and collections, and I'm proud to have steered it to new successes and a period of growth and expansion, including new partnerships around the UK and internationally," said Roth in the official announcement about his departure.
"Our recent accolade as Art Fund Museum of the Year feels like the perfect moment to draw to a close my mission in London and hand over to a new director to take the V&A forward to an exciting future," he added.
The V&A is now launching a search for his replacement.
Roth's departure is the latest example of the fallout from the EU referendum results. A number of architects have already reported that projects have gone on hold due to uncertainty surrounding Brexit, with ensuing job losses.
At a Brexit event hosted by Dezeen in June, architects and designers described the shock in their offices following the announcement of the leave victory.
Amanda Levete, whose architecture firm AL_A employs 50 people, said the mood in her office the morning after the vote was "so dark, I had to send an email to make people realise they're valued".
"If there is a psychological feeling that you're not welcome, people will leave," she added.
Johanna Agerman Ross, the Swedish founder of design magazine Disegno, said it was the first time she had felt unwelcome in the UK.
The V&A did not respond to requests for further comment.