The charity has applied to the UK's High Court to have a judicial review into Liverpool council's decision to grant planning permission for the attraction.
Zip wire "will harm important historic area"
The Victorian Society took legal action as it believes that the zip wire will harm the historic area through both its physical infrastructure and the resulting noise.
"The proposed zip wire could not be in a more insensitive or inappropriate position, right in the heart of Liverpool's great historic civic buildings and monuments," commented Victorian Society's conservation adviser Tom Taylor.
"The noise and movement, as well as the physical infrastructure required, would harm this important historic area," he added. "There are many places in Merseyside where a zip wire would be acceptable, but such a sensitive site is the wrong choice."
The group also believes the council should have required listed building consent for the zip wire, as the Central Library is Grade II* listed.
Proposal criticised by local councillor and organisations
The zip wire, which has been billed as the "fastest sit-down zip line in the world" was approved by Liverpool Council in a five to three vote on June 30.
Zip World's proposal attracted criticism when it was first announced, with Liberal Democrat local councillor Richard Kemp starting a petition against the project.
A committee report on the original proposal notes that 38 responses were received with 29 of those objecting to it, citing the impact on the listed buildings as well as on St John's Gardens, which is home to war memorials.
Zip World has already had to change one aspect of its original plan, which was to have its Adventure Terminal check-in on the ground floor of the Central Library itself. The check-in now planned to be located in the St Johns Shopping Centre, which is below Radio City Tower.
The Victorian Society has started a crowdfunding appeal to raise funds that will help make its legal challenge possible.
The zip line is the latest threat to the UNESCO World Heritage status of the English city. Plans for a pair of high-rise residential blocks facing the city's River Mersey lead to warnings from the body that had already placed the city on its "in danger" list for possible removal of its World Heritage status following a spate of tall building proposal in 2012.
The warning from UNESCO led to a revised scheme being developed.
Drawings are by DK Architects.