Do some opinions matter more in issues of public art?: Robert Serra & the Tilted Arc controversy

I saw the image of Richard Serra's Betwixt the Torus and the Sphere on pg. 200 of the text, and while that particular work is visually interesting, it didn't particularly move me, but I kept having the nagging feeling that I had heard the artist's name before & that I should know something about him. It was really bothering me, so I looked it up & here's why I remembered him:

In 1981 Serra installed Tilted Arc  in the Federal Plaza in New York as a site-specific commissioned public art piece.

The sculpture was immediately controversial, largely because the workers who were used to crossing the Federal Square unimpeded now had to navigate around the large work. Eventually, this lead to a lawsuit in 1989.

According to notes on the public hearing,"122 people testify in favor of retaining the sculpture, and 58 testify in favor of removing it"--which one would expect to indicate the sculpture would be retained. But it wasn't. It was ruled that the sculpture would be removed & scrapped.

The negative votes were nearly all from the office workers from the Federal Square.

So here's the question:  Is it OK to give more weight to one set of voices than others in matters of public art? Was the ruling correct? Should the workers have had more "right" than the general public to object to the work because it impeded their daily comings and goings?