I suppose it would depend on several factors, but it sounds like these restrictions are VERY old and only permit a very narrow and limited use of the property, which may not be practical anymore. Perhaps the restrictions regarding abortion and pornography would still be enforceable, however the requirement that it only "be used to serve people who are poor and infirm" may not make sense anymore. 

If the neighborhood has changes signifcantly, why not allow other uses like condominiums?  If nobody is willing to operate a charity serving the poor and infirm, the building will only deteriorate to the point where it would be condemned.  I think it would make more sense to remove the restriction and see it return to productive use.

I'm not suggesting that ALL restrictions should be removed.  I don't think anyone is arguing that the property should not be subject to setbacks and building height restrictions. But, those restrictions that prevent productive use are not doing anyone any good.

And, if the property were put to a productive use, which would increase its value and tax revenue for the City, I do not think the City would object or intervene to enforce the restrictions.