As long as there are signs, and they should replace the name of the building if there is a sign (but it should still be changeable - the sign should just be the default display), it's all good. No sign = "a stone building."
It's still less of a hassle than the switch from where you could see every character's name to dynamic naming.
As for established characters remembering the names: how many buildings do most characters use / care about? Not many, in most cases. Would you remember the names of those buildings you never notice? And it makes no sense for already existing characters to be magically able to know the name of every building, unless you mean only in their current location. That would make a bit more sense, but not everyone stays in their hometown where they know most of the buildings. (Does that make sense?)

