Image information about the state of a building after an earthquake, which can be collected without endangering the post-earthquake reconnaissance activities, can be used to reduce uncertainties in response predictions for future seismic events. This paper investigates the impact of using data from image-based inspection of building facades on reducing the uncertainty in predictions of demand parameters that are useful for seismic assessment and retrofitting. Data consist of observable cracks in masonry walls. Experimental data from shear-compression tests conducted on masonry walls is used to define a criterion that associates the demands on the walls to the onset of observable shear cracking to use it then during the analysis of a complete building. The procedure is validated using experimental data from a shake-table test conducted on a half-scale building with unreinforced masonry elements for which, based on an equivalent frame model approach, nonlinear dynamic simulations are performed on a set of model instances of the building. A model falsification methodology is used to discard models for which the simulated response does not match the observed behavior, thus leading to a reduced model set with which the uncertainties in response predictions are reduced. Compared to when no data related to the damaged state of the building is used, the number of models is significantly lowered when the damage recognized in the building is used as a criterion for falsification. Furthermore, models that are not falsified provide accurate predictions for maximum roof displacements, maximum base shear, and the ability to predict the activation of several failure mechanisms, such as out-of-plane failure and toe crushing in the masonry walls, showing that detection of shear crack patterns is a powerful falsification criterion.