In dilute nitride InyGa1-yAs1-xNx alloys, a spatially controlled tuning of the energy gap can be realized by combining the introduction of N atoms-inducing a significant reduction of this parameter-with that of hydrogen atoms, which neutralize the effect of N. In these alloys, hydrogen forms N-H complexes in both Ga-rich and In-rich N environments. Here, photoluminescence measurements and thermal annealing treatments show that, surprisingly, N neutralization by H is significantly inhibited when the number of In-N bonds increases. Density functional theory calculations account for this result and reveal an original, physical phenomenon: only in the In-rich N environment, the InyGa1-yAs host matrix exerts a selective action on the N-H complexes by hindering the formation of the complexes more effective in the N passivation. This thoroughly overturns the usual perspective of defect-engineering by proposing a novel paradigm where a major role pertains to the defect-surrounding matrix.