Gene Ontology (GO) is a taxonomy of biological terms related to the properties of genes and gene products. It can be used to define a similarity measure between two gene products and assign a confidence score to protein-protein interactions (PPIs). GO is being evolved regularly by the addition/deletion/merging of terms. However, there is no study which evaluates the robustness of a particular similarity measure over the evolution of GO. By robustness of a similarity measure, we mean it should either improve or keep its performance similar over the evolution of GO. In this paper, we systematically study the same for the task of scoring confidence of PPIs using GO-based similarity measures. We observe that the performance of similarity measures gets affected due to the regular updates of GO. We find that similarity measures are not robust in all conditions, rather they keep their performance quite similar over the evolution of GO in certain conditions. © 2019, Springer Nature Switzerland AG.