に限って

夜寝れない時に限って、時計のカチカチいう音が気になるものだ。

It's only when I can't sleep at night that the ticking of the clock becomes loud enough to bother me.