Sooner or later your CI server will became very busy, and your frustration will grow because of increasing wait time for each build. One thing you can do to improve the situation is to avoid running jobs when there is no need to do so.

I use Jenkins + Maven + SVN in this example, but you can use this "tricks" also when using different technologies - Gradle, Git etc.