Limiting the channel spells to those types forces them into the evocation school, for the most part. The question becomes "why play a Spellsword when I can just use the ray/cone/etc at higher level sooner". Evokers are arguably one of the least useful of the mages, since they throw away all of their utility to go pew pew.

One thought I had was give the Spellsword a need to use a concentration check to maintain certain effects, though that once again raises the question of things like Bestow Curse and Blindness/Deafness.