
Summoners in and of themselves aren't necessarily unplayable. Canonically, many people have the capability of having both the echo and Hydaelyn's blessing. In 1.0, they established that while "rare" on the grand scheme of things, there was literally an entire order of people who had the echo (of which your character eventually became a part of). That being said, given the frequency of primal summons and that key note, a Summoner is certainly doable.
Of course, being a Summoner means your character would have had to have defeated any primal they wished to summon, though this is a given. Not necessarily something you can do accidentally, even with a group of others who cannot be tempered.
Lastly, canonically, the horses are not actual summons. They were once nightmares, which are beasts that were given to the tribes by the Ascians. The primals themselves then tempered these nightmares into the legendary horses you see in-game. The established canon for these horses (and nightmares) is that they can only be tamed and called to one's side by use of a unique whistle. That's just established canon, however.
All that being said, you should ultimately do what you want to do. Some people may dig it, others might not, but at the end of the day the only thing that matters is that you enjoy yourself. If you're doing just that, well, that's all that matters.
Of course, being a Summoner means your character would have had to have defeated any primal they wished to summon, though this is a given. Not necessarily something you can do accidentally, even with a group of others who cannot be tempered.
Lastly, canonically, the horses are not actual summons. They were once nightmares, which are beasts that were given to the tribes by the Ascians. The primals themselves then tempered these nightmares into the legendary horses you see in-game. The established canon for these horses (and nightmares) is that they can only be tamed and called to one's side by use of a unique whistle. That's just established canon, however.
All that being said, you should ultimately do what you want to do. Some people may dig it, others might not, but at the end of the day the only thing that matters is that you enjoy yourself. If you're doing just that, well, that's all that matters.