Here's my two cents, with *SPOILER WARNING* in place, just to be safe:
We know that ancient elves (Solas' time) used magic like breathing.
We know that Arcane Warriors/Knight Enchanters are based on ancient elven combat-magic disciplines.
In Dragon Age Origins, Arcane Warrior enabled you to use Magic to substitue Strength for equipment/weapon requirements, explained as using magic to augment their physical body.
In Dragon Age Inquisition, Knight Enchanters are front-line mages, and according to Solas is based on the Arcane Warrior; being on the front line requires a sturdy constitution.
My *theory* is that elves make good warriors because every elf has at least a little bit of "Arcane Warrior" in them, a residue of their "true" elven heritage that simply manifests as being able to wield weapons disproportionate to their smaller frames.
There's also the fact that lean doesn't always equal weak.
As a real-life example, I have a little brother who is very small, but who is very strong for his size. While I could overpower him if I so desired (heaven forbid), he would undoubtedly put up a very good fight.
If you poke your finger with a thumbtack, you'll draw blood, because the point is so small that all the force gets really focused.
If you use the same amount of force and try to poke your finger with a quarter, you won't break the skin.
Same force, different area size of application.
--------
Long story short:
Combat magic was "invented" by elves.
Magic runs in elves natively, if not naturally.
Said magic is not a "respecter of persons", or species.
Small doesn't equal weak.
A plausible conclusion is that all elves have a slightly augmented physiology as a result from their "native" magic.