Pretty much 95% of the non-native English speakers I encounter are way better at English than they realize.
Given the ubiquity of English, I find that many of us who grow up speaking it take for granted how hard it is to learn. And (in the US at least) we tend not to emphasize learning others' languages (many schools require taking classes, but it's treated more like an extracurricular than a necessity or truly important component to our studies). The result is that not only are the non-native English speakers I meet great at English, they are also far better in a second language than I may ever be.
Also, having actually majored in English, and worked as a teacher of writing, I am especially aware of that "restrained" feeling you describe. Because I sometimes struggle to figure out exactly how to phrase something in the language I've spoken my whole life and am actually trained in, underscoring how far from sufficient my knowledge of another language will ever be. It's part of why I've turned back to trying to practice French: I don't know if I'll ever get fluent, and even then I'll probably still feel quite restrained, but I hate only being able to really convey my thoughts in this one language.