I am not a penis expert... to begin with, I don't even have one
but I was taught in school that what makes your penis big or small is the size of the "cavernous bodies" (cuerpos cavernosos in Spanish, don't know if my translation of the words is correct), that are simply wholes that get filled with blood... No muscle action.
But I may be wrong, my degree is far away from being in the medical field.