Lol, that's the way I am in a group - the guy everybody likes to draft off of. I'm not a huge guy but I'm pretty big as people who know me can attest.
In the group I used to ride in there was a woman who was a very good cyclist. I always hated trying to draft off of her because she was so small. It was like getting no effect at all.