For me it does matter. I wouldn't want to be seen with a shirter man as a dating partner while if someone else is doing so doesn't matter to me.
I prefer tall guys. I feel height makes a huge difference in personality statement but so does the right weight. I wouldn't prefer tall lanky guys.