Disclaimer: I own nothing
Warning: Hints of child abuse
In many ways the Slytherins know perfect Potter better than all of his little sycophants. Not Professor Snape, because for whatever cause he loathes the boy beyond reason. Not Draco either, because he is too wrapped up in his all-consuming rivalry to think, really think. But the vast majority knows Potter quite well.
They notice that he is scrawny, just like Theodore and Tracy. He has their haunted, sunken eyes. They remember Theodore's narrow, awed face as food appeared that first night. His hands shook as he served himself a small amount of stew. They compare it to Harry's wide-eyed delight in seeing the same sight. His own shaking fingers. He grabs at food on the table as though worried it will vanish in front of him. He is afraid to take too much, and what he takes he eats quickly. He eats sweets like he's never tasted them before.
They notice that he is careful, just like Blaise and Millicent. He watches people closely, and only just manages to contain a flinch when anyone moves too suddenly. It's so ingrained in him that he probably doesn't even notice. They certainly don't, anymore. It's second nature to them to search for the exits in a room, to avoid waving arms and raised voices, and to read people to make sure they would not be on the receiving end of anger.
They notice that he is fast, just like Pansy. He moves quickly and succinctly, never taking unnecessary steps in unnecessary directions. He knows shortcuts through the school. He knows the timing of the staircases and when people generally frequent certain hallways. He knows how to get away if anyone starts chasing him, just like she does. Where she pictures scissors to cut hair and pinches to bruise skin, he pictures fat fists and screams of "Harry hunting."
They notice enough about Potter to understand that he's abused. What they don't understand is why he isn't in Slytherin. Abused children do so well there.