Institutional critique is when people talk about things they don't like about big organizations. It's like when you're in a big school and you say you don't like the rules or the way the teachers teach. When grown-ups talk about institutional critique, they're usually talking about big museums, galleries, or even governments.
They might say things like, "That big museum only shows art by certain kinds of people, and it's not fair to other artists," or "The government only listens to rich people when they make decisions, and it's not fair to everyone else."
People who do institutional critique want to make things better by pointing out what they don't like or what could be improved. It's kind of like how you might tell your parents if you think something is unfair or if you don't like something that's happening in your school.