No matter how unattractive the leaves are, if they are green, they stay. Yellow, brown etc. you can cut off. Banana leaves may begin to look very tattered after heavy winds. This is perfectly normal. The banana leaves split into strips allowing wind to pass through more easily. If the leaf stayed in one large piece it would be easier for the plant to be knocked over or damaged.
When cutting the leaves, be sure to have a sharp set of gardening scissors. DO NOT pull or rip the rest of a leaf off if there is some left over. It will tear all the way down the stem and damage the plant.