In my opinion, a casual, comfortable look can also be chic. It doesn't necessarily mean dressing in baggy sweats. When I tell people that my casual look is a button up with denim shorts or black pants, I get those side stares! Looooook, loose button ups are comfy to me AND they're simple.
White button up with gold studs: diy/thrift store - .99