Buried like some deep ancestral memory, I vaguely remember that my dad had a jacket and swim trunks set like this ca. 1960. I think they were called cabana sets.
My dad's was a Hawaiian print with flamingoes on it (uh-huh). Though it's painful to admit, he wore black leather shoes and black socks with it. Of course, being a tot, I thought this made him look very sophisticated. A few years later my mother, who never threw anything away, threw this outfit away, and a fashion trend died (at least in our house in Brooklyn).
This photo is probably ca. late 1940s/early 1950s. I really like the gal's simple set but the men's line still looks awfully goofy to me.