Despite the ubiquity of these expressions, "nother" isn't really a word. The speaker is splitting up "another" by dropping "whole" in the middle of it.
What would be more correct? Standard English offers some alternatives to "a whole nother thing"—"a whole other thing" or perhaps "an entirely different thing"—but these don't really cut it. The first comes off as stiff and awkward, and the second is, well, an entirely different thing. So although "nother" may not be suitable for business or academia, English speakers' natural feel for their language's texture gives "a whole nother thing" a certain degree of legitimacy.
"Nother" has even made it into dictionaries as a misdivision of "another," and the expression has been around at least since the country's bell-bottomed days. It appears on 1970s funk albums. When young Luke Skywalker was stuck on a remote, desert farm in Star Wars and his uncle delayed his plans, he complained, "But that's a whole nother year!"