
The World Color Survey is a massive project which attempts to understand how colors are categorized in different languages. The researchers studied 110 different languages, none of which had a written component, which ensured that only spoken word categories would be used to describe the colors.
Forty-one of the 110 languages in the survey have words corresponding to each unique hue. In each language, the range of colors identified as corresponding to the gloss for each color spans nearly the identical range of hues. Here's a graph summarizing those results for 38 of the languages (3 were eliminated from analysis because too few participants spoke those languages):

But what about the other 69 languages? Their results actually followed a similar pattern in most cases. For example, 40 of the languages had a term corresponding to "grue" -- a mixture of green and blue. These languages still broke the colors into similar categories, but lumped two of the categories together. Even among languages that had all four color categories, "blue" and "green" were the most frequently confused colors. Other languages had words for "grelow" and "relow," but overall, there was close to 90 percent agreement on the boundaries of color categories.
Kuehni argues that this is evidence for common color categories in all humans. He's careful to point out that while this doesn't rule out the possibility that colors are culturally determined, given the wide range of cultures studied here, it's strong support for the idea that color categories are determined neurophysiologically -- that the way we categorize colors is embedded in the structure of the brain.