In addition to innate or learned color perception, there are many other factors that affect color vision:
Lighting: Light plays a HUGE role in color perception. It's actually the color of the light that determines the color your brain will perceive. Examine samples side-by-side when comparing colors, preferably under controlled lighting conditions.
Age: As we get older, our perception of color starts to fade. Luckily, color perception is not just inborn, it's also a learned skill.
Poor Color Memory: Human color memory is horrible! Even looking at two colors across the room from each other to see if they match is futile.
Retinal Fatigue: Our eyes get tired very easily. When we stare at an object for longer than a few seconds, chemicals in our eyes start to deplete, and begin sending incorrect information to our brains.
Backgrounds Effects: There is a phenomenon called simultaneous contrast in which the background on which we are evaluating color strongly affects our eye's ability to correctly perceive the color.