What is color and how does it relate to how we see it

I have a hard time grasping this. First I know that white light that passes through a prism - which itself is a transparent object with flat sides (think glass pyramid) - defracts, bends and then the colors show depending on the angle of the defraction. The group of colors i called the spectre, like the spectre of the rainbow is the collection of the colors within it. But I also know that we see color since objects absorb or reflects light (if its all absorbed it will deflect black, nothing absorbed the obejct will deflect white light). But how do these two examples go together? Is the eye a kind of prism and thats why we can "process" it?

No, the eye is not a prism. We have cells on or retinas called cone cells, and they respond to the different wavelengths of color and send that information via the optic nerve to the optical center at the back of the brain. Part of this area is for the interpretation of color, and so the brain "decodes" the colors for us. It is because of the different wavelengths that the cone cells are able to differentiate between them. You are correct in your analysis of the physics of light, but we process color in the brain, not the eye. The eye reports to the brain which wavelengths we are seeing, and it is put together for us in the optic center of the brain.
People who are genetically unable to differentiate colors well, the most common condition being red/green color deficiency, have cone cells in their retinas which do not react properly to certain wavelengths of light and so their brains interpret these colors differently from average people. The problem is in the cone cell structure, not in the brain.
Seeing different shades of color is a result of activation of more than one type of cone cell. For example, to see pink, the wavelength of red light is mixed with white so that the red is pale, which we call pink. Navy blue is blue mixed with black pigments, while light blue is mixed with white. The cone cells report both wavelengths to the brain and it interprets them as different shades of colors.