The Oxford Dictionary of English Etymology on yes:
word expressing an affirmative reply (peculiar to Eng.). OE. ġese, ġīse, ġȳse, prob. for *gīese, f. *ġīa sīe 'yea, may it be (so)'; formerly used spec[ifically] in response to a neg. question [contrary to] YEA.
OE. ġe, (WS.) gēa, corr. to Ofris. gē, jē, OS., OHG, jā̆ (Du., G. ja), ON. já, Goth. ja, jai; ult CGerm. *ja, *je, which was variously modified through stress or emotional emphasis. The standard sp. yea and pronunc. jei show retarded development, perh. partly due to assoc. with nay (but the normal development jī is current locally).
In other words, yeah and yea are older than yes. A form of yeah is found in most Germanic languages. yes is specific to English and was formed from (a form of) yea plus sīe "may it be so" (from PIE *h₁es- "to be".)
The ODEE states that yes was originally used for an affirmative response to a negative question - much like French si. Could si also be derived from a form of "to be"?