I also have this problem. I have tried setting browser.sessionstore.interval to a higher number, which helped somewhat, but did not eliminate the problem.
The lag tends to occur at points where the video changes rapidly, or there is a lot of movement within the video. Unfortunately, this means you often can't figure out what happens during the lag by watching before and after the lag. Stopping the video and replaying the lagged part rarely if ever solves the lag, and it always happens in more or less the same spot.
It's gotten to the point where I don't even try to watch most videos in Firefox. I watch them in Chrome instead. Hopefully this gets fixed in one of the bazillion new versions scheduled to be released this year.
I have the same problem with firefox 11, too. After restarting firefox I can watch some videos without any problems but after some time the lag comes back. I read somewhere that these lags can be caused by add-ons or plugins. Did you tried a restart without add-ons?
In my case it seems to be Speed Dial which causes the lag so I have it disabled at the moment. I have to do a longer test though to be sure...