A sensible approach would be similar to the nice warning given when a script takes too much CPU, offering the user the choice to continue or stop the script when a tab exceeds some reasonable size: a fixed MB size, or say 10% of real RAM memory.
Ideally there should also be some hard limit calculated by Firefox according to the machine's available memory and Firefox's reasonable percentage of it. For example, maybe limit Firefox (all tabs combined) to 50% or maybe 75% of real memory, so that the O/S has a reasonable chance to remain operational so that a bad tab can be closed. Otherwise, Firefox could cause the O/S to lock up, resulting in data loss if there are tabs or other apps outside of Firefox that have unsaved changes.
Thanks for your consideration!