Can browser.tabs.unloadOnLowMemory be enabled and will it work

I don't think so. Some work has been done under https://bugzilla.mozilla.org/show_bug.cgi?id=1752625 and https://bugzilla.mozilla.org/show_bug.cgi?id=1782178, but it looks like it's not ready yet.

Will it get triggered when SWAP is being used? Or when system free RAM falls say to about 25% of total physical RAM? Or will it use the commit memory to determine when to trigger?

Tab Unloading is a feature that automatically unloads tabs to prevent Firefox from crashing due to insufficient memory when the system’s available memory is low.

The feature consists of two parts: memory pressure detector and tab unloader. When the memory pressure detector detects a low memory situation, it triggers the tab unloader that prioritizes tabs and chooses a tab to unload, or if there are no unloadable tabs, triggers the internal memory-pressure warning allowing the browser’s subsystems to reduce their memory use.

There are two modes to prioritize tabs.

Firefox basically unloads tabs in least-recently-used order, excluding tabs playing media, using Picture-in-Picture, or using WebRTC. Pinned tabs are deprioritized and are less likely to be unloaded.

When there are more tabs opened, in most cases when there are more than eleven tabs, Firefox does extra calculations identifying processes hosting tabs and estimates memory usage of each tab, and then unloads tabs with larger memory and more processes that will be terminated by tab unloading.

The about:unloads page shows how Firefox prioritizes tabs and which tab will be unloaded when the tab unloader is triggered. You can trigger tab unloading manually by clicking the Unload button in the page.