Is the problem only with YouTube? What about other sites? Are you using a short-cut to get to YouTube?

What is in the unwanted window?

Start Firefox in Safe Mode to check if one of the extensions ("3-bar" menu button or Tools -> Add-ons -> Extensions) or if hardware acceleration is causing the problem.

• switch to the DEFAULT theme: "3-bar" menu button or Tools -> Add-ons -> Appearance
• do NOT click the "Refresh Firefox" button on the Safe Mode start window

• https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-mode
• https://support.mozilla.org/en-US/kb/troubleshoot-extensions-themes-to-fix-problems

You can also check for malware.

• https://support.mozilla.org/en-US/kb/how-does-phishing-and-malware-protection-work

So far, it's only on the YouTube subscription page. It doesn't happen in Chrome or Edge. When I finish watching a video and go to the subscription page, the video I stopped watching continues playing in this new window. I've turned off notifications and pop ups, restarted Firefox and my computer, but the window is still there.