This is most likely caused by a misbehaving add-on. You can try to start Firefox in safe mode to see if that's the cause. Diagnose Firefox issues using Troubleshoot Mode

If that doesn't work, try to reset Firefox, Refresh Firefox - reset add-ons and settings

Also, you are infected by mywebsearch, you should run malwarebytes immediately to remove it. http://www.malwarebytes.org/

Note that the X is a close button and that the new tab button is styled as a plus (+) sign.

You can find the New Tab button showing as a '+' on the Tab bar.
You can open the Customize window and drag the New Tab button that shows as a plus sign (+) from the Tab bar on another toolbar and it will become a regular toolbar New Tab button.
If you can't find the New Tab button then click the "Restore Default Set" button in the Customize window.
If you want the New Tab button at the far right end of the Tab bar then place a flexible space to the left of it.

You can check for problems with the localstore.rdf file.

• http://kb.mozillazine.org/Corrupt_localstore.rdf
• https://support.mozilla.org/kb/Toolbar+keeps+resetting