It is quite normal to see "Use custom settings for history" selected and it is likely that nothing is wrong and no need to worry. You only need to understand how showing custom settings works. You may have made changes to the default history settings if the custom setting gets selected. "Use custom settings for history" allows to see the current history settings, but selecting this choice doesn't make changes to the history settings.

The "Use custom settings..." choice will be selected when one or more history settings are not default to make you aware that you made changes. Only when all History settings are default then the custom settings are hidden and you see "Firefox will: (Never) Remember History". Private Browsing is active in case "Always use private browsing mode" has a checkmark.