There is still a huge problem compacting mailboxes
Once again I heard from someone in great distress because Thunderbird took away some critical mail messages. In this case they went to get mail and TB gave her an ALERT and told her there was no room in her IN box (there were a whopping 20 messages). Thinking that the file was too big (which is a logical deduction from a 'no more room' message) she compacted her IN box and then there were 4 messages. The remaining messages are gone.
And worse - all the information on the forums are explanations and defense of TB - as if it was her fault for having compacted, not backing up or (my personal favorite) - an article that says TB is just doing what she told it to do.
When we combine this with all the other message traffic here and on other sites regarding the loss of emails the one thing that is crystal clear is that the module or modules dealing with the indexing and compacting of email boxes are tragically flawed. Sadly, and this is the #1 problem with free software - the people building the product are doing it for ego, not money, so once they've decided that their program is right and the users are wrong - there it sits, cast in concrete.
Thunderbird has been and still is a leader in form and features but losing user data - for ANY reason - EVER - is and will always be the fault of the developer and no user - EVER - should have to alter his behavior in order to avoid a problem that the developer doesn't know how to fix.
Of course there are solutions - each and every one a work-around to avoid a bug.
Chosen solution
As an explantion, I can offer this info. Emails are not stored in individual files. They are stored in mbox files, one email listed after the other in the order they were downloaded. So all emails in an Inbox folder as seen in the Folder Pane are actually all in one Inbox File. When you choose to delete a message, that message is 'marked as deleted' and from your visual point of view, it is removed from the Inbox Folder in the Folder Pane and reappears in the 'Deleted' folder. In reality, it is still in the Inbox just 'marked as deleted'. When you compact the folder, those 'marked as deleted' emails are removed, thus creating more space and tidying up the file. If you do not compact the folder on a frequent basis, the file gets larger and larger and at some point may become corrupted due to a lack of maintenance. Then Thunderbird may not be able to tell where a marked as deleted section ends, so resulting in the lose of a good email.
So, like all documents etc on your computer, you need to do a couple of basic things to reduce the risk of lose of data.
Use the information you learned from the 'keep it working' link. Use the Inbox folder as an Inbox folder and not as a general storage. Compact folders on a regular basis. Create backups on a regular basis. Use 'Archive Options' to reduce the size of folders, including the 'Sent' folder. https://support.mozilla.org/en-US/kb/archived-messages?esab=a&s=archive+options&r=0&as=s Archived folders are still within the Thunderbird Profile and so still accessible and also require backing up.
Losing emails is a pain and can be inconvenient to say the least. But if you follow the guidelines then you reduce the risk of experiencing an issue. I will manually compact my Inbox each day as I will probably have deleted a dozen emails. I backup once a month and leave emails on the server for a month as well, so that between the two, in any event of lose, I can recover probably most if not all emails.
This hopefully explained what probably occured and how to manage things so that in the future you are less likely to suffer from corrupted files and in the more unlikely event that you do suffer a loss, then there is a backup plan which you can employ.
Read this answer in context 👍 0All Replies (5)
I have 14,350 messages in my IN box on Thunderbird 1.5.0.5
Is that the version used by the people having problems?
I have 16,339 in my inbox on 33.0a1 (Daily build) but my usage has covered all major and most minor versions of Thunderbird since V2 without compacting data loss. I had issues with V2, but I really do not remember what they were and got caught by a bug that lost mail if the anti virus quarantine option was turned on and you moved messages with filters. (they just were no more)
I have in the last 2 years installed IMAP accounts, just to see if I can duplicate some of these issues. I can not.
All valid points, Matt. I'm having a problem visualizing the process. An IN box contains 7 messages and the Index file contains 7 indices. The mails are read and marked read. At some point 3 hours later
the 7 messages are gone from the IN box the Index retains all 7 3 NEW mails are added to the IN box AND Index (now containing 10 indices)
The only factor that points me in the direction of Thunderbird is that the IN box is not DAMAGED. There are no message fragments. The three messages in the box are intact and the box itself is intact. As theorized by others - *IF* an AV program attempted to clean an infection, unless it had specific understanding of the inbox message structure, it would trash the entire box.... so we should be able to rule that out.
Unless the typical report is A) The box structure is damaged B) ALL Emails are gone C) ALL Emails after a certain date are gone --failing that, it's hard to visualize an outside process doing the deed (too precise)
As far as compacting being the culprit, you touched on an important issue. MichaelD's problem did not happen in a vacuum. Nor did Toni's similar problem (27 mails lost). Danny kept getting a pop-up (or message, I can't recall) saying that his Inbox needed compacting and when he finally did it, several HUNDRED emails were gone... yet the indices remained.
These were different installations, different versions, etc.
What you reminded me is that I've made an unwarranted deduction: Since I have on several occasions lost emails specifically to the compact process - it does not follow that ALL losses are due to the compact process.
And it's damned hard to debug a problem you can't see. Staring at code and trying to wonder how it's getting the better of you when you just can't see it -- and worse, when you believe in your code... makes you feel that people that complain are just haters.
More verbose logging, perhaps?
Still - the compact process is about to modify the INBOX -- it can timestamp a copy of the inbox. "Repair" the Index? timestamp a copy of the index first. Not rocket science.
What I can say is the compact process creates a new file and deletes the old one.
So the compact starts and the first mail to be "kept" is written to a file called NSTMP and so on until the compact process completes. The original file is then deleted and the NSTMP file is renamed to replace it. You can actually see this at work by force closing Thunderbird while it compacts and a collection of NSTMP folders will start appearing in the local folders. one of each close, as the compact process is sequential and only does one folder at a time, even if you select compact folders from the File menu to compact all folders. These are the compacted folders, the originals are right where they always were. There is one small part of a second between rename start and rename end that could cause issue, but you would be very unlucky to loose power exactly at that point in time
It also covers your "backup to back out" scenario from earlier I suppose. Now this code is fairly old, as it was written by Netscape, but it has had some serious review in the last few tears and a number of bug with it's processing closed.
There is one thing I have not checked, and perhaps should. but the write to file for mail uses the index to get the file offset. So if the file is missing and the offset says after the existing 10 mails it would be probable that a new file would be created (that happens automatically if the file is missing) and it would create empty space followed by the new mail as you suggest it appears.
My experience is that;
- Outgoing mail does not need virus scanning, just another failure point without real purpose.
- Exempting the Thunderbird profile from any "on access scanning" is also helpful as anti virus does not need to scan the store file every time is is opened having just scanning the incoming mail. It just slows things down and again opens a failure point if the scan locks the programs access and a script times out (everyone says stop). This is about the same as exempting an exchange store from virus scanning
- Try one of the troubled on another anti virus program. I pay excessively for ESET nod32 because it just works, (so far it's only been three years), and has a server version for my windows server. I have in the past tried AVG, McAfee, Trend, Nortons and MSSE. Nortons were alarmist putting up red screens all the time and basically lying about risks. Tracking cookies are a fact of life and having an AV program harping on about they was just wasting time. AVG was just clunky and not reliable McAfee just did not appear to detect stuff and the same for Trend. MSSE made Thunderbird unusably slow with script timeouts every few minutes and a typing lag that made me look like a champion typist.
- Make sure people are using a current version of Thunderbird, The compact has been seriously improved in 24. You might even "promote" someone to the earlybird release. Just to see if it is a different experience. Restrict the seriously prone to no add-ons for a while.
Upgrade from 1.5 yourself. One of the major reasons compact has gained extra profile is that a decision was made to turn it on by default. Many people never compacted, hence the bugs etc that existed prior to V3 were really hidden as, if you never compact you never see the missing mail, because it never appears to go missing. but people mail stores have grown exponentially in recent years and folders with 10gb of mail in them are not unusual (thanks Google, I think).
All of the compact bug I have read have involved fixing issues that have been there since version 1, not fixing issues that were introduced in latter versions.
This topic is a bit too long too ready everything, but I have skimmed it.
One of the most common causes of folder loss is related to AV software. Some AV are better than others, but I wouldn't say that any AV package is immune from causing trouble, because it is essential that the AV software not scan Thunderbird data, the files in the Thunderbird profile.
1. The loss of Thunderbird data need not require that the AV software found a virus. 2. Thunderbird updates can trigger a problem, because sometimes the AV software doesn't recognize that it shouldn't scan files used by the new version - so past good behavior of AV software is NOT necessarily a solid predictor of future good behavior.