Basically compacting works just fine.

To avoid problems follow these guidelines.

http://kb.mozillazine.org/Keep_it_working_-_Thunderbird

Wow. It works for you and you've never had the problem, so it's not a problem. Thanks for the insightful response. I guess all of the people who have the problem are at fault for not catering to Thunderbird.

Keep it working???? ARE YOU OUT OF YOUR MIND????

Compacting works GREAT. Most of the time. Until it loses a bunch of mails. Then it's the user's fault???

Why should I have to pay scrupulous attention to a process in order to keep a program from losing data? Because the programmer who wrote the botched compact routine never heard of FILE COPY?? Make a safety copy of the file first - then compact it?

The very idea that it's OUR fault that Thunderbird alerts that the Inbox is full at 27 messages ... is OUR fault?

Wow Christ - please do EVERYONE a favor - and stop trying to help

Thunderbird has probably been begging to compact for a long long time, unless it was turned off, if it is full with 27 messages. We changed the default behavior to see compacting occurring regularly instead of the edge case where the inbox is already full and data has probably been lost years ago. Bug 794303 landed late last year to try and fix some parts of the 4Gb limitation, and that will most likely be in the next mainstream release see https://bugzilla.mozilla.org/show_bug.cgi?id=794303

So lets get some things straight. The 4Gb limit is a given. It is at this time immutable and must be accommodated or you loose data. Ranting, and being generally disagreeable will not change that situation, nor will it modify the way Thunderbird does it's job. If you want to do something constructive you might test current daily builds and see if the problem is genuinely resolved. Daily builds can be downloaded here http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-central/

Keeping you inbox basically empty is good advise. I don't take it, but I also know the consequences. One day I will probably loose the contents of my inbox. All of the things contained in the article you were refered to are good advice. So no Christ1 is not out of their mind, but I seriously doubt the mental ability of those that think sensible advise if abuse worthy. Complete blissful ignorance got your user here. Please help them move on by educating them about the complex software they use on a daily basis.

Compact is not file copy, nor is it anything like it. Before making any more such fallacious statements please examine the source code, it is here http://mxr.mozilla.org/comm-central/source/mailnews/base/src/nsMsgFolderCompactor.cpp Perhaps you can point out where the errors are and file a bug to have them fixed if you are to busy to code them yourself. Bugs need to be filed in Bugzilla https://bugzilla.mozilla.org/ I am sure there is a volunteer somewhere with free time to examine the code needed once the exact issue is identified.

Just remember that the people you are dealing with are not paid they volunteer their time here. Simply because you do not like the answer you are given is no reason to demonstrate your lower social skills.

Matt, I apologize if you think I demonstrated lower social skills.

I wrote to complain about a problem that I and many other people have posted about and your friend Christ wrote back to tell me that we don't really have a problem in the first place -- except maybe self induced.

No user of ANY program should EVER have to take steps to protect himself from the program accidentally losing his data.

That was offensive to me and I responded in kind. I find it VERY offensive that people defend software bugs by accusing the user of not operating the program properly.

My most recent case was a Windows 8 system on which Thunderbird had been freshly installed for a total of 11 days - and so no, Thunderbird had not been begging for compaction for a "long long time." In fact it never begged for compaction - it simply started alerting that there was "no more room" in the INBOX. 11 days and 27 messages and nowhere near 4Gb.

It is fascinating to me that someone would go so far as to make up their own scenario in order to keep their world-view safe.

As far as file copy being a compaction ... where did that come from? As I recall, what I said was that the PROGRAM should make a safety copy of the file and index before it compacts - so if the user finds missing messages the process can be undone.

But then, since the data loss is the user's fault in the first place, that is unnecessary, right?

An afterthought

I am absolutely thunderstruck that you think the end user is ignorant and that HER ignorance is what lost her emails. The program is fine, the user is ignorant. And yet YOU, Matt - were the one that made false and ignorant assumptions about the facts and circumstances

Just .... wow.

DB - I have your answer, although it's not the answer you were hoping for. For reasons passing my understanding the Mozilla Support Forum is not the place to go to discuss deficiencies in Mozilla programs. You will invariably rile egos, just as you have done and garner responses that are non productive, as you have also done. The top contributors have volunteered a great deal of time and effort and are understandably vested in their work and the product itself and many of them infer a personal attack when you discuss the product design. It's 10 times worse on the Firefox boards.

As soon as someone suggests to you that you should debug the source code yourself, which carries the mean-spirited implication than non-programmers are expected to just live with whatever behavior exhibits, you should know that the train, so to speak, has come off the tracks.

Do yourself a favor and mark the thread solved. The programmers will either find the problem and fix it one day or they'll never find it and fix it, but either way discussing the problem in public just makes people angry.

I agree with you on one thing though, it is the problem with free software.

As an explantion, I can offer this info. Emails are not stored in individual files. They are stored in mbox files, one email listed after the other in the order they were downloaded. So all emails in an Inbox folder as seen in the Folder Pane are actually all in one Inbox File. When you choose to delete a message, that message is 'marked as deleted' and from your visual point of view, it is removed from the Inbox Folder in the Folder Pane and reappears in the 'Deleted' folder. In reality, it is still in the Inbox just 'marked as deleted'. When you compact the folder, those 'marked as deleted' emails are removed, thus creating more space and tidying up the file. If you do not compact the folder on a frequent basis, the file gets larger and larger and at some point may become corrupted due to a lack of maintenance. Then Thunderbird may not be able to tell where a marked as deleted section ends, so resulting in the lose of a good email.

So, like all documents etc on your computer, you need to do a couple of basic things to reduce the risk of lose of data.

Use the information you learned from the 'keep it working' link. Use the Inbox folder as an Inbox folder and not as a general storage. Compact folders on a regular basis. Create backups on a regular basis. Use 'Archive Options' to reduce the size of folders, including the 'Sent' folder. https://support.mozilla.org/en-US/kb/archived-messages?esab=a&s=archive+options&r=0&as=s Archived folders are still within the Thunderbird Profile and so still accessible and also require backing up.

Losing emails is a pain and can be inconvenient to say the least. But if you follow the guidelines then you reduce the risk of experiencing an issue. I will manually compact my Inbox each day as I will probably have deleted a dozen emails. I backup once a month and leave emails on the server for a month as well, so that between the two, in any event of lose, I can recover probably most if not all emails.

This hopefully explained what probably occured and how to manage things so that in the future you are less likely to suffer from corrupted files and in the more unlikely event that you do suffer a loss, then there is a backup plan which you can employ.

Thanks Toad -- I'm familiar with the structure of the Inbox. Most mail systems in Unix/Linux use that same concept, although I've witnessed unattended mailboxes in Linux amass thousands (maybe tens of thousands) of messages and never lose one.

I, personally, have been using TB for years and my Inbox contains over 17,000 messages and to my knowledge, I've never lost a single one. In checking my settings, TB is set to automatically compact when it can recover 20Mb. ** And I'VE NEVER HAD A PROBLEM **

So you can imagine my surprise when the first time a client reported losing email messages, but I resisted the temptation to tell him that since I had never lost messages the program doesn't lose messages, or if it does lose a message, it was somehow HIS fault. Instead I researched and found other people reporting similar issues.

My latest case has been the most baffling because it was a brand new installation. Her XP system died hard and we gave her a brand new Windows 8 system with a new copy of TB (no import, no conversion, no nothing) and after 20 messages TB alerts that the Inbox is full. Compacting deleted 16 of them. Lost forever. I made a copy of her INBOX at that point, moved it to a Linux system and used string matching tools of various sorts to verify that the messages themselves were GONE from the inbox... not simply missing from the display.

We instructed the user to be scrupulous in moving things from her Inbox and she was! The next time she called, two days later, she had a folder (not under Inbox, but separate) for every person! The problem was she had just received several messages and when she went to open them they were blank. I immediately copied the entire profile, moved it to a different system and explored and what I found again was MISSING MESSAGES. The difference, this time, was that the message index was there ... so you'd see a message from "Jack" subject "About that meeting Tuesday" but the INBOX contained no message body at all for that message, so when you click on the message you see a blank message.

The "repair" option does repair this ... by removing the message data from the index (Msf, I think) -- and now the MESSAGE is gone *AND* the index to it is gone.

Now - something is BADLY wrong with this installation -- no one would argue that. No one (even me!) is saying that ALL of Thunderbird is like this or that it happens to everyone or it happens ALL the time. If you start this thread at the top, you'll see that I like the product and think it has more and better features than any other program out there

-- but it DOES lose messages.

As for solutions - THAT is what makes the responses I read so perplexing. A) It doesn't lose messages B) OK it does lose messages but it's your fault C) All complex programs lose data, don't single out Thunderbird D) Here's the code - YOU fix it.

I go as far back as Data General 16bit Nova computers and I've worked with many programs and many programmers and the BEST programs are written by the BEST programmers and they have a universal characteristic: They have a lower opinion of their programs and their abilities than other people do. The author ASSUMES that reported problems are genuine and he ASSUMES that he (the author) just can't find it ... yet.

The worst programs are written by programmers that assume that unless you can duplicate the problem before his eyes - it must be a user error.

Abel's suggestion (don't bring problems to the public arena) is the only one that fits.

Have you checked the anti virus quarantine? Nortons in particular will routinely pull the MBOX file out and quarantine it because they found something they don't like. Nor do they care about the disruption their kill at all costs approach takes. Matter of fact they are proud their product "detected" something. You might try ensuring that Tool > options> Security > Anti Virus is turned on. I have mine turned off, there was a bag in V5 I think it was that lost mail if it was turned on and you used filters to move mail. NOD32 is non destructive in it's scanning and I have never had the need to turning it back on.

I have never been a fan of Norton AV and since this is an HP PC it was originally, as I call it, infected with Norton but it had been taken off and AVG installed. As many complaints as I have about AVG it's been pretty reliable about how it treats emails. Personally I tell people to get NOD32 - I think it's the best.

The situation in her first case was that the messages were known to be there - and then the indexes and messages went mia. In her second round, the INDEX contained valid data (subject line, etc.) but the inbox was missing the messages. I would like to believe that any AV program does a port/service intercept and prevents Thunderbird from ever getting near the quarantined message. Still -- I'll look outside the Inbox (to make a terrible pun).

The one user has gone to Outlook, which is fine with me.

I don't care what client they use either. I have a copy of Outlook I use for support.... but i don't like it. My wife however manages her life with it, but in conjunction with an exchange server it is a much better product. It is not really a mail client at all. IMNSHO it is an exchange client.

The issue is not usually new messages, it is deleted messages in most cases. Today I get a dodgy mail. I delete it. Next week Symantec decide it is a problem and update their signatures accordingly.

Thunderbird does not compact daily or weekly it compacts the store file when XX amount of space will be recovered. Makes for a far more responsive system. Norton Scans the file, finds last weeks deleted email and quarantines the entire thing. Not a single mail, the whole inbox.

So your left with an index showing mails that are physically no longer present. running repair removes the invalid entries from the index and the user appears on one of these forums complaining that Thunderbird lost their mail.

The truly sad thing is because the blame goes the wrong way they keep paying for Nortons.

I normally don't respond to conversations like this because the product is what it is and complaining about it won't change that. I used Eudora and Outlook and even Pine and Mutt before that and my inbox was never my enemy until Thunderbird. I find the notion that complex data bases require constant manually intervention to prevent them from randomly deleting records to be absurd on it's face. That "keep it working" article would be more aptly named "keep bailing the boat."

On a more practical level I've been watching a few installations carefully in hopes of catching an antivirus in the act. This morning we found exactly the signature we would expect to find: Intact indexes pointing to missing messages. This is what I would expect to find if AVG removed an infected message since it would know nothing of the index.

We found an index with every message after 4/22 (16 messages) all blank. Repairing the box simply loses the pointers. And that same morning, AVG excised 3 viruses. But just as I had my EUREKA! moment I saw that AVG had removed three infected messages from JUNK, not IN, so unless IN and JUNK are really the same physical file, then this was just a cruel coincidence.

I very must like the features of Thunderbird, just as the OP suggested, but based on the technical implementation of Thunderbird and Firefox I shudder to think of these same people writing an O/S.

Unless the inbox has been compacted whatever was there is still there, simply hidden from view. So the messages removed from junk would also have been excised from the inbox as well. Given my usages, and a disk space saving of 100mb which is now the default, compaction is not a daily or even monthly event. I need to receive 100mb of mail and delete it from the inbox (by moving and deleting) to trigger compaction. I use 10Mb as that in my view is a better trade off with having rubbish lying around in the file and time with a slow instance of Thunderbird.

Much of the Manual you see in the keep it working is simple good advise and good housekeeping. Sure Thunderbird occasionally looses the plot. So does any mail program you can mention with more than a few users. Keep it working would apply equally to Outlook, except Outlook uses a single file and Thunderbird uses one per folder. Outlook gobbles millions of mails through corruption annually. The professionals looking after it know this and have backups, the less professional complain and google shows millions of such complaints.

Compaction is supposed to be an Automatic process, it is a background task. It is in my instance. It runs and does it's housekeeping without interruption, but I don't close Thunderbird for days at a time and don't reboot for similar periods. But the killer difference is I do not have mail folders with more than 4Gb of mail in them, nor do I have what from my perspective looks like paranioa of "not deleting any mails". Users trying to turn off anything that in their ignorance might delete a mail.

Personally I have tens of Thousands of messages in the inbox and have no issues. But I also do not have my anti virus removing infected mail. It can tell me and I will take action Or I get a new anti virus.

No, the messages are not still there. I think this is part of why the problem doesn't get fixed. No one, including the authors, accept the testimony of the people with the problems. I opened the INBOX folder with Wordpad and the only messages in the INBOX are the three that were visible inside Thunderbird. The user did not compact and if the compact happens automatically at a certain point, how is it that 19 messages is that trigger point? When she called me in panic, the first thing she did was apologize that she'd let t he inbox get so full. (Full? 19 messages? I have 17,343 messages in MY Inbox version 1.5.0.5)

As far as blaming the antivirus, that's what I was hoping to do. But it quarantined 3 viruses from JUNK and not from IN. So if AVG killed her INBOX, it would to have found a virus in THAT box. The virus got to "JUNK" either when the user moved it or when Junk Mail Controls moved it.

Assuming the move actually leaves the message text in the INBOX when the user or JMC moves it - then the virus exists in two places (but only INDEXED in one place) -- so if AVG found it in the JUNK box and reported it, then AVG would have found it in the INBOX as well -- and reported it.

Problems that can't be duplicated on demand are the most frustrating of all and I empathize with the programmer who thinks he's done everything just right, but problems like this are found and fixed by people who accept the reports and compile the data and look for commonalities until the discern a chain of events that must cause it. What we have here is, as the OP suggested, a chain of people that seem vested in discrediting the testimony

I really like the design, the features and the way they've approached everything. Every single email client I'm looking at is a step BACK as far as I'm concerned. But emails that are there one day and gone the next - for whatever reason - no matter what anyone says - is the most serious flaw an email client can have.

If The User Didn't Delete The Messages Then The Programmers Seriously Screwed Up The Design.

Anyway --- that's why I don't normally respond. No one's mind is changed and no new action will be taken.

When you get an email to Inbox and Junk controls 'move' message to 'Junk' folder.

The email is in the Inbox folder marked with X-Mozilla-Status: 0008 The email is also shown in Junk folder (if read marked with X-Mozilla-Status: 0001 or 0000 if unread).

0008 - Messages with this bit set will be irreversibly removed from a database file during the next compaction.

So if this was a compacting issue, those emails with 0008 would be deleted and if there was some corruption in that file, TB might not know where a marked as deleted mail started and finished and so may have deleted other adjacent emails. So anyone with documents on a computer should backup files.

As the file was in Inbox and Junk, it would also be possible that the anti-virus may have quarrantined both folders. hence why, it is a good idea to make sure quarrantined files are not immediately deleted.

I'm unclear on one thing you said here. Whether the email contains a virus or not. Whether the user moved it to Junk or Junk Mail Controls moved it. Either way, the message is physically copied to a different file named "junk" and a pointer is created in a file called "junk.msf" that points to that message in the junk file. Fine. Meanwhile, the pointer to that message is removed from "INBOX.MSF" and the message in the INBOX file is set to status 0008.

That's what I have understood from you (and thanks, by the way).

So now AVG2014 comes along and scans the JUNK mailbox, finds a virus, quarantines that virus and leaves a message in the AVG log. If AVG scanned the INBOX and found that same message with the same virus it may very well quarantine that message as well, but it would leave a message in it's log regarding that file, too.

Of course I can have them back up. They can backup every hour on the hour it they want. Right now they are actually putting clients on hold to read and move emails from the INBOX the instant they come in before they lose them. That's no way to live.

The AVG does a good job of scanning incoming emails, so one possible solution is to exempt the profiles folder from the disc scan, but the problem the users are witnessing doesn't fit the profile of what we're assuming happened.

Cannot resist putting my 2 cents worth in this. I have also been using TB since like I believe v2, am now at 24.6. In all that time I never had any issues with compacting, no issues where INBOX items "simply vanish". At the peak of my email reception, back when I was working one particular contracted job, I was receiving on average 200 to 500 emails a day. And, yes, never once any issues (mind you that was back at like vI think 14). So, as to what is ACTUALLY happening here, I could only guess. I too have seen all those postings on the net about TB "losing emails", and just sometimes wonder if it really isn't something on that particular users machine or some weird combo of software causing the issues. Well, anyway, I just figure I'd chime in with this post. I mean , why not?

Why not indeed cyberWolf.

There have been bugs, and there have been a number fixed in the past couple of years, mostly related to mail folders greater than 4Gb in size. By far the greatest issue faced by Thunderbird these days is the assault of anti virus programs. They appear to be becoming more aggressive in their approach and more difficult to use with non Microsoft software.

I missed Stans post in February, but he appears to be under the misconception that mails are treated by anti virus programs as files. They are not. Folders are treated as files, because they are. Folders consisting of an MBOX file with the mail in it and an MSF file that indexes the folder and provides the information to populate the various mail list boxes.

What appears in the AV log is something like Virus XXXX found in file inbox. And the entire inbox is gone (although Thunderbird may not immediately display the issue as the MSF file still exists and the lists are populated from that.

For daily reception Thunderbird has an option in Tools > options > Security > Anti virus which causes each vnew incoming email to be written to a temp file before it is appended to the main mail store. This allows the anti virus to quarantine the temp file and when it goes missing Thunderbird writes nothing to the mail store. But that is not the anti virus program playing nice, it is Thunderbird's developers trying to get out from under a yolk imposed by the actions of Third party software.

What is most troubling to me are the staunch defenders that assert that since it can't be reproduced on demand, that it doesn't exist.

My latest incident was MichaelD. His system crashed, no backup, so we wiped the disc loaded a fresh Windows 7 all the updates, etc. and Thunderbird. Having had problems in the past we have made our people paranoid about the INBOX and tell them to clear it every time they read an email.

One week after installation, he came in on Tuesday and found 7 emails. He read them but did not move them. After lunch there were 3 more emails - however the 7 that had been there and had been read were completely blank. I copied his INBOX to a Linux system and used VI (and even od -c) to read the inbox -- and the messages were just gone. The Index still had the subject, etc. but "repair" killed those quickly.

The Antivirus had not run a scan, nor did its logs report anything at all.

I have 14,350 messages in my IN box on Thunderbird 1.5.0.5 (20060719) and perhaps I have so many I'd never know if I lost one -- but that's beside the issue.

As long as the core programers believe that their code is fine and people are using the programs incorrectly (like that preposterous "keep it running" article) the problem will never be truly solved. Bouyed by comments that "all sophisticated software required maintenance or it will lose data" the programmer has no reason to believe the fault is his - and that "someone" must be doing "something" wrong.

How many software packages in the world make "safety copies" before they update so that we can roll back on error? Is that a sign of defeat? An admission that their code is faulty?

"I'm about to automatically execute some code who's design purpose is to modify the user's data without their knowledge of what I intend to do -- I wonder if making a time-stamped copy of the Inbox and Index might be prudent?"

Guess not.

As a Thunderbird update does not change the profile data (the update from 2-3 added a new file for passwords and some conversion just did not work) there is Zero point in making a backup of something that remains unchanged. I would really get nasty with someone if my 10Gb profile started duplicating itself.

Is windows allowed to make restore points? I suggest disabling that functionality. I have not seen anything on Windows 7, but I have seen reports of Windows 8 silently restoring the system to a earlier point following a start issue. (it just works) and mail files being lost in the shadow copy restore of Thunderbird to an earlier point.

You can open an MBOX file in notepad if it is small enough. Is there a consistency of AV, security or management software across the systems. Keyloggers, internet activity monitors, general snooping software that some businesses install, all could be implicated.

I do not claim there is and never has been an issue, but what leads you to think the compact process removed the data, or even was executed in the time frame? I just realized there is no log reference to compact running