This thread was archived. Please ask a new question if you need help.
Spam filters need a "does NOT end with" option
I don't think it's news that the proliferation of TLDs has just meant more spam. But legitimate email so often is misclassified by Thunderbird's "adaptive filtering" that I have to wade through my spam folder every day anyway.
My current main filter is called BigSpam, and nearly every day I have to add a new TLD to it. All entries are: From -> Ends with -> TLD
Just today I added ".fun" TLD. Yesterday it was .ir (Iran). Last week I added .za (South Africa). And of course .cn, .mx, .info, .xxx, and .ru were charter members of my spam filter.
What I would like, and what I think the vast majority of users would like is a filter that says, "If the from address does not end in .com, .net, or .org, trash the message. Don't leave it in a junk folder, that i have to check to make sure I didn't miss something important. I don't know anyone in South Africa, and while this isn't a slap at South Africans, most of them know no one in the US. Maybe everyone (except the spammers) would be better off if we could block the TLDs coming from foreign countries we have no contact with.
Here's what it could look like: Radio button "Match ALL of the following"
From Does NOT end with .com
From Does NOT end with .net
From Does NOT end with .org
Modified by maegenweird
All Replies (18)
re : Radio button "Match ALL of the following" That means ALL conditions must be true in order for filter to work. So, if email is a from a .com address then it cannot be from a .net. Thus filter would not work. You would want the filter to match ANY of those conditions.
You could set up a filter which if email address is not in your address books then Move message to a specific folder.
But in all honesty, there is little point in you trying to manually write filters for general Junk as this will run into several thousands; it will consume loads of your time; need constant updating as Junk mail is often slightly modified to get through such filters.
Message Filters is designed to eg: move etc good mail into folders for organising.
Thunderbird Junk Controls are designed to deal with the massive amounts of Junk. There is a special bayesian filter designed to be trained by you to cope with tons of junk.
Suggest you give the links a read and use them to get Thunderbird to work for you.
Added a couple of images as guide to complement Junk Controls info.
I don't mean to be insulting, but it seems clear you don't understand what I wrote. Perhaps I had better explain again, and see if I can make it clearer.
Let's say there is someone who wants mail from China, Iran and Russia, but nowhere else.
Radio button "Match ALL of the following" means the message has to match all the tests below. In comes a message from Germany, .de and what happens?
From Does NOT end with .cn (China, does not match so TRUE)
From Does NOT end with .in (Iran, does not match so TRUE)
From Does NOT end with .ru (Russia, does not match so TRUE)
All tests are TRUE, all match so the action taken is: Delete message.
If they receive a message from Iran, the second test is false, and since all test have to be matched, the message is not deleted. This is EXACTLY the way to deal with the proliferation of TLD domains almost exclusively used to send spam. If enough people implemented a spam filter like what I listed above, the pressure to create new TLDs would drop to near zero.
Think about it - if every spammer using the .xxx domain KNEW that 99.9% of Firefox users were dumping their mail straight into the trash unread, they might get a real job, and save at least some of the estimated 50% of all messages which are spam. And Thunderbird would have a temporary advantage, while Outlook and the rest get around to writing a similar filter.
The point remains that if you want to delete scads of messages without having to review your junk folder regularly, this is a quick and easy way to do it. And since I get about 50 spams a day, and often have mail falsely sorted by the spam filters into the spam folder, and then have to periodically wade through the collection of spam for what shouldn't be there, I think this is a good solution worthy of implementation.
I have tried to train the spam filters, and I never seem to get them trained up. In part, I'm sure it's because I am an eclectic surfer, with wide interests. In any case, the spam filters dump everything into the spam folder, when with just a LITTLE help, a great portion of them could be deleted instead of being saved for examination.
I don't care if the Prince of Nigeria REALLY DOES want to give me $500,000,000 dollars. I would rather ALL mail from Nigeria, and every other country where I don't know a single soul, were dumped straight in the bit bucket.
If someone will give me some hints on what source code to look at, I'll learn whatever the language is and do it myself.
Modified by maegenweird
+1 to the advice given above. For myself, I can't know in advance for sure what TLDs will be used. You have a US-centric viewpoint that seems to say only US-originated email can be of any interest. My own eclectic interests lead to me needing to read emails from all over the world, such as from mozilla or Thunderbird developers.
But you can install FiltaQuilla, learn to use regular expressions and implement your "doesn't end with" like this:
Modified by Zenos
Well, I did give an example of a person who wanted only email from China, Iran, and Russia. Besides, there's an "is in my address book" filter for making sure those you correspond with get mail through even if they are in a spam-friendly country.
I can't make anyone do this, but just because a few people don't like it doesn't make it a bad idea. There are both "contain" and "doesn't contain" as well as "is" and "isn't" filters. Are you really trying to convince me that "ends with" paired up with "doesn't end with" is a bad idea?
Still I appreciate your information about FiltaQuilla. I don't mind rolling my own. I'd just like to help the Internet too, providing a simple way for people with a country-centric view to eliminate trash from their email.
Isn't the purpose of Thunderbird to make email better for everyone, attracting users by providing the best experience?
Modified by maegenweird
Define best experience... not getting your mail delivered because someone puts is a TLD filter is really not a good experience. Regardless of your opinion to the contrary spam can not be categorized by the TLD of the originating email. Especially as spammers very rarely use a real email address. They "borrow" someone elses. If you are genuinely getting marketing emails from loads of locations one of the things you need to be doing is applying pressure to your national legislators to fix what is clearly broken. In my country to send a marketing email to someone who has not asked to receive them is illegal and the level of spam I get reflects that this is not a right of sellers in this country to cram your inbox with their rubbish.
If you want to talk about spam friendly nations, the United States of America probably wins as the most spam friendly nation on the planet. It is one of the reasons they are so plagued by spam. Their legislation on it's regulation is very weak compared to just about every other nation. In the US the "right" of the marketer is considered more important than the recipient. But don't take my word for it about spam frienly nations, have a look here. https://www.spamhaus.org/statistics/countries/
Now as for your specific request with an ends with. I think that is exactly described in bug 1381373. This was declared invalid only 4 months ago.
Thanks, Matt, for your response. It's clear to me that this is a political issue that I stepped into unknowingly. The link to the bug report confirms it, although as noted, the issue isn't a bug, but is a request for an additional and useful feature that at least some people clearly want to help them deal with email.
I did not intend to get in a political discussion over whether the US is spam friendly or not, or whether China is worse than the US. I certainly did not intend to get into a fight with 3 top contributors to this forum. But you can't convince me that this isn't a useful feature, particularly when all 3 of you above have given political or personal reasons why you don't think such a feature should be added.
Best experience is giving people tools to allow them to construct filters that support THEIR needs. Expecting people to adapt other tools, like FiltaQuilla, doesn't qualify as best. Downchecking a tool requested by people because you don't like it doesn't qualify as best. Upvoting the response given by Toad-hall, where he demonstrates that he didn't understand what I was writing about, given that he incorrectly did the truth table on the filter I suggested isn't best.
I will bet you $1,000 (or the equivalent in whatever currency you prefer) that the author of the section which includes the "is", "isn't", "contain", and "doesn't contain" filters could add a "doesn't end with" filter in less than 1 hour. I'll double or nothing on that bet. If you send me that code, regardless of programming language or language of comments, I can make the new filter in less than 1 week.
Again, I cannot force anyone to do something, whether it's easy to do or not. I can't make people think about the community as a whole, and not about their dislike of US spam laws (actually lack thereof.) I'm unhappy about them too. That's why I want more filter options so EVERYONE get's what they want, not just some.
You are the MODERATOR. You can delete this thread if you want to. But you cannot make me say the problem is solved simply because you don't like my solution. I can only hope that someone will decide it's a good enough idea to work on.
I will say, if this is an example of the kind of control you exercise over suggested improvement, it's certainly not going to attract me to help out with the programming. But you probably don't need me anyway. I'm a C and machine language programmer, and I doubt any of the Thunderbird filter code is written in them.
But Matt's example does it, in a very much simpler fashion than mine. I'd have to correct it only in that in must be a "matches all" not a "matches any".
I would never set up a filter that deletes all, sight unseen, and that is the prime thrust of your request. Too many things to go wrong.
Well, at least you admit he made a mistake, since Matches Any is clearly wrong.
"I would never set up a filter that deletes all, sight unseen" OK, great, your choice. Are you suggesting that what you think is right is the only way to do things? If so, no problem.
Toad says, "But in all honesty, there is little point in you trying to manually write filters for general Junk." Yep, in this he is absolutely right. That's why I asked for a new filter so a person can CHOOSE what domains he is willing to receive email from and save the endless updating. I assume that anyone would put in an earlier test so that email coming from someone in your address book would pass, even if they are in a domain you aren't interested in hearing from.
Toad adds, "Message Filters is designed to eg: move etc good mail into folders for organising." Like you, he apparently thinks that everyone should do things his way only, and that any other way is wrong.
But, I'm not asking anyone to do things my way. What I'm asking for is one simple filter so that people can have that much more control over their email. There is NOTHING in such a filter that forces people to delete email. Such a filter could move messages to an "Unwanted TLD" folder. It doesn't have to delete them, and that also counts a organizing.
I don't happen to think one size fits all. I wish I could manage to persuade any of you that a simple additional filter is NOT some kind of criminal act, or the death of Thunderbird. You all are spending a lot of time telling me why you don't like it, and none at all trying to understand why someone might like it.
Modified by maegenweird
You can do what you want with the provided tools. No change is needed.
Yes, I can. Again, I thank you for pointing me at FiltaQuilla. If it's too much trouble to extend Thunderbird's filters, I can certainly make do.
And thanks also for not continuing to tell me what a terrible idea it is to ask for a simple filter.
Well, at least you admit he made a mistake, since Matches Any is clearly wrong.
The actual posting was the mistake on my part, I had decided not to post the image, but forgot to remove it.
If I have it located correctly the source is here https://dxr.mozilla.org/comm-central/source/mailnews/base/search/src/nsMsgFilter.cpp
SPAM management is unfortunately entirely a political issue and all the tools we have to manage is are the result of political failure and inaction. I felt, and still feel, I have an obligation to correct statements that are based on incorrect information. I also still feel the blocking mail based on TLD is a terrible idea. But each to their own.
Matt, the only clear error I saw (as opposed to differences of opinions) was when Toad-Hall posted that "Thus filter would not work. You would want the filter to match ANY of those conditions." He didn't go through the truth table of the original posting, or I wouldn't have had to go through it again.
In any case, you were absolutely correct about bug 1381373. It was invalid - not because it was a bad idea, but because it's an enhancement request, not a bug.
I do strongly agree with your final paragraph. It is a political issue, and each person as to make the best choice they can on how to deal with it. It apparently really bothers people that I, a citizen of the US, want a method of dealing with non-US spam, when the US is so spam friendly. I detest spam, but at least I can do something about US spam. I remember when Amazon was called Spamazon, but they learned their lesson. Uunet, once called ScrewYouNet, seems to be catching on. You can't win if you aren't willing to fight, but it's smart to pick battles you have a chance to win.
Zenos, thanks for the red herring regarding FiltaQuila, which hasn't worked since Thunderbird version 45. I had no problem altering the version number, but could not make it be signed, so FiltaQuilla is not an answer. (Unless sticking with Thunderbird 45 is your idea of a good answer.)
In any case, I think I've figured out a work-around.
Filter 1, if From "is in my address book" "Stop filter execution" (That will allow people I know to email me regardless of what their TLD is.) Filter 2, (match any) if From ends with .com or .net or .org "Stop filter execution" (Substitute in domains the user wishes in place of mine.) Filter 3, (Match all messages radio button) "Move Message to" BlockedTLD folder.
It's not as elegant as having the right tool, and requires stepped testing, but it accomplishes the purpose I needed. Hopefully it will reduce the number of Junk messages I have to go through when trying to find a wanted email falsely assigned to Junk.
FiltaQuilla is working fine here in the current Thunderbird. Gosh you do leap to premature conclusions.
Thunderbird does not use signed add-ons, that is a Firefox thing, so me thinks you might have been double clicking, or Firefox was trying to take over and install the file type is recognized. Thunderbird and Firefox both use XPI as the file extension for add-ons. This may have to change in the future as Firefox extension and Thunderbird extension platforms are becoming radically different. With Firefox adopting webextension and dropping old style add-on and Thunderbird keeping the old style and perhaps adopting web extensions as well.
Second here is that enhancement requests go through the Bugzilla bug process exactly the same as so called bugs. My reading of the bug is it is considered invalid because the capability requested already exists. Although perhaps not in the same layout as the requested enhancement foresaw.
Premature? Read the Thunderbird info page ...
Works with Thunderbird 3.0b3pre - 45.*
I kept fiddling with it, and although I clicked on the Thunderbird gear, it does appear that Firefox intercepted it. I closed Firefox and tried again, and I did get it to load. Odd that it claims to only work through Thunderbird 45 while I;'m using 52.4. Doesn't sound like it's as well supported as I would like.
I do not see a "From (RegEx)" selection. There is a "Subject Regex Match" and "Header Regex Match", neither of which can do what I wanted.
I'm accepting ideas, if someone can spare the time.
Modified by maegenweird
My bad. I have two add-ons that add regular expression capabilities, but the way they appear in the filter dialogue doesn't tell you the origin. I guessed at the wrong one.
You could in principle use FiltaQuilla to do this using its "header regex" capability but this is a tad kludgy.
The actual filter I used in the earlier example comes from the Expression Search / GMail UI add-on. It's not an obvious candidate for anyone looking to modify the Message Filter operation because its ostensible function is to enhance the Quick Filter bar. There's no indication that it usefully adds new tools to the filter dialogue. But it is in fact responsible for the "From (RegEx)" that appears in my Message Filters dialogue.
But you don't need "does not end with"; "does not contain" does the job. Hence the bug request being closed off as a "wontfix".
Zenos, if the three of you are the only ones who are fixing things and if you won't fix it, then it won't be fixed. I can't argue you into doing what you don't want to do. No problem.
However, you cannot post an error and expect me not to respond to it. You claimed, ""does not contain" does the job." That's not true. Here's why.
Suppose for whatever reason you want to sort messages on country code. Let's say you want messages from the US Commercial domain .com sorted differently. The problem is, ".com" in contains or doesn't contain, includes forum.community.co.uk (not real AFAIK, but made up to demonstrate the fallacy.)
There's a REASON "ends with" exists in the filters, and it has to do with the hierarchy of internet names. And there's no reason it's complement "doesn't end with" shouldn't exist, except stubbornness and pique. If I had access to the proper file, I have no doubt I could have written it int he time we've wasted arguing about it.
Thank you for the (hopefully helpful) information about the kludgy way "header regex" can be adapted. I'll try to make sense of it and solve my problem that way.
Matt gave you a link to the source, and to a previous bug.
Make your case to the developers and maybe the bug will be reopened. As far as I can see, you and that bug's raiser are the only users who have expressed an interest.
The actual application of the rule would appear to be here:
Yes, you take:
case nsMsgSearchOp::EndsWith: if (StringEndsWith(utf16StrToMatch, needle, nsCaseInsensitiveStringComparator())) result = true; break;
and from it build another element for the switch:
case nsMsgSearchOp::DoesntEndWith: if (!StringEndsWith(utf16StrToMatch, needle, nsCaseInsensitiveStringComparator())) result = true; break;
…then define nsMsgSearchOp::DoesntEndWith and add it to the several tables of inclusions and exclusions where the various search terms are activated or deactivated according to context.
Matt, Toad-Hall and I are just fellow users. We don't write the code, just help users to make the best of what is available.
If I wanted to use this bizarre concept then I can do so right now, today, using the Expression Search facility and a regular expression. Ask for a code change and it will be months away, at best even if someone picks it up to work with. But I have no use for a whitelist of trusted TLDs, and the devs who closed the bug last time apparently couldn't see a need for it either.