Workaround to Dreamhost stopping the support of catchall emails by using Gmail and scripts

About 3 months ago, my hosting provider Dreamhost announced that it was stopping the support of catch-all email addresses claiming that this was to reduce the server load and improve antispam features.
Unfortunately this is a feature that I rely on heavily as I always provide unique email address’s to companies such as [email protected] or [email protected] so that I can filter out emails into different labels, blacklist a spammer and lastly see who has had their website compromised and leaked or sold my email address to someone else. I’ve had at least two email addresses, one of which was a local government agency, start to be used as a spam destination. Unfortunately it is frustrating when I report it to the company and they deny it has happened even though I’ve shown them headers of the email address after the initial contact to let them know they have been compromised. One of the companies I contacted denied they had been breached and I hadn’t even provided them the email address so I knew they hadn’t looked very hard!

Anyway….that is all now over as Dreamhost no longer provide this option.

The solution I am using is as follows:-

  • Use gmail as the primary interface for email (No change since I got gmail back in 2005ish)
  • Use gmail pop3 access to pull in the original catch all email and apply a “ca” label to the email (No change)
  • Use Google Sheets to analyze email and retrieve email address used
  • Create a list of incoming emails and destination email addresses
  • Add emails to Dreamhost control panel

If you wish to continue using the email addresses you can either add them as an alias or set up a forward for the email address. I was previously using the catchall mailbox and had gmail setup to retrieve the email on a regular basis.  The upside of this was that I didn’t have to do anything after setting up the pop account in Google. The downside is that I would have to wait for Google to poll the email account, or go to the accounts tab and pull the mail down manually if I was using a computer. The tablet/mobile access did not provide the ability to “check for mail now”

I had also setup gmail to label all mail retrieved from this catchall account with “ca”. This shows up in the google interface and can also be filtered/searched on.

This meant that I already have email in my gmail account. If you haven’t done that already and wish to use these instructions, then  set up gmail to pull in the email as a pop3 account.

Once you have all email coming into Google I then used a slight modification of Helge Klein’s spreadsheet that lists all email address’s for a single label. Use the provided link to copy the sheet into your own Google Drive account. Select the Tools menu, Edit script and review the script to make sure it’s not malicious.

Around line 45 will be the text

Replace this with

Yes I know the variable name doesn’t make sense, but this is the easiest way to quickly change the code. The neater solution is to then search and replace mailFrom with MailTo

Save the document and then use the text “ca newer:2017/12/31 older:2018/09/01” (without quotes) in B2in the sheets document. Obviously adjust the dates as appropriate.Sheets label to search for emails in 2018

Note that I found that if my date selection was too large the script would time out. Depending on how many emails you have in gmail, I would probably filter out a year at a time. The filter syntax you enter into the B2 field is the syntax you would use to search in gmail.

Once you have your filter, selectHK Scripts/Extract email addresses.

Hopefully all goes well and you now have a list of emails that were sent to your catch all address in a new tab.

The neat thing is that you can run the script multiple times and each search result will generate a new tab. This way you can run this periodically to capture mail received each month. The caveat is that you do need to keep emails with the ca label in your gmail for at least a month!

The screenshot below shows a few results from this search.

Note that I do have a couple of oddities in this result (which is why I included them). There are a couple of email address’s that go to my catchall address but the emails were not forwarded to my original domain. My gravatar email address is actually another gmail address forwarded to a catch all address. As the original TO field is not my domain it doesn’t look quite right. Some other oddities have included google groups targets or some other nefarious/badly formed bcc emails. These should be treated as an exception and handled manually (outside the scope of this document.

 

L:ist of catch all email addresses used

Grab the contents of column B and save to a new tab – this new tab will be used to split the email addresses into a format suitable to paste into Dreamhosts mail control panel.

I then did the following to parse the email address into a useable format. Note that this may not be the most efficient but it gives me the ability to easily verify each stage rather than debug a complicated excel formula.

The screenshot below shows what it should look like when completed. (The two shaded rows are the odd ones that I deal with manually)Spreadsheet of email addresses ready to paste into Dreamhost

Column A should be your original To field. The other cells are filled out as follows. Note you need the starting = sign to signify a calculated field.

D   =split(A1,“@”)

E – not used.

=” yourgooglealias+” & D1 & “@gmail.com”

Note there is a space at the start of G as this is used as the separator in the next step.  Yourgooglealias is the bit before the @ sign in your gmail address

=(substitute(substitute(A1& G1,“<“,“”),“>”,“”))

This replaces any chevrons that may be around the email address and then takes the original email address, adds a space and then adds the new gmail address where the alias now becomes +alias.

 

Finally, copy the contents of column K into dreamhosts control panel. https://panel.dreamhost.com/index.cgi?tree=mail.addresses&current_step=Index&next_step=Bulk should hopefully work but make sure you are pointing to the right domain!

Select Mail, your mail domain, Forwarding Only/Edit All.

Pasting emails into Dreamhost

Hit the save button and you are done.

Your email should be quickly delivered to gmail and still easily filtered out based on the to field.

The downside is that there are still some webforms out there that do not recognise the + character as valid in the email address. Spammers can also easily work out what your main email address is and send directly to that mailbox.

 

Please let me know if you have any questions or comments

 

Bypass two factor authentication to gmail?

Update – Never mind – see bottom of article.
Discovered an interesting flaw in the requirement for two factor authentication with gmail today. I like to use the application on my cell phone to ensure that only I have access to my account – and if somehow a keylogger was in place, my password to gmail is not any use as the 2nd factor authentication would also require access to my cell phone.
However, today I logged into google reader first (which doesn’t support 2nd factor authentication) and used my username and password only. I then clicked the gmail tab at the top of the reader – and hey presto I’m into gmail.
Bottom line – don’t think that just because you have enabled 2nd factor authentication you are safe from keyloggers on a pc or network sniffing/man in the middle attacks. I’ve not reported this to Google yet but it will be interesting to see what they say.
Update After signing out AND restarting firefox I was prompted for the 2nd factor password. Interestingly I wasn’t prompted until I restarted the browser – so as usual – always restart browsers once you’ve finished with them.

Gmail unavailable for a Blackberry 8830 – fixed

A couple of weeks ago I decided to do a wireless upgrade of my Blackberry. It warns you that it will take a couple of hours to do – and also stated that I need to remove some applications as I was running out of memory. This seems to be a common theme with the Blackberry – I don’t know what is so hard about getting Blackberry to store data on SD cards but instead they insist on storing everything on the devices memory – and Windows SmartPhones and Treo’s were no better from my past experience. Anyway – after the upgrade I went to access gmail and initially gmail would just lock up after loading. Thinking a deletion and redownload would fix it, I went ahead and deleted the application. When I went to download it again, the gmail website stated that gmail was not available for the 8830. I knew it was as I’d been using it about 4 hours earlier!  I tried several methods of installing and nothing worked. The Sprint website gave me the ability to download my gmail into the main blackberry email application which then meant I had a mixture of gmail and work mail in one inbox. It was also only downloading some of the emails  – not what I wanted.  I tried searching online and couldn’t find much information and no solutions on this problem.

For some reason I then decided to do another check for wireless update and sure enough, there was another wireless update. Considering the blackberry state couldn’t get much worse I downloaded the new update and voila – gmail was available again. If this happens again, go to Options, Advanced Options, Wireless Update, Check for Updates. I’m now running v4.5.0.186

So far I’ve not seen much difference but yesterday I did see the option to enhance the call quality whilst on a call (but seeing as though I saw this when trying to work out why neither of us on a call could hear each other I don’t think the button really helped!)

The other difference I’ve noticed is that pushing 1 for Voicemail no longer worked and instead the phone asks me to assign a shortcut for the w key. The solution for this is to delete the empty speed dial entry for w and then reboot by pulling the battery out.

More gmail accounts are being hacked.

This afternoon I received spams from fellow colleagues at work from their gmail account. Emails went to both my personal gmail account and to my work accounts. It looks like the emails are in the sent items, which is rather worrying as it means the spammer sent mail from the account rather than forging the headers to make it look like it came from the account. I know for a fact that the password was secure on at least one of the accounts so a weak password is not the culprit. A quick (ironic) google search shows that several people are twittering this in the past couple of hours (mine came in at 3.43pm (and I had another at 7.30pm).
Google’s standard answer is to change your password, which doesn’t really help when there is obviously a back door that is letting people into the account in the first place. The solutions provided are as follows:

If your account has been compromised/hacked/stolen you will need to check at least all of the following things:

Account Security:
Settings -> Accounts and Import -> Google Account Settings -> Change Password [pick a new secure password]
Settings -> Accounts and Import -> Google Account Settings -> Change Password Recovery Options [verify secret question, SMS and secondary e-mail address]

Potential Spam:
Settings -> General -> Signature [make sure nothing as been added]
Settings -> General -> Vacation Responder [make sure it’s disabled and empty]

E-mail Theft

Settings -> Accounts and Import -> Send Mail As [make sure it is using your correct e-mail address]
Settings -> Filters [no filters that forward or delete e-mail]
Settings -> Forwarding and POP/IMAP -> Forwarding [disabled or correct address]
Settings -> Forwarding and POP/IMAP -> POP Download [disabled]
Settings -> Forwarding and POP/IMAP -> IMAP Access [disabled]

Additional Information
Keeping account secure: https://mail.google.com/support/bin/answer.py?hl=en&answer=46526
Protecting your account:  https://mail.google.com/support/bin/answer.py?hl=en&answer=29407
If your account is compromised:  http://mail.google.com/support/bin/answer.py?hl=en&answer=50270

Ciao is also reporting similar issues today.

It would be interesting to see if any of the compromised accounts were on the Google Apps servers as this probably has greater repercussions for Google’s business model as people will trust Google even less. It will certainly raise questions at work on Monday as to whether we would recommend moving some clients to Google Apps. Even if you haven’t been hacked (check your sent items, filters and your frequent contacts for spam messages) I would still highly recommend you change your password NOW and ensure it is a complicated, non-dictionary based one.

gmail antispam broken?

Why is it that gmail seems to be incapable of blocking the emails that start “Hello! I am bored tonight. I am nice girl that would like to chat with you”. It’s not like its a difficult phrase to detect and not something that is likely to be used in everyday emails. I would have thought that marking 99 (or more) of these as spam would have clued the antispam engine to make sure that these emails don’t get delivered to my mailbox.
Update I checked my spam folder this morning and I actually had about 4 or 5 of these emails in the spam folder (amongst the other 350 spams) so maybe it is working (now). Deleting the 20,000 spams that were in the spam folder previously made it hard to work out which were new spams. It would also be nice if marking an email as spam would tag it slightly different so you could tell which was taggedspam and which was detectedspam.

gmail issues?

Anyone having problems accessing their gmail recently? I can log in, but clicking on an email doesn’t do anything (apart from make it unread) OR I get a warning that a script on this page is taking too long to run,do I wish to continue (when using firefox). The problem occurs on two machines that I’ve used and also occurs in internet explorer (so it is not related to any firefox extensions). Also occurs on two different networks (home and office lan). Just wondered if anyone else was having the same issue – I know it’s not site wide as someone else from the office can access their email account ok.
In the meantime, If you’ve sent me an email – you’ll have to wait for a reply…..
Update I worked out what the problem was – yesterday I followed some instructions on enabling Miranda and gmail so that I could talk to msn users on the blackberry. However I think this was bogging down my contact information in gmail as it tried to get the status updates of my buddies in msn. I actually found that this integration didn’t work very well anyway, so I wanted to remove this setup and promptly found my gmail is back to normal.

Lotus Notes Productivity

I’ve been doing a lot of research,reading and studying in improving my productivity and time management recently. This morning I came across the Getting Things Done with Lotus Notes document which really strikes me as an oxymoron. Notes seems to be the most counterproductive piece of software out there and it is not helping me in my productivity. So maybe I’ll get this document to see how it should be done.
One of the things that I picked up from one of the books was to use 1 calendar for everything which is ok if you are single and don’t have a spouse that also needs to see your calendar but they don’t work for your company and have access. As we both have google accounts, I thought that syncing the Notes calendar to google calendar would work as then my wife can see the google calendar and we could both use the calendars to ensure we didn’t doublebook events. I did find the Companion Link for Google Calendar software but it doesn’t work very well.
The synchronization takes forever (when it does work) and I’ve ended up with duplicate entries in google but with different times – 1 hour apart I could understand due to some funky dst issues, but these are a couple of hours apart. Recurring appointments are not supported (although recurring appointments that have been canceled appeared in gmail but active recurring appointments didn’t)
Still, at least I now have a base copy in Google calendar which I will hopefully be able to keep up to date.

secure gmail

I’ve noticed in the past that gmail isn’t secure by default – the initial signon is secured with ssl but then you go back to standard web pages, and therefore open for sniffing. This has prevented me from checking my mail in certain places as I’ve no idea what else is on the network. However, if you go to https://gmail.google.com to sign on, then your ensuing gmail traffic is still encrypted.