Workaround to Dreamhost stopping the support of catchall emails by using Gmail and scripts

About 3 months ago, my hosting provider Dreamhost announced that it was stopping the support of catch-all email addresses claiming that this was to reduce the server load and improve antispam features.
Unfortunately this is a feature that I rely on heavily as I always provide unique email address’s to companies such as [email protected] or [email protected] so that I can filter out emails into different labels, blacklist a spammer and lastly see who has had their website compromised and leaked or sold my email address to someone else. I’ve had at least two email addresses, one of which was a local government agency, start to be used as a spam destination. Unfortunately it is frustrating when I report it to the company and they deny it has happened even though I’ve shown them headers of the email address after the initial contact to let them know they have been compromised. One of the companies I contacted denied they had been breached and I hadn’t even provided them the email address so I knew they hadn’t looked very hard!

Anyway….that is all now over as Dreamhost no longer provide this option.

The solution I am using is as follows:-

  • Use gmail as the primary interface for email (No change since I got gmail back in 2005ish)
  • Use gmail pop3 access to pull in the original catch all email and apply a “ca” label to the email (No change)
  • Use Google Sheets to analyze email and retrieve email address used
  • Create a list of incoming emails and destination email addresses
  • Add emails to Dreamhost control panel

If you wish to continue using the email addresses you can either add them as an alias or set up a forward for the email address. I was previously using the catchall mailbox and had gmail setup to retrieve the email on a regular basis.  The upside of this was that I didn’t have to do anything after setting up the pop account in Google. The downside is that I would have to wait for Google to poll the email account, or go to the accounts tab and pull the mail down manually if I was using a computer. The tablet/mobile access did not provide the ability to “check for mail now”

I had also setup gmail to label all mail retrieved from this catchall account with “ca”. This shows up in the google interface and can also be filtered/searched on.

This meant that I already have email in my gmail account. If you haven’t done that already and wish to use these instructions, then  set up gmail to pull in the email as a pop3 account.

Once you have all email coming into Google I then used a slight modification of Helge Klein’s spreadsheet that lists all email address’s for a single label. Use the provided link to copy the sheet into your own Google Drive account. Select the Tools menu, Edit script and review the script to make sure it’s not malicious.

Around line 45 will be the text

Replace this with

Yes I know the variable name doesn’t make sense, but this is the easiest way to quickly change the code. The neater solution is to then search and replace mailFrom with MailTo

Save the document and then use the text “ca newer:2017/12/31 older:2018/09/01” (without quotes) in B2in the sheets document. Obviously adjust the dates as appropriate.Sheets label to search for emails in 2018

Note that I found that if my date selection was too large the script would time out. Depending on how many emails you have in gmail, I would probably filter out a year at a time. The filter syntax you enter into the B2 field is the syntax you would use to search in gmail.

Once you have your filter, selectHK Scripts/Extract email addresses.

Hopefully all goes well and you now have a list of emails that were sent to your catch all address in a new tab.

The neat thing is that you can run the script multiple times and each search result will generate a new tab. This way you can run this periodically to capture mail received each month. The caveat is that you do need to keep emails with the ca label in your gmail for at least a month!

The screenshot below shows a few results from this search.

Note that I do have a couple of oddities in this result (which is why I included them). There are a couple of email address’s that go to my catchall address but the emails were not forwarded to my original domain. My gravatar email address is actually another gmail address forwarded to a catch all address. As the original TO field is not my domain it doesn’t look quite right. Some other oddities have included google groups targets or some other nefarious/badly formed bcc emails. These should be treated as an exception and handled manually (outside the scope of this document.

 

L:ist of catch all email addresses used

Grab the contents of column B and save to a new tab – this new tab will be used to split the email addresses into a format suitable to paste into Dreamhosts mail control panel.

I then did the following to parse the email address into a useable format. Note that this may not be the most efficient but it gives me the ability to easily verify each stage rather than debug a complicated excel formula.

The screenshot below shows what it should look like when completed. (The two shaded rows are the odd ones that I deal with manually)Spreadsheet of email addresses ready to paste into Dreamhost

Column A should be your original To field. The other cells are filled out as follows. Note you need the starting = sign to signify a calculated field.

D   =split(A1,“@”)

E – not used.

=” yourgooglealias+” & D1 & “@gmail.com”

Note there is a space at the start of G as this is used as the separator in the next step.  Yourgooglealias is the bit before the @ sign in your gmail address

=(substitute(substitute(A1& G1,“<“,“”),“>”,“”))

This replaces any chevrons that may be around the email address and then takes the original email address, adds a space and then adds the new gmail address where the alias now becomes +alias.

 

Finally, copy the contents of column K into dreamhosts control panel. https://panel.dreamhost.com/index.cgi?tree=mail.addresses&current_step=Index&next_step=Bulk should hopefully work but make sure you are pointing to the right domain!

Select Mail, your mail domain, Forwarding Only/Edit All.

Pasting emails into Dreamhost

Hit the save button and you are done.

Your email should be quickly delivered to gmail and still easily filtered out based on the to field.

The downside is that there are still some webforms out there that do not recognise the + character as valid in the email address. Spammers can also easily work out what your main email address is and send directly to that mailbox.

 

Please let me know if you have any questions or comments

 

New Greasemonkey Script – Add Google Reader to Google navigation bar

I am a heavy user of Google Reader and find it annoying that it is one of the sites that does not appear on the top of the Google Navigation Bar when you are on other Google sites such as gmail, Google Plus etc. Instead you have to click the More button, then select Reader from the drop down menu.
I managed to hack together the Greasemonkey script AddGoogleReader that adds the link to Reader next to the Gmail link. What I’ve not managed to do is get the Reader to appear in line with the other buttons – I’ll play around with that later, but in the meantime, if you use chrome or have Greasemonkey installed, then let me know what you think. Follow up posts can be found in the addgooglereader tag

Bypass two factor authentication to gmail?

Update – Never mind – see bottom of article.
Discovered an interesting flaw in the requirement for two factor authentication with gmail today. I like to use the application on my cell phone to ensure that only I have access to my account – and if somehow a keylogger was in place, my password to gmail is not any use as the 2nd factor authentication would also require access to my cell phone.
However, today I logged into google reader first (which doesn’t support 2nd factor authentication) and used my username and password only. I then clicked the gmail tab at the top of the reader – and hey presto I’m into gmail.
Bottom line – don’t think that just because you have enabled 2nd factor authentication you are safe from keyloggers on a pc or network sniffing/man in the middle attacks. I’ve not reported this to Google yet but it will be interesting to see what they say.
Update After signing out AND restarting firefox I was prompted for the 2nd factor password. Interestingly I wasn’t prompted until I restarted the browser – so as usual – always restart browsers once you’ve finished with them.

Gmail unavailable for a Blackberry 8830 – fixed

A couple of weeks ago I decided to do a wireless upgrade of my Blackberry. It warns you that it will take a couple of hours to do – and also stated that I need to remove some applications as I was running out of memory. This seems to be a common theme with the Blackberry – I don’t know what is so hard about getting Blackberry to store data on SD cards but instead they insist on storing everything on the devices memory – and Windows SmartPhones and Treo’s were no better from my past experience. Anyway – after the upgrade I went to access gmail and initially gmail would just lock up after loading. Thinking a deletion and redownload would fix it, I went ahead and deleted the application. When I went to download it again, the gmail website stated that gmail was not available for the 8830. I knew it was as I’d been using it about 4 hours earlier!  I tried several methods of installing and nothing worked. The Sprint website gave me the ability to download my gmail into the main blackberry email application which then meant I had a mixture of gmail and work mail in one inbox. It was also only downloading some of the emails  – not what I wanted.  I tried searching online and couldn’t find much information and no solutions on this problem.

For some reason I then decided to do another check for wireless update and sure enough, there was another wireless update. Considering the blackberry state couldn’t get much worse I downloaded the new update and voila – gmail was available again. If this happens again, go to Options, Advanced Options, Wireless Update, Check for Updates. I’m now running v4.5.0.186

So far I’ve not seen much difference but yesterday I did see the option to enhance the call quality whilst on a call (but seeing as though I saw this when trying to work out why neither of us on a call could hear each other I don’t think the button really helped!)

The other difference I’ve noticed is that pushing 1 for Voicemail no longer worked and instead the phone asks me to assign a shortcut for the w key. The solution for this is to delete the empty speed dial entry for w and then reboot by pulling the battery out.

More gmail accounts are being hacked.

This afternoon I received spams from fellow colleagues at work from their gmail account. Emails went to both my personal gmail account and to my work accounts. It looks like the emails are in the sent items, which is rather worrying as it means the spammer sent mail from the account rather than forging the headers to make it look like it came from the account. I know for a fact that the password was secure on at least one of the accounts so a weak password is not the culprit. A quick (ironic) google search shows that several people are twittering this in the past couple of hours (mine came in at 3.43pm (and I had another at 7.30pm).
Google’s standard answer is to change your password, which doesn’t really help when there is obviously a back door that is letting people into the account in the first place. The solutions provided are as follows:

If your account has been compromised/hacked/stolen you will need to check at least all of the following things:

Account Security:
Settings -> Accounts and Import -> Google Account Settings -> Change Password [pick a new secure password]
Settings -> Accounts and Import -> Google Account Settings -> Change Password Recovery Options [verify secret question, SMS and secondary e-mail address]

Potential Spam:
Settings -> General -> Signature [make sure nothing as been added]
Settings -> General -> Vacation Responder [make sure it’s disabled and empty]

E-mail Theft

Settings -> Accounts and Import -> Send Mail As [make sure it is using your correct e-mail address]
Settings -> Filters [no filters that forward or delete e-mail]
Settings -> Forwarding and POP/IMAP -> Forwarding [disabled or correct address]
Settings -> Forwarding and POP/IMAP -> POP Download [disabled]
Settings -> Forwarding and POP/IMAP -> IMAP Access [disabled]

Additional Information
Keeping account secure: https://mail.google.com/support/bin/answer.py?hl=en&answer=46526
Protecting your account:  https://mail.google.com/support/bin/answer.py?hl=en&answer=29407
If your account is compromised:  http://mail.google.com/support/bin/answer.py?hl=en&answer=50270

Ciao is also reporting similar issues today.

It would be interesting to see if any of the compromised accounts were on the Google Apps servers as this probably has greater repercussions for Google’s business model as people will trust Google even less. It will certainly raise questions at work on Monday as to whether we would recommend moving some clients to Google Apps. Even if you haven’t been hacked (check your sent items, filters and your frequent contacts for spam messages) I would still highly recommend you change your password NOW and ensure it is a complicated, non-dictionary based one.

Firefox slow, especially gmail? – remove skype extension.

Ever since I updated the home computer to run the RTM version of Windows7 my gmail and greader experience has been very frustrating in Firefox. GReader just had no idea what data to display in the right hand side of the screen and gmail would just stick in the “please wait”, “loading” or “still working” status after sending an email (but would actually send the email).

Rebooting firefox into safe mode, or running ie8 worked fine so I knew it was not the computer but had to be an extension. Fortunately this was 100% reproducible so I was able to do some diagnostic tests.

I disabled all the likely culprits – flashblock, adblock,greasemonkey, better gmail, web developer, noscript but I still had the same problem. It wasn’t until I disabled the skype extension that the performance of google reader was back to normal and I could now send mail again.  Seeing as though I only really use Skype to call my parents I don’t need the extension so it is now disabled. For some reason the uninstall button is disabled.

I did find a thread on the Google Help forums with the same solution which was nice to know that I wasn’t the only one with the problem and I came up with the same solution.