Workaround to Dreamhost stopping the support of catchall emails by using Gmail and scripts

About 3 months ago, my hosting provider Dreamhost announced that it was stopping the support of catch-all email addresses claiming that this was to reduce the server load and improve antispam features.
Unfortunately this is a feature that I rely on heavily as I always provide unique email address’s to companies such as [email protected] or [email protected] so that I can filter out emails into different labels, blacklist a spammer and lastly see who has had their website compromised and leaked or sold my email address to someone else. I’ve had at least two email addresses, one of which was a local government agency, start to be used as a spam destination. Unfortunately it is frustrating when I report it to the company and they deny it has happened even though I’ve shown them headers of the email address after the initial contact to let them know they have been compromised. One of the companies I contacted denied they had been breached and I hadn’t even provided them the email address so I knew they hadn’t looked very hard!

Anyway….that is all now over as Dreamhost no longer provide this option.

The solution I am using is as follows:-

  • Use gmail as the primary interface for email (No change since I got gmail back in 2005ish)
  • Use gmail pop3 access to pull in the original catch all email and apply a “ca” label to the email (No change)
  • Use Google Sheets to analyze email and retrieve email address used
  • Create a list of incoming emails and destination email addresses
  • Add emails to Dreamhost control panel

If you wish to continue using the email addresses you can either add them as an alias or set up a forward for the email address. I was previously using the catchall mailbox and had gmail setup to retrieve the email on a regular basis.  The upside of this was that I didn’t have to do anything after setting up the pop account in Google. The downside is that I would have to wait for Google to poll the email account, or go to the accounts tab and pull the mail down manually if I was using a computer. The tablet/mobile access did not provide the ability to “check for mail now”

I had also setup gmail to label all mail retrieved from this catchall account with “ca”. This shows up in the google interface and can also be filtered/searched on.

This meant that I already have email in my gmail account. If you haven’t done that already and wish to use these instructions, then  set up gmail to pull in the email as a pop3 account.

Once you have all email coming into Google I then used a slight modification of Helge Klein’s spreadsheet that lists all email address’s for a single label. Use the provided link to copy the sheet into your own Google Drive account. Select the Tools menu, Edit script and review the script to make sure it’s not malicious.

Around line 45 will be the text

Replace this with

Yes I know the variable name doesn’t make sense, but this is the easiest way to quickly change the code. The neater solution is to then search and replace mailFrom with MailTo

Save the document and then use the text “ca newer:2017/12/31 older:2018/09/01” (without quotes) in B2in the sheets document. Obviously adjust the dates as appropriate.Sheets label to search for emails in 2018

Note that I found that if my date selection was too large the script would time out. Depending on how many emails you have in gmail, I would probably filter out a year at a time. The filter syntax you enter into the B2 field is the syntax you would use to search in gmail.

Once you have your filter, selectHK Scripts/Extract email addresses.

Hopefully all goes well and you now have a list of emails that were sent to your catch all address in a new tab.

The neat thing is that you can run the script multiple times and each search result will generate a new tab. This way you can run this periodically to capture mail received each month. The caveat is that you do need to keep emails with the ca label in your gmail for at least a month!

The screenshot below shows a few results from this search.

Note that I do have a couple of oddities in this result (which is why I included them). There are a couple of email address’s that go to my catchall address but the emails were not forwarded to my original domain. My gravatar email address is actually another gmail address forwarded to a catch all address. As the original TO field is not my domain it doesn’t look quite right. Some other oddities have included google groups targets or some other nefarious/badly formed bcc emails. These should be treated as an exception and handled manually (outside the scope of this document.

 

L:ist of catch all email addresses used

Grab the contents of column B and save to a new tab – this new tab will be used to split the email addresses into a format suitable to paste into Dreamhosts mail control panel.

I then did the following to parse the email address into a useable format. Note that this may not be the most efficient but it gives me the ability to easily verify each stage rather than debug a complicated excel formula.

The screenshot below shows what it should look like when completed. (The two shaded rows are the odd ones that I deal with manually)Spreadsheet of email addresses ready to paste into Dreamhost

Column A should be your original To field. The other cells are filled out as follows. Note you need the starting = sign to signify a calculated field.

D   =split(A1,“@”)

E – not used.

=” yourgooglealias+” & D1 & “@gmail.com”

Note there is a space at the start of G as this is used as the separator in the next step.  Yourgooglealias is the bit before the @ sign in your gmail address

=(substitute(substitute(A1& G1,“<“,“”),“>”,“”))

This replaces any chevrons that may be around the email address and then takes the original email address, adds a space and then adds the new gmail address where the alias now becomes +alias.

 

Finally, copy the contents of column K into dreamhosts control panel. https://panel.dreamhost.com/index.cgi?tree=mail.addresses&current_step=Index&next_step=Bulk should hopefully work but make sure you are pointing to the right domain!

Select Mail, your mail domain, Forwarding Only/Edit All.

Pasting emails into Dreamhost

Hit the save button and you are done.

Your email should be quickly delivered to gmail and still easily filtered out based on the to field.

The downside is that there are still some webforms out there that do not recognise the + character as valid in the email address. Spammers can also easily work out what your main email address is and send directly to that mailbox.

 

Please let me know if you have any questions or comments

 

Bing and Google links to report malicious/fake content in Search engines.

You can report url’s to Bing via http://help.bing.microsoft.com/#apex/18/en-US/10011/0 – It took a while to track that link down – hopefully they won’t change it again unlike the rest of the links I found.

Google’s report site is https://www.google.com/webmasters/tools/spamreportform?hl=en which is a much better url and one that doesn’t look like it will change much.

Fixed: Facebook update constantly downloading and google play not working afterwards.

Last week my mobile phone started to constantly download a Facebook update which was draining the battery due to the constant downloading attempts. Trying to stop the download, I somehow managed to disable the download manager – this did fix the problem but for some reason also stops the Google play store from working. Every time I opened Google Play the application would just disappear from the screen. There was no forced close error message.  This was a bit of a problem as I had removed the Facebook app from my phone trying to stop the download and then was unable to reinstall it as the store would not open.

To fix this issue, Open Settings, Application Manager, scroll to All applications and then scroll down to the bottom of the screen. It is necessary to scroll all the way down as the disabled applications are at the bottom of the alphabetical list. Click on Downloads and then enable the application.

The app store will then start working and hopefully you won’t get the facebook update pushed down to your phone (I didn’t at least)

BPOS active Directory Synchronization does not work on 64bit or domain controllers.

Thinking about BPOS to host the exchange mail for your small office? – I discovered a small gotcha this afternoon but first a couple of thoughts in our recent experiences between Google Apps and Microsoft BPOS.

We’ve had several requests from clients to host their email in the cloud using services like google apps or Microsoft Hosted Exchange. Our first deployments were with google apps due to the lower cost per year, but as we’ve found out the support from Google is severely lacking.  We had a peculiar issue where one users email sent through outlook would get marked as spam all the time, yet if they used the gmail web interface and sent exactly the same content the mail would get sent with no problems.  The service that marks the destination mail as spam is postini – now owned by Google so it should really be a simple matter of turning over the problem to Google to investigate. Unfortunately Google’s response so our request was basically “thanks – if we feel like getting back to you, we might do – in a couple of days”.   As it turned out, the issue was resolved by deleting the google sync profile and recreating it again.

Contrast this to a Microsoft issue which started off as a pre-sales technical call to get a user id setup and ended up with the gtlv owa issue I blogged about earlier. I had several calls from Microsoft within an hour to work on the issue. They worked really hard to ensure my problems were solved and it almost felt like they were harassing me as they kept following up for a status even though I’d told them a couple of times that I was happy for the case to be closed – I even got a call at 5pm on Saturday afternoon from support – something I’ve not experienced (at least from a end user experience!)

So, after singing Microsoft’s praises – the gotcha.  Microsoft’s Hosted Exchange service has a directory sync service that synchronises AD information to the cloud ready for newly created users and distribution lists to appear in the hosted exchange environment. Unfortunately, according to the Directory Synchronization prerequisites the server needs to be running 32bit AND not be a domain controller.   Unfortunately for small companies just starting out from a peer to peer network and getting their first server (but don’t want SBS2008 for some reason), this first server is very likely to be 64bit (crazy not to nowadays) and also a domain controller – possibly/probably even THE domain controller. Installing a 32bit member server is totally out of the question.   It’s not *that* big a deal as all the information can be created online, but it’s twice the amount of data entry along with the possibility of typos but also every AD change of membership and user creation/deletion now needs to be duplicated online.  For large organizations this is not going to be a problem as they’ll likely have extra servers lying around, but for small businesses this is very unlikely. For very large organizations (ie Universities hosting mail through live.edu) it seems that the AD sync program actually requires extra software functionality with the Identity Lifecycle Manager (ILM) package (however I’m not too familiar with these details just yet)

This is the second flaw in Microsoft’s online feature list that extols the virtues of Microsoft vs Google that I’ve found – Active Directory synchronization is not always possible and the other is the benefit of not having to download an application to synchronize data from outlook to the cloud/google – fair enough a client doesn’t have to be downloaded for that application but instead a single signon client needs to be downloaded to prevent the various Microsoft apps asking for the password multiple times. I did think that was the whole point of the “save password” option is for in the outlook and browser applications!

I was also really surprised that the Exchange online is not running Exchange2010 – the Outlook Web App is so much nicer in 2010 compared to 2007 – as I run Outlook 2010 at home and Exchange2010 in the office I’m spoilt (but I still need Office 2010 on the work laptop – thats hopefully coming real soon)

Firefox slow, especially gmail? – remove skype extension.

Ever since I updated the home computer to run the RTM version of Windows7 my gmail and greader experience has been very frustrating in Firefox. GReader just had no idea what data to display in the right hand side of the screen and gmail would just stick in the “please wait”, “loading” or “still working” status after sending an email (but would actually send the email).

Rebooting firefox into safe mode, or running ie8 worked fine so I knew it was not the computer but had to be an extension. Fortunately this was 100% reproducible so I was able to do some diagnostic tests.

I disabled all the likely culprits – flashblock, adblock,greasemonkey, better gmail, web developer, noscript but I still had the same problem. It wasn’t until I disabled the skype extension that the performance of google reader was back to normal and I could now send mail again.  Seeing as though I only really use Skype to call my parents I don’t need the extension so it is now disabled. For some reason the uninstall button is disabled.

I did find a thread on the Google Help forums with the same solution which was nice to know that I wasn’t the only one with the problem and I came up with the same solution.

Hospital tech.

Kristen woke up in the middle of the night with chronic stomach pains and so at 5am I was searching through the United Health Website trying to find the nearest Hospital/ER to where we are staying. The strange thing is that the hospital 11 miles away was not listed. However this doesn’t seem to mean it is not “in-network” as other hospitals were listed ok. Their 24hr careline is only open from 8am to 8pm which is not much use when trying to find out whether your hospital costs are going to be less than $3600 or up to $10,000
So, we ended up going to the hospital anyway as by that point Kristen was in too much pain. The hospital was pretty good with not a lot of waiting to see someone – thankfully Kristen has been ill early in the morning both times we’ve had to go to ER so we’ve not had to wait for too long. Whilst Kristen was being looked at I was reading my book (when the needles came out) and looking at the technology during other waiting periods.
The triage area was computer driven using citrix to connect to their health system. This seemed to be a little slow, but still faster than the triage nurse could type. Extensive use of lookups ensured that data was entered correctly into the database, but meant that “daily” was not recognised and therefore a lookup was provided with options such as WEEKLY or DAILY. You’d have thought that the nurses would have realised to push the caps lock button OR a UCASE function made to ensure that the text typed in was correctly matched. I must admit I was impressed at the medicine lookup – it must be pretty tough keeping all the medicines in the db up to date.
GoogleShirtWhen we were moved into the ER Room itself, I was suprised to see a computer mounted high up on the wall with a flat screen monitor on a swing out arm. I was also surprised to see that they were running Novell for their network (and Mcafee antivirus). I wasn’t sure why the people would need icon’s on the desktop for “my computer” and network neighbourhood – with a locked down desktop I would have thought that the programs needed would already be on the start menu or the desktop.
One of the amusing conversations occurred when one of the nurses took a double take at the t-shirt I was wearing (shown in the picture) and then said that my t-shirt looked familiar. I really would hope that the Google logo is now familiar to most people!

The final analysis is that Kristen likely has a kidney stone on it’s way out, but as the CAT scan equipment is broken we won’t really know for sure (although all the signs are there that this is the case). For an amusing (PG rated) description of Kidney Stones, read this Blogography Kidney Stone metaphor posting.

Ipod reboot

My ipod locked up this morning – it would not respond to any buttons but it would detect the lock being slid across as the lock icon would appear or disappear on the screen. I got to work, mentioned it to my colleague who promptly started to google it. Much to our amusement the first result after typing in reboot and then a space was reboot ipod.
Top result in Google when searching for reboot followed by a space. (by absoblogginlutely)
After pressing the menu and center buttons twice, the ipod unfroze and I was able to listen to my podcasts in the car this morning.

mcpmag blocked.

I subscribe to redmond mag and mcpmag but pages from these sites in google results (see top result here) are being flagged as possibly malicious and clicking through the “go ahead show me the site anyway” then leads to a blocked page (in firefox beta 5) at stopbadware.org stating that the site has been “reported bad”. Unfortunately there is no way to see *why* it’s reported as bad so I can’t make the decision myself and there is no continue button either. Instead my only other choice is to use internet explorer which is more likely to expose me to problems as there isn’t a flashblock/noscript plugin for ie like there is with firefox to provide basic protection straight away.
stopbadware.org sounds like a good idea but could actually expose someone to more dangerous content (and there really should be a “yes I know what I’m doing” button!)