Some of you may be aware that I’ve had some hosting issues over the last few weeks. I thought I’d take a little time to explain, for the benefit of clients – so you know what’s been going on and what I’ve done about it; for the benefit of general readers – so you can learn from my experiences; and for my own benefit – to record and reflect on what went on.
My Hosting Background
Just to be clear that I know a bit about what I’m talking about, I used to work for one of the UK’s biggest telecommunications firms in their server hosting department. We hosted both applications and websites for large corporate clients from many different sectors, including retail, construction, travel and the government.
I’m not expert on all of what I’m about to talk about, but I know how a hosting company works, what it has to do, and how it should go about doing it.
The Evolution of Oikos’ Hosting
In the Beginning
It’s worth being honest to start with. Oikos started out as a hobby, and my first proper web hosting package reflected that. I paid about £70 for an account with JustHost. It was a few years ago and can’t remember why I chose them over other providers – I think it may have been one of the companies recommended by WordPress.org at the time and they provided good value for money whilst letting me set up as many subdomains and “add-on” domains as I wanted. Interestingly they’re not listed by WordPress.org any more.
They’ve done me really well and, in particular, reliability and speed has been very good. Their support database isn’t brilliant, but they have usually responded to support requests quickly – if not always with the greatest of technical ability. Oh, and they don’t give you much control over DNS settings for domains that they serve DNS for. I’ve generally been pleased and if you had a site or two to set up and didn’t mind US-based servers then they’re good value.
I then took on a client that had some specific requirements that JustHost couldn’t meet. The client was a church and they wanted to have audio files of talks and sermons available for download from their website. I checked the JustHost terms and conditions and found that downloads and streaming of media files was not allowed.
At this point Oikos was taking off and I was thinking of looking for alternative hosting anyway. So I started looking for hosts that would allow me to have media files for download and pseudo-streaming. It turns out that such hosts are few and far between and often very expensive. However, one client of mine had found a UK-based company who seemed to provide extraordinary value for money and who would allow me to host media files. For now, they shall remain nameless.
Now, I approached a company offering such good value with some caution. I checked out reviews and contacted them with some questions and it all seemed to check out. So I took them up on a trial basis, all looked good, and so I stuck with them.
Now, as someone who cares about hosting, I do use some services to test uptime and availability of some of the websites that I host for myself and for clients. As time went on the frequency of downtime on my hosted sites seemed to increase, and the speed of pages, and the reliability of them too, seemed to decrease. I would sometimes get blank pages or server errors when there was nothing apparently wrong.
Then, back in August, my WordPress Network/Multisite platform hosted with them went down completely. Investigation showed that my files had been moved and when I contacted them they said that I, and many others on their network, had been the subject of an attack, with some bot-nets trying to take advanage of the “TimThumb” vulnerability. I did a full write up of the issue at the time.
I did a search and found that I didn’t have TimThumb installed on any of my sites. The attackers could not have got server access by attacking me, it was just a scan of my hosted files to see if there was a vulnerability. All totally out of my control.
To be fair, they responded quickly to support requests, kept me informed, and reinstated my files quickly. But, ultimately, they left me feeling like it was my fault and that I’d caused their servers to get overloaded, resulting in downtime for lots of their other clients. I was also very annoyed that they hadn’t informed me that there had been a problem and that they’d taken my files offline. I understood their need to protect their servers and networks, but simple customer service etiquette tells you that if you’re disabling someone’s service, you should at least tell them, and explain why.
The support getting my sites back online was so good that I persevered with them for the sites that I already had hosted with them, but decided not to move any more onto that server.
Then, in early November, it happened again. This time I was accused of ‘causing massive issues…a 20 min outage to ALL domains on that server‘ and was told that ‘If this is not the only time this has happened to you, there must be something you are doing that is problematic‘ and ‘A lot of companies would have just terminated your account for this level of damage caused‘.
I was, frankly, pretty insulted by all this. However, first-line support were overridden and I had a late-night call from a network engineer who took over and explained that I seemed to be suffering a DDOS attack from the far east, with masses of requests coming at their server, asking for pages from my domain (oikos.org.uk). This was hard to intercept because they were rapidly changing the source IP’s so simple blocking wouldn’t work – something more sophisticated was needed.
Eventually they claimed that they had shifted my account onto some DDOS protection system and brought it all back up. This mostly happened out of office hours. Again, support felt like a really good experience and they gave me some free credit on my account as a goodwill gesture, plus, I now had proper DDOS protection.
At this point I was pretty fed up, confused about why someone would want to DDOS my site, and more concerned than ever for the future reliability of my sites. I confess to being really very skeptical of this hosting company – my JustHost account was hosting a whole load of WordPress sites and none of them had ever been TimThumb attacked or DDOS’ed – my assumption is that a good hosting company can and does protect themselves against such things so that their customers never need to hear about such issues.
When I had a repeat of the TimThumb attack part-way through my paternity leave, where I was told:
- ‘your domain caused a 1hr outage to over 80,000 shared hosting domains‘
- ‘the version of timthumb in your domain is vulnerable‘ (remember, there are no instances of TimThumb in my domain!)
- ‘there were over 130 [requests] – this caused the shared apache instance to crash taking down all shared customers account‘
I began to think that they were actively trying to get me to move my business somewhere else.
So I started ramping up my on-going alternative hosting investigations, with the aim that Oikos was really taking off as a business, I could put together a business case for some more expensive hosting that would be UK-based and more professional for my clients.
I still needed the multimedia hosting and streaming so I still had a pretty narrow range of options. The two I really looked into were:
- Heart Internet – a UK based company who do a fantastic reseller hosting package. I don’t want to become a hosting company myself, but the ability to brand the control panel and provide clients with access to their own settings would be great. I know several people that have used them for a long time and they have a great reputation. My reservations were two-fold: a) cost – this would be more of an investment than I had really hoped to make and b) their control panel is a custom-built affair, rather than a standard cPanel or Plesk setup and I wondered if their ability to develop features and the UI was less than I’d get from an off-the-shelf control panel.
- Better Web Space – run by local businessman, Twitter followee and fellow geek-dad Keiron Skillett, I would love to have invested my hosting cash in another local business. As I investigated Keiron quickly replied to emails at weekends giving detailed and very helpful answers. Clearly he would help me get what I needed and provide brilliant support. BUT…his reseller hosting was sold in units of 1GB of storage and that was going to work out as very expensive for hosting lots of audio.
Hope and Light at Christmas
Some other events have transpired recently that have made the business case for better hosting much clearer. I’m sad that I can’t invest in Keiron’s business, but there are many good reasons for me to sign up with Heart. And that’s where I now am.
The domain that the hosting company were saying was being attacked (on all three occasions) has been with Heart for two weeks and it’s been fast, stable and I’ve had no one from Heart tell me there’s any problems with it.
I don’t believe the hosting company were lying to me – they had detailed logs showing otherwise! My hunch is that they simply have inadequate defences against modern bot-net attacks.
I’m steadily working to migrate as much of my hosting as possible onto the Heart servers and will continue to monitor the sites for improvement.
I don’t really expect anyone to have read this far. But if you did, thanks for your time and patience.
The summary is, my host had some issues and I’ve now moved somewhere else where:
- those issues don’t seem to exist; and
- the service offered is far superior.
The move is ongoing and has and will continue to be an investment of money and time, but this is the right thing for Oikos to do for its clients.
I’m sorry it took so long to respond to these issues and to make the investment, but I look forward now to being able to provide a better-than-ever hosting service for those who choose to work with me.
I hope you’ve had a very merry Christmas and look forward to working with you in 2012!