This weekend there were two instances where ChurchSuite was either suffering from a full outage (Saturday from approx. 9:00am to 9:45am) or where things were running slower than normal (Sunday from approx. 11:20am to 11:30am).
Our mission is to build software that supports your church and extends God's Kingdom. Anything that interferes with this core mission of supporting our customers is something we take incredibly seriously.
If you were among the affected customers this weekend, we would like to start by offering our sincere apologies to you. We appreciate that for some of you this will have interfered with the smooth running of your ministries. We are sorry and we want to be transparent and clear about the problem and reassure you of the steps we are taking to move forward and prevent this from happening in the future.
What happened?
We became aware of an outage to our service on Saturday morning. Nobody could access ChurchSuite and we began investigating straight away. We were able to restore service within the hour and updated our status page accordingly as we went.
On Sunday morning we became aware that service had slowed down and degraded for many users. We were able to fully restore service within half an hour. Our system health monitoring systems correctly alerted our developers, who quickly resolved the issue, but not before some of your users were affected.
We believe in open and transparent communication with our customers. To that end, on both days, we updated our status page as the situation was developing. In addition, we replied as promptly as we were able to any customers who had emailed through or contacted us via social media.
Moving forward...
After we had made sure that ChurchSuite was running as normal, we started digging deeper in order to determine the root cause. We believe the underlying issue has now been fixed but we will continue to keep a close eye over the next few weeks to ensure things continue running smoothly.
In both instances, the server load we were experiencing hit a level that was too high for the capacity we had put in place. While ChurchSuite has been growing quickly, we have also been working hard to scale our infrastructure - and despite this weekend, the 99.95% uptime record on our status page is testimony to that effort. The system is designed to auto-scale our servers to meet rising server demand and in order to cater for future spikes in traffic, and off the back of this weekend, we have further increased our server capacity.
This weekend's incident is an opportunity for us to review, learn and further improve our internal monitoring and response processes. We remain dedicated to building world-leading software to support your ministry. We are sincerely sorry that for some customers this past weekend we failed to meet the high standards we set for ourselves. If you have any questions relating to this issue please do get in touch.
On behalf of the ChurchSuite team,
