Dear Valued Customers,
We extend our sincerest apologies for the disruptions experienced by some of our customers on January 7th and 8th. Approximately one week ago, our upstream provider informed us of scheduled maintenance, commencing on January 7th at 9:00 AM, with the intention of upgrading the network’s backend infrastructure. Regrettably, this notification was overlooked in the middle of our focus on the installation of a new node and the seamless transfer of our billing software.
Incident Details
The maintenance, originally anticipated to result in minor disruptions, proceeded smoothly until unexpected switch movements caused a 20-minute service outage. We deeply regret any inconvenience this may have caused and acknowledge the impact on your operations. Post-maintenance, our monitoring detected intermittent issues throughout the night; however, we did not communicate this to you or our upstream provider because they fell within the maintenance window and did not affect any customers.
On January 8th, at approximately 4:10 PM CST, an outage was identified on our new VPS node, affecting a subset of customers, as the node had not yet been fully operational for the general public. Following a thorough investigation, we identified an issue with our upstream provider. A swift response was initiated, and within 10 minutes, we were informed that some network settings were misconfigured, resulting in the temporary unavailability of the new node. Unfortunately, this then triggered a subsequent 5-minute outage on our main nodes due to miscommunication, leading to the deactivation of critical IPs for our core machines. We sincerely apologize for any inconvenience caused by these events.
Investigation and Resolution
In response to these incidents, we are implementing the following measures to enhance our services and prevent similar occurrences:
- Improved Communication Protocols: We are enhancing our communication channels with our providers to ensure timely and accurate information regarding scheduled maintenance and potential disruptions.
- Proactive Customer Communication: To better serve you, our valued customers, we are implementing more proactive communication strategies. This includes real-time updates during maintenance windows and immediate notification of any issues that may impact your services. More information will be available soon on how you can enroll.
- Preventive Measures: We are working diligently to implement additional safeguards and preventive measures to minimize the risk of similar incidents in the future.
Your understanding and patience during this time are immensely appreciated. We are committed to learning from these experiences and further improving our services to provide you with the reliable and efficient solutions you deserve.
Best Regards,
Ethan
CEO | Mellow Services