Technical Incident - Mail Delays, v2/v3 API Calls
Incident Report for Robly
Resolved
Our team has monitored our system and confirmed that the issue has been resolved. Delayed mail has been dequeued and all impacted services are up and running again. We sincerely apologize for the inconvenience this has caused and thank you for your patience.
Posted Jul 29, 2020 - 00:10 EDT
Update
We are continuing to monitor system performances. We will provide another update in 30 minutes or sooner. We sincerely apologize for the inconvenience this has caused and thank you for your patience.
Posted Jul 28, 2020 - 17:16 EDT
Monitoring
We have recovered the remaining impacted services and are currently monitoring system performances. We will provide another update in 30 minutes or sooner. We sincerely apologize for the inconvenience this has caused and thank you for your patience.
Posted Jul 28, 2020 - 15:44 EDT
Update
We are confident we have identified the source of the issue and are continuing work to recover the remaining impacted services. We will provide another update in 30 minutes or sooner. We sincerely apologize for the inconvenience this has caused and thank you for your patience.
Posted Jul 28, 2020 - 15:14 EDT
Identified
We believe we have identified the source of the issue which has been affecting our services. Our team has tested the remediation steps and are now working toward recovering all remaining impacted services. We will provide another update in 30 minutes or sooner as they become available. We sincerely apologize for the inconvenience this has caused and thank you for your patience.
Posted Jul 28, 2020 - 14:52 EDT
Update
Due to an issue in one of our data centers this morning, mail is still stuck in delivery. We are working to get those delayed messages delivered as quickly as possible. Customers may still see errors with non-mail send related API calls. We sincerely apologize for the inconvenience and thank you for your patience. We will provide another update in 30 minutes or sooner as we get updates. Please reach out to our Support team at support@robly.com with any questions.
Posted Jul 28, 2020 - 14:28 EDT
Update
We are still seeing mail and stat delays. Mail is being accepted but could be stuck in queue. Users may be experiencing a higher volume of v2/v3 API call errors, especially 5xx errors around 9:45 AM MT. We sincerely apologize for the inconvenience and thank you for your patience. Please reach out to our Support team at support@robly.com with any questions.
Posted Jul 28, 2020 - 12:53 EDT
Update
We are still investigating issues around mail and stat delays, and issues with v2/v3 API calls. We will continue to provide updates as they become available. We thank you for your patience and sincerely apologize for any trouble this is causing. Please reach out to our Support team at support@robly.com with any questions.
Posted Jul 28, 2020 - 12:27 EDT
Investigating
Starting around 6AM MT, users may be experiencing issues regarding mail delays and delays surrounding v2/v3 API calls. We are currently investigating and will provide updates as they become available. We sincerely apologize for the any inconvenience this may cause. Please reach out to our Support team at support@robly.com with any questions.
Posted Jul 28, 2020 - 11:19 EDT
This incident affected: Email Delivery and API.