Simwood Status

Wed Jan 27 2021 19:49:06 GMT+0000 (Coordinated Universal Time)

Portal / API working slowly for some functions

Jan 27, 19:49 UTC
Resolved - This incident is now resolved. Apologies again for any interruption accessing information.

Jan 27, 19:05 UTC
Monitoring - A change has been implemented and we believe all services are responding correctly and promptly. Please raise a ticket or comment in our Community Slack channel should you find otherwise. We will continue to monitor the service in the interim. We apologise for any interruption to your service with us.

Jan 27, 17:14 UTC
Identified - Some mitigation has been applied and some functions appear to be working correctly. Those are being monitored but work continues in other areas

Jan 27, 17:03 UTC
Investigating - The API is working slowly for some functions and as the Portal works off of that, it is also affected. We have noted that the download of CDRs, rates and invoices are reported as being affected and are investigating


Mon Feb 08 2021 14:03:40 GMT+0000 (Coordinated Universal Time)

Support Ticket System down

Feb 8, 14:03 UTC
Resolved - The Zendesk service is now operating satisfactorily.

Feb 8, 12:36 UTC
Monitoring - Zendesk have reported this as resolved and we are not experiencing any major issues although some minor internal only issues appear to remain.

Feb 8, 10:53 UTC
Update - Zendesk is still in the middle of a service interruption but is mostly operational for our staff now. We are mitigating the areas where it isn't. We anticipate that ticket and porting functions should work for our customers albeit that Zendesk are still working on their service

Feb 8, 10:10 UTC
Update - Service has partially been restored but it remains slow and sporadic so you may experience some issues or delay. Please continue to mail to [email protected] directly.

Feb 8, 09:33 UTC
Identified - Our support system, supplied by Zendesk, is down as per status.zendesk.com. This will affect our ability to receive tickets from the portal and some porting functions. We will continue to process support tickets mailed into [email protected].


Wed May 26 2021 19:01:35 GMT+0000 (Coordinated Universal Time)

London container maintenance

May 26, 19:01 UTC
Completed - The scheduled maintenance has been completed.

May 26, 19:01 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

May 26, 17:47 UTC
Scheduled - We will be performing maintenance in our Volta site this evening which will involve a restart of the edge proxy container. This will last about a minute and there will be no disruption for those configured in accordance with our interoperability guide.


Wed Sep 08 2021 03:01:56 GMT+0000 (Coordinated Universal Time)

DDoS Preparation Testing

Sep 8, 03:01 UTC
Completed - The scheduled maintenance has been completed.

Sep 8, 02:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 7, 11:45 UTC
Scheduled - At 3am BST / 2am GMT/UTC we will be forcing all wholesale inbound traffic through our contingency Availability Zone to enable customers to test their own preparations. This is further to two rounds of voluntary testing and is described more fully at http://blog.simwood.com/2021/09/voip-ddos-preparations-mandatory-testing/


Mon Sep 13 2021 14:04:54 GMT+0000 (Coordinated Universal Time)

VoIP DDoS Preparations – IMPORTANT customer update

Sep 13, 14:04 UTC
Resolved - We have completed work in connection with this potential threat and interacted and tested with customers and so this notice is marked as resolved. The potential threats for DDoS attacks remain.

Sep 4, 20:56 UTC
Monitoring - We have been monitoring the current DDoS situation and working with industry colleagues. Our adapted plans for handling such an attack against Simwood properties has been published on our blog at

https://blog.simwood.com/2021/09/voip-ddos-preparations-important-customer-update/

Some of those changes require customer action in order to benefit from them. Please digest this blog posting and contact us for any advice.


Tue Nov 02 2021 13:29:24 GMT+0000 (Coordinated Universal Time)

Reports of audio issues

Nov 2, 13:29 UTC
Resolved - We saw network instability in our Telehouse North location which we believe to be due to malformed routes propagated externally from peers, triggering a reload of the BGP process. This in turn caused instability and packet loss for any calls traversing that equipment.

Affected external BGP sessions were disabled restoring stability. The underlying issue will be investigated out of hours, and mitigated over the medium term.

Nov 2, 11:29 UTC
Monitoring - We received 7 examples of affected calls which helped track down a network disturbance leading to audio loss on a very small number of calls.

Once isolated, we have been able to resolve the issue and have received no further reports.

If you do notice any further issues, then please get in touch.

Nov 2, 10:56 UTC
Identified - We believe we have identified the cause of these audio issues and from the examples provided we can see that the issues aren't widespread - it has only affected a very minor volume of calls (less than 10).

The audio issues have been highly intermittent and we are currently working on implementing a fix to ensure these issues no longer persist.

Nov 2, 10:24 UTC
Update - We are continuing to investigate the cause of the audio issues and performing the necessary tests in order to identify where the cause lies. We have received a few examples which is helping in the diagnosis, and when we're able to identify and/or confirm anything further, we'll provide the relevant updates.

Nov 2, 09:53 UTC
Investigating - We have received reports of audio issues and are currently obtaining and investigating examples.


Fri Nov 12 2021 11:47:02 GMT+0000 (Coordinated Universal Time)

Reports of Network issues

Nov 12, 11:47 UTC
Resolved - We saw instability in a single router within our Telehouse East site. This was due to a hardware limit being reached which we believe was the indirect result of overnight maintenance at a peer network. Addressing this caused widespread reconvergence across the network. We are monitoring the status of this router in case of recurrence but it has been stable since 9.34am.

Nov 12, 09:49 UTC
Monitoring - A fix has been implemented and we are monitoring the results.

Nov 12, 09:35 UTC
Update - We are still investigating, and traffic appears to have improved.

Nov 12, 09:10 UTC
Update - We are continuing to investigate this issue.

Nov 12, 09:01 UTC
Investigating - We are currently investigating this issue.


Mon Nov 29 2021 16:39:14 GMT+0000 (Coordinated Universal Time)

Intermittent API/portal timeout

Nov 29, 16:39 UTC
Resolved - Reports are processed sequentially and a single customer was effectively causing a Denial of Service with thousands of requests for a heavy report, which was in turn delaying others beyond the timeout the API has. The API, portal and the platform in general were otherwise fine and this functioned as expected.

Nov 29, 13:23 UTC
Monitoring - A fix has been implemented and we are monitoring the results.

Nov 29, 13:04 UTC
Identified - The issue has been identified and a fix is being implemented.

Nov 29, 12:54 UTC
Investigating - We are currently investigating this issue.


Tue Dec 14 2021 21:08:40 GMT+0000 (Coordinated Universal Time)

Intermittent issues with portal / API - degraded performance

Dec 14, 21:08 UTC
Resolved - This incident has been resolved.

Dec 14, 16:25 UTC
Monitoring - We are continuing to investigate the cause of the issue and identifying a robust resolution - the portal is back to being accessible.

Dec 14, 15:54 UTC
Identified - The issue has been identified and a fix is being implemented.

Dec 14, 15:28 UTC
Investigating - We have received reports of intermittent issues that are impacting the API / Portal. This is currently being investigated and we'll update accordingly.


Fri Dec 31 2021 12:25:07 GMT+0000 (Coordinated Universal Time)

Portal issues

Dec 31, 12:25 UTC
Resolved - The Simwood Portal has been stable for some time and the issue is now resolved.

Dec 31, 11:00 UTC
Monitoring - We have implemented a fix and the Simwood Portal is now accessible again. We will monitor the situation for the next hour or so.

Dec 31, 10:41 UTC
Investigating - We are investigating an issue with the Simwood Portal.