Jun 7, 12:41 MDT
Completed - Scheduled Maintenance has been completed. Apologies for the extended window and disruption. All Systems Operational.
For questions or concerns regarding this maintenance please contact Splunk On-Call (formerly VictorOps) Support. Please review the linked Knowledge Base article: How to Contact Splunk On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
Jun 7, 12:38 MDT
Verifying - Verification is currently underway for the maintenance items.
Jun 7, 12:08 MDT
Update - UPDATE: Platform has returned to operational status.
We will continue testing features and components.
Jun 7, 11:57 MDT
Update - UPDATE: Please reference the On-Call Mobile App for access to alerting and notifications.
The Web Client is currently impacted.
Jun 7, 11:53 MDT
Update - Update: The On-Call platform is still in a degraded state. Extending maintenance window to 2:00 PM Mountain. Thank you for your patience while we work these issues to resolution.
Jun 7, 11:34 MDT
Update - Update: Deploy complications are impacting our downtime. Please bear with us while we work towards resolution.
Jun 7, 11:15 MDT
Update - Scheduled Maintenance is still in progress. Entering a bit of platform degradation and approximately 5 mins of pause. Please bear with us and thanks for your patience.
Jun 7, 10:00 MDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jun 5, 12:05 MDT
Scheduled - **UPDATED / RESCHEDULED MAINTENANCE WINDOW**
Please note, the Splunk On-Call Platform will be undergoing scheduled maintenance for approximately two hours on June 7th from 10:00 AM to 12:00 PM (Mountain time).
*We expect zero interruptions to alert ingestion. No dropped alerts.
*We expect some platform disruption for up to an hour.
*We expect delays in paging/notifications of up to ten minutes.
*We expect that the Public API, SMS replies for Ack and Resolve, the Web Client, and Mobile Clients will be unavailable for up to ten minutes.
To contact Splunk On-Call (formerly VictorOps) Support during this scheduled maintenance window, please review the linked Knowledge Base article: How to Contact Splunk On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
Thank you for your patience while we proceed with maintenance.
Sep 14, 15:04 MDT
Resolved - Host Update - "The original issue has been successfully resolved." Accordingly, we will be resolving this incident.
Please reach out Splunk On-Call Support with any questions or concerns regarding this incident!
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
Sep 14, 13:29 MDT
Update - Host Update - "Site availability is continuing to improve. We are actively monitoring and continue to see a full recovery in progress.
Sep 14, 2023 - 12:12 PDT" ... Splunk On-Call will resolve this Statuspage incident when our host gives the all clear. For all intents and purposes, our Knowledge Base is back to fully operational.
Sep 14, 12:54 MDT
Update - We are continuing to monitor for any further issues.
Sep 14, 12:52 MDT
Monitoring - A fix has been implemented by our host and the Knowledge Base (https://help.victorops.com/) is once again available. If you are still experiencing loading issues please clear your cache and refresh the page. Thank you!
Sep 14, 12:42 MDT
Identified - The issue has been identified by our KB host and they are working as quickly as possible towards resolution. Thanks for your patience while we/they restore functionality.
Sep 14, 11:53 MDT
Investigating - We are currently investigating an issue with our Knowledge Base hosting service. The ability to contact Support is unaffected and fully operational.
We will provide updates as soon as they're available.
Sep 29, 10:30 MDT
Completed - Maintenance completed. Voice notification service to China (+86) is deprecated.
Affected end-users are encouraged to configure personal paging policies to diversify their Splunk On-Call notification methods (i.e., utilize SMS, Push, and/or Email notification methods).
Please note that the Splunk On-Call product continues to be supported globally and all functionality will be available to customers with responders outside of the +86 country code (China).
We sincerely apologize for any inconvenience this may cause
Sep 29, 10:26 MDT
Verifying - We are verifying maintenance on deprecating Splunk On-Call voice call notification +86 country code (China).
Sep 29, 10:06 MDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 8, 16:39 MDT
Scheduled - As you may know, we are all experiencing increased restrictions and regulations between the U.S. and China. As a company, Splunk has made the decision to no longer provide Voice Call notifications to responders in +86 country code (China) effective on September 29th, 2023.
In response, affected end-users are encouraged to configure personal paging policies to diversify their Splunk On-Call notification methods (i.e., utilize SMS, Push, and/or Email notification methods).
Please note that the Splunk On-Call product continues to be supported globally and all functionality will be available to customers with responders outside of the +86 country code (China).
Again, please be advised that we will be deprecating Splunk On-Call voice call notification service for +86 country code (China) effective on September 29th, 2023.
We sincerely apologize for any inconvenience this may cause.
Dec 18, 17:19 MST
Resolved - This incident has been resolved. All On-Call systems are operational. Thank you for your patience while we completed our testing.
For questions regarding this incident, please contact Splunk On-Call Support:
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
Dec 16, 12:08 MST
Update - We are still monitoring the status of this incident. Testing continues. All systems are operational.
Dec 16, 02:13 MST
Update - We continue to monitor the issue but have moved our iOS SSO Mobile login back to operational status. Thanks again for your patience while we continue to work the root cause of this issue to ground.
Dec 16, 02:10 MST
Update - We are continuing to monitor for any further issues.
Dec 16, 01:45 MST
Monitoring - We've identified the issue and have implemented a fix. Moving this incident to a monitoring status. iOS SSO mobile login should now be available. Updates to follow.
Apologies again for any inconvenience.
Dec 16, 01:02 MST
Investigating - We are currently investigating an issue with iOS mobile SSO log-in. For immediate access to Splunk On-Call please use the Web Client until we can identify the root cause of this issue. Again, this is only affecting iOS mobile. Thanks your your patience while we troubleshoot this issue.
Jan 24, 11:47 MST
Resolved - This incident has been resolved. All systems are operational. If you're still experiencing an issue with loading large lists of on-call users in the Incident tab under "Team: All Teams" please refresh the page and give it a few seconds. If the problem continues, please reach out to On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
Jan 24, 10:57 MST
Identified - The issue has been identified and a fix is being implemented. Please stay tuned for further updates.
Jan 24, 10:31 MST
Update - We are continuing to investigate this issue.
Jan 24, 10:30 MST
Investigating - Splunk On-Call is currently experiencing an isolated disruption to the Incident Page for large instances. We are actively working to resolve this issue as quickly as possible.
Please use the Subscribe to Updates option or follow @VOSupport on Twitter for updates. If you have any immediate questions or concerns, you may contact our support team as outlined here: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
Thank you for your patience while we work this issue to ground.
Feb 28, 16:46 MST
Resolved - This incident has been resolved. Reporting is fully operational. We apologize again for any inconvenience.
Please direct any questions regarding this incident to On-Call Support:
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
Feb 28, 14:08 MST
Update - UPDATE: Reporting back to Operational status.
We continue to monitor our Reporting data and will be keeping this incident open for the time being.
Thanks again for your patience while we work to completely resolve this incident.
Feb 28, 12:24 MST
Monitoring - UPDATE: Remediation complete; watching the latency decrease.
Thanks again for your patience while we work this issue to resolution.
Feb 28, 09:28 MST
Update - Apologies for the Reporting inconvenience!
Current Status:
- Data is up to date as of 6:00 AM (UTC)
- No reporting data lost
- Data replication is in progress and will be 100% available once complete
Please direct any questions to Splunk On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
Feb 28, 01:58 MST
Identified - We have identified the issue and are working towards a resolution.
Feb 28, 01:57 MST
Investigating - We are currently experiencing lag in our report tooling. New Incidents will not show-up in any reports until the incident is resolved.
Mar 1, 12:32 MST
Resolved - The issue has been identified and resolved! Apologies for any inconvenience!
Please direct any questions regarding this incident to On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
Mar 1, 11:52 MST
Update - We are continuing to investigate this issue.
Mar 1, 11:51 MST
Investigating - We are currently investigating a problem regarding the performance of our MS Team integration. Please bear with us while we troubleshoot this issue — updates to follow.
Mar 21, 23:30 MDT
Completed - Scheduled maintenance has been completed, reporting tools are available again and all systems are fully operational.
Mar 21, 21:15 MDT
Update - Scheduled Maintenance is currently ongoing. During this time all reports will be inaccessible. This includes creation of a Post-Incident Review, Response Metrics, On-Call Review, and Incident Frequency reports. We thank you for your patience as we perform this maintenance and will let you know when these reports are accessible again.
Mar 21, 21:00 MDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 18, 11:33 MDT
Update - UPDATE: Estimated Reporting unavailability time during scheduled maintenance has been extended to FOUR HOURS.
Mar 11, 09:42 MDT
Update - We will be undergoing scheduled maintenance during this time.
Mar 11, 09:36 MDT
Scheduled - Please pardon our dust! Splunk On-Call Reporting will be unavailable during time of maintenance: Estimated Reporting unavailability time: TWO HOURS.
After initial maintenance, we expect delays in Reporting data. Estimated Reporting data delay: FOUR to SIX hours. No data loss is expected during or after maintenance, only delays to Incident Frequency Reporting (IFR), Response Metrics (MTTA/R), On-Call Reporting (OCR), and Post Incident Review (PIR).
Please reach out to Splunk On-Call Support with any questions or concerns!
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
Apr 13, 01:45 MDT
Completed - The scheduled maintenance has been completed.
Apr 13, 01:00 MDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 29, 15:02 MDT
Scheduled - We will be conducting maintenance on our Splunk On-Call Knowledge Base (https://help.victorops.com/) from 1:00 - 1:45 AM on Saturday, April 13 (Mountain Time). The Splunk On-Call web UI, mobile app, APIs, and other functionality will be unaffected. Apologies for any potential disruption.
The maintenance is expected to last for 30 minutes.
Please contact Splunk On-Call support at (1.855) 775.8657 with any questions or concerns.
May 6, 07:33 MDT
Resolved - Our provider is no longer experiencing SMS delivery delays when sending messages to the Proximus Mobility network in the US (AMER) over a subset of shortcodes. This incident has been resolved. All systems are operational.
If you're still experiencing issues with SMS please reach out to On-Call Support:
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
We apologize for this inconvenience.
May 5, 12:02 MDT
Update - The Splunk On-Call notification provider continues to experience SMS delivery delays when sending messages to the Proximus Mobility network in the US (AMER) over a subset of shortcodes. Our provider is working with the carrier partner to resolve the issue. Splunk On-Call will continue to monitor and provide updates as soon as more information becomes available.
As mentioned, please diversify your paging policies to include WhatsApp, Voice (phone call), and Push notifications to avoid single points of notification failure.
Documentation is found here:
Paging Policies: https://help.victorops.com/knowledge-base/paging-policy-setup/
Notification Types: https://help.victorops.com/knowledge-base/notification-types/
We apologize for this inconvenience. Please reach out to On-Call Support with any questions!
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
May 4, 18:59 MDT
Update - Update: This issue affects shortcode SMS to the domestic US (AMER). Our provider continues to work with carrier partners to resolve this issue.
You may track the progress toward the resolution of this issue here:
https://status.twilio.com/incidents/p6jvzrw06911
May 4, 15:59 MDT
Monitoring - The issue has been identified. Our provider is working with carrier partners to resolve the issue.
In the meantime, please diversify your paging policies to include WhatsApp, Voice (phone call), and Push notifications to avoid a single point of failure.
Documentation is found here:
Paging Policies: https://help.victorops.com/knowledge-base/paging-policy-setup/
Notification Types: https://help.victorops.com/knowledge-base/notification-types/
We apologize for this inconvenience. Please reach out to On-Call Support with any questions!
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/