Aug 31, 13:06 UTC
Resolved - This incident has been resolved.
Aug 31, 10:46 UTC
Monitoring - A fix has been implemented and we are monitoring the results.
Aug 31, 10:07 UTC
Identified - The issue has been identified and a fix is being implemented.
Aug 31, 10:05 UTC
Investigating - Users attempting to log in via SSO experience intermittent success. When trying to log in through the dataset UI, they are redirected to Okta for authentication. However, after successful authentication, they are redirected back to the login UI instead of gaining access.
Sep 29, 19:00 UTC
Resolved - Certain older Agent installations were unexpectedly affected by a new scalyr.com wildcard certificate deployed on September 29. Specifically, environments using a Scalyr Agent prior to 2.1.33 (August 17, 2022) and Python prior to 2.7.9 (December 14, 2014). DataSet will stop receiving logs relayed by affected agents. Customers are advised to upgrade to the latest version of the Scalyr Agent, or at least version 2.1.33. Please contact [email protected] for further assistance.
Oct 5, 22:00 UTC
Resolved - As a result of the deployment of a new certificate on eu.scalyr.com, ingestion was interrupted for certain older Scalyr Agents sending data to this cluster as of approximately 15:00 GMT. We have now reverted this change.
This issue impacts those environments using a Scalyr Agent prior to 2.1.33 (August 17, 2022) and Python prior to 2.7.9 (December 14, 2014).
Affected customers are advised to update their Scalyr Agents to the latest version (or at least version 2.1.33) by Wednesday, October 11. We plan to deploy this change again on Thursday, October 12, 2023.
UPDATE 6 Oct 17:00 UDT: Based on customer requests for more time for agent updates, we have canceled plans to deploy this change on Thursday October 12 and will post another update here by end of day, Tuesday October 10.
UPDATE 10 Oct 17:30 UDT: We have identified a path forward which will continue to support these older environments (Scalyr Agent prior to 2.1.33 and Python prior to 2.7.9), so we have eliminated the need to update for the time being. We will be sunsetting support for these environments within the next year but will provide advance notice.
Dec 6, 06:56 UTC
Resolved - This incident has been resolved.
Dec 6, 05:15 UTC
Monitoring - We have implemented remediations to address these issues. A large ingest queue remains, so some ingestion delays may continue for the immediate future. We are continuing to monitor.
Dec 6, 04:57 UTC
Update - Query performance has been restored as of 04:20 GMT. Ingest recovery is still in progress.
Dec 6, 03:35 UTC
Identified - Customers on app.us1.dataset.com are experiencing delayed ingestion and query issues. We've identified the root cause of the issue and are actively working to mitigate the impact at this time.
Dec 21, 18:32 UTC
Resolved - Customers on app.us1.dataset.com had delayed ingestion issues from 14:18 to 18:19 GMT. The issue has been resolved.
Feb 8, 20:30 UTC
Resolved - Some users across multiple clusters experienced slow query performance and queries not returning results. This issue resulted from a code change that was introduced at 11:00 GMT on February 7. This change was reverted on February 8 at 20:45 GMT on February 8, and query performance has been restored.
Feb 25, 14:00 UTC
Completed - The scheduled maintenance has been completed.
Feb 25, 13:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 24, 00:06 UTC
Update - We will be undergoing scheduled maintenance during this time.
Feb 24, 00:05 UTC
Update - We will be undergoing scheduled maintenance during this time.
Feb 22, 22:08 UTC
Scheduled - For app.scalyr.com, app.eu.scalyr.com, app.us1.dataset.com and app.eu1.dataset.com: UI, API and ingestion will each be interrupted for 5 minutes during planned maintenance Sunday, February 25, between 13:00 and 14:00 UTC". (Updated)
Feb 26, 20:30 UTC
Resolved - Alert distribution was disrupted for several environments. The incident began 23 Feb 15:00 UTC for app.us1.dataset.com and on 23 Feb 20:00 UTC for app.eu.scalyr.com and app.eu1.dataset.com, concluding 26 Feb 20:30 UTC for all three environments.
Mar 7, 19:03 UTC
Resolved - The ingestion latency has remained normal since 3/7 5 am UTC. The incident is now resolved.
Mar 7, 07:16 UTC
Monitoring - As of approximately 3/7 5 am UTC, the ingest latency has decreased and this incident is now mitigated. We are continuing to monitor.
Mar 6, 18:45 UTC
Identified - Elevated ingestion latency on app.scalyr.com started from 3/5 11 pm GMT.
Mar 8, 18:00 UTC
Resolved - Users of app.us1.dataset.com experienced slow queries and errors between 16:50 UTC and 17:50 UTC. We have applied a fix to address the immediate issue and are implementing a code change as a long-term solution.