We are investigating delays and errors in the creation of Private Spaces on the Fir-generation in our Oregon region
Beginning at 19:37 PM UTC on April 9, 2025 and continuing until 19:49:15 PM UTC on April 9, 2025, Heroku Platform API was unavailable during automatic database failover. During this time some requests to the Platform API failed. Customers would have received HTTP 503 Service Unavailable errors when making API requests, and seen errors when using the Heroku Dashboard or CLI.
Engineers are investigating errors with the Heroku Platform API. Customers may receive HTTP 503 errors when making API requests, and when using the Heroku Dashboard or CLI.
Engineers are investigating an issue with high request latencies and reduced availability for HTTP requests on custom domains for a subset of US Common Runtime customers.
Salesforce Technology Service Delivery Performance Degradation on Heroku Connect on 17 Mar, 2025
Preliminary Root Cause Analysis/Root Cause Analysis
We sincerely apologize for any impact this incident may have caused you and your business. At Salesforce, Trust is our #1 value, and security is our top priority. We value transparency and want to take this opportunity to outline the facts regarding a recent service disruption that may have impacted your ability to use multiple Salesforce services, based on our current understanding. Our investigation is ongoing, and we will provide you with updated information as it becomes available.
Executive Summary - What Happened? Starting at 2025-03-17 22:29 UTC and continuing until 2025-03-18 at 08:59 UTC, a small subset of Heroku Connect data synchronizations between Heroku Postgres and Salesforce instances were delayed. During the 10 hour impact window, 3% of Heroku Connect Customers experienced varying durations of delay. How did this issue impact Salesforce services? For this subset of Heroku Connect Customers, Heroku Connect experienced a degraded synchronization experience between their Heroku Postgres and Salesforce instances.
Technical Details
Detection and Initial Impact
Impact began at 2025-03-17 at 22:29 UTC and continued until 2025-03-18 at 08:59 UTC. Total degraded synchronization impact varied by customer. Certain customers experienced synchronization interruptions during this incident, while others could not set up new Heroku Connect add-ons. This performance degradation was detected by the Heroku Technology team's automated monitoring. An initial attempt at remediation was incorrectly believed to have resolved the issue.
Remediation
Heroku altered its Web Application Firewall configuration. The change was reverted at 2025-03-18 08:59 UTC.
Root Cause Analysis The Technology team’s post-incident investigation and analysis determined that the incident was triggered by a change to the Heroku platform which put Heroku Data Services behind a Web Application Firewall. The new Web Application Firewall provider rejected certain requests from Heroku Connect that submitted GET requests with a non-empty body as part of the normal Heroku Postgres synchronization.
Next Steps
To maintain the performance level that our customers expect from Salesforce and to prevent this defect from recurring, our focus is on continuous improvement. The Technology team has identified and are implementing the following actions:
Complete: With the new Web Application Firewall configuration, the communication between Heroku Connect and Heroku Postgress will no longer reject requests with a non-empty body as part of the request.
In Progress: Enhanced monitoring and staging environment improvements to increase detection.
We sincerely apologize for the impact this incident may have caused you and your business; Salesforce is fully committed to minimizing downtime when incidents do occur. We also continually assess and improve our tools, processes, and architecture to provide you with the best service possible.
Salesforce Technology Service Delivery Performance Degradation on Heroku Connect on 17 Mar, 2025 Preliminary Root Cause Analysis/Root Cause Analysis
We sincerely apologize for any impact this incident may have caused you and your business. At Salesforce, Trust is our #1 value, and security is our top priority. We value transparency and want to take this opportunity to outline the facts regarding a recent service disruption that may have impacted your ability to use multiple Salesforce services, based on our current understanding. Our investigation is ongoing, and we will provide you with updated information as it becomes available.
Executive Summary - What Happened? Starting at 2025-03-17 22:29 UTC and continuing until 2025-03-18 at 08:59 UTC, a small subset of Heroku Connect data synchronizations between Heroku Postgres and Salesforce instances were delayed. During the 10 hour impact window, 3% of Heroku Connect Customers experienced varying durations of delay. How did this issue impact Salesforce services? For this subset of Heroku Connect Customers, Heroku Connect experienced a degraded synchronization experience between their Heroku Postgres and Salesforce instances. Technical Details
Detection and Initial Impact Impact began at 2025-03-17 at 22:29 UTC and continued until 2025-03-18 at 08:59 UTC. Total degraded synchronization impact varied by customer. Certain customers experienced synchronization interruptions during this incident, while others could not set up new Heroku Connect add-ons. This performance degradation was detected by the Heroku Technology team's automated monitoring. An initial attempt at remediation was incorrectly believed to have resolved the issue.
Remediation Heroku altered its Web Application Firewall configuration. The change was reverted at 2025-03-18 08:59 UTC.
Root Cause Analysis The Technology team’s post-incident investigation and analysis determined that the incident was triggered by a change to the Heroku platform which put Heroku Data Services behind a Web Application Firewall. The new Web Application Firewall provider rejected certain requests from Heroku Connect that submitted GET requests with a non-empty body as part of the normal Heroku Postgres synchronization.
Next Steps To maintain the performance level that our customers expect from Salesforce and to prevent this defect from recurring, our focus is on continuous improvement. The Technology team has identified and are implementing the following actions:
Complete: With the new Web Application Firewall configuration, the communication between Heroku Connect and Heroku Postgress will no longer reject requests with a non-empty body as part of the request.
In Progress: Enhanced monitoring and staging environment improvements to increase detection.
We sincerely apologize for the impact this incident may have caused you and your business; Salesforce is fully committed to minimizing downtime when incidents do occur. We also continually assess and improve our tools, processes, and architecture to provide you with the best service possible.
This issue is now resolved.
Beginning at 16:30 UTC on March 25, 2025, some API events are not being captured in application logs. These log lines are proceeded by the app[api] identifier and include details of deploys, restarts, CLI commands and other API events.
This does not affect the execution of those commands, only their visibility in application logs.
Beginning at 16:30 UTC on March 25, 2025, some API events are not being captured in application logs. These log lines are proceeded by the app[api] identifier and include details of deploys, restarts, CLI commands and other API events.
This did not affect the execution of those commands, only their visibility in application logs.
Engineers are investigating an issue with Application Logs. Application logs normally preceded with the app[api] identifier may not appear.