December 4, 2023 10:22AM PST
Scheduled - The RAID controller on ftp-osl needs to be replaced and the machine will be offline for up several hours (likely less) while this happens. This system was taken out of rotation today however other projects may still be pointing directly at this system.
The actual service window might change depending on the arrival time of the IBM tech.
December 6, 2023 9:30AM PST
Active - Scheduled maintenance is starting.
December 6, 2023 11:30AM PST
Completed - Scheduled maintenance is complete.
December 18, 2023 12:53PM PST
Scheduled - We're going to be performing an upgrade of the cluster from Stein to Train on December, 19 2023 at 10AM-12PM PST (1800-2000 UTC). Any virtual machines that are running at the time should remain online during the upgrade and I do not expect any major outage for those VMs. However, the API interface will be down and partially available during this time period so new VM creation/deletion/etc will not work.
This will be the first phase of the massive upgrade project we are doing with this cluster. After the new year, we will start migrating all cluster backend nodes to AlmaLinux 8 which will result in a larger outage. We will send out an announcement for that in a few weeks with more details.
December 19, 2023 10:00AM PST
Active - Scheduled maintenance is starting.
December 19, 2023 12:00PM PST
Completed - Scheduled maintenance is complete.
December 18, 2023 12:54PM PST
Scheduled - We're going to be performing an upgrade of the cluster from Stein to Train on December, 19 2023 at 1-3PM PST (2100-2300 UTC). Any virtual machines that are running at the time should remain online during the upgrade and I do not expect any major outage for those VMs. However, the API interface will be down and partially available during this time period so new VM creation/deletion/etc will not work.
This will be the first phase of the massive upgrade project we are doing with this cluster. After the new year, we will start migrating all cluster backend nodes to AlmaLinux 8 which will result in a larger outage. We will send out an announcement for that in a few weeks with more details.
December 19, 2023 1:00PM PST
Active - Scheduled maintenance is starting.
December 19, 2023 3:00PM PST
Completed - Scheduled maintenance is complete.
December 18, 2023 12:55PM PST
Scheduled - We're going to be performing an upgrade of the cluster from Stein to Train on December, 20 2023 at 10AM-12PM PST (1800-2000 UTC). Any virtual machines that are running at the time should remain online during the upgrade and I do not expect any major outage for those VMs. However, the API interface will be down and partially available during this time period so new VM creation/deletion/etc will not work.
This will be the first phase of the massive upgrade project we are doing with this cluster. After the new year, we will start migrating all cluster backend nodes to AlmaLinux 8 which will result in a larger outage. We will send out an announcement for that in a few weeks with more details.
December 20, 2023 10:00AM PST
Active - Scheduled maintenance is starting.
December 20, 2023 12:00PM PST
Completed - Scheduled maintenance is complete.
March 8, 2024 9:34AM PST
Scheduled - Service(s) affected:
FTP mirroring service which includes (but not limited to) the following hostnames:
- ftp-chi.osuosl.org
Reason for outage:
We will upgrade the operating system from CentOS 7 to AlmaLinux 8. Unfortunately, due to an issue with how the disks were partitioned, we are unable to do in-place upgrades. This will require to do a full reinstall including re-syncing all of the FTP content on each system after reinstallation. The re-sync will likely take multiple days due to the size of the content.
This will be a multi-day outage and will only impact one server at a time. This server has been removed from the DNS rotation of ftp.osuosl.org and rsync.osuosl.org during its outage which will minimize the impact of issues with users. If projects or users are directly referencing ftp-chi, please make sure you change it to ftp.osuosl.org otherwise you will be impacted by this maintenance.
March 11, 2024 9:00AM PDT
Active - Scheduled maintenance is starting.
March 15, 2024 5:00PM PDT
Completed - Scheduled maintenance is complete.
March 8, 2024 9:35AM PST
Scheduled - Service(s) affected:
FTP mirroring service which includes (but not limited to) the following hostnames:
- ftp-nyc.osuosl.org
Reason for outage:
We will upgrade the operating system from CentOS 7 to AlmaLinux 8. Unfortunately, due to an issue with how the disks were partitioned, we are unable to do in-place upgrades. This will require to do a full reinstall including re-syncing all of the FTP content on each system after reinstallation. The re-sync will likely take multiple days due to the size of the content.
This will be a multi-day outage and will only impact one server at a time. This server has been removed from the DNS rotation of ftp.osuosl.org and rsync.osuosl.org during its outage which will minimize the impact of issues with users. If projects or users are directly referencing ftp-nyc, please make sure you change it to ftp.osuosl.org otherwise you will be impacted by this maintenance.
March 18, 2024 9:00AM PDT
Active - Scheduled maintenance is starting.
March 22, 2024 5:00PM PDT
Completed - Scheduled maintenance is complete.
March 26, 2024 9:30PM PDT
Investigating - One of the servers (ftp-nyc) behind ftp.osuosl.org appears to be unable to connect to most of the internet currently. We have temporarily taken it out of rotation until this gets resolved by the provider.
March 27, 2024 1:04PM PDT
Resolved - The outage was due to a router being upgraded and BGP routes not being added back properly. Both IPv4 and IPv6 routing has been restored and ftp-nyc is back in rotation.
March 8, 2024 9:38AM PST
Scheduled - FTP mirroring service which includes (but not limited to) the following hostnames:
ftp2.osuosl.org
ftp-osl.osuosl.org
rsync2.osuosl.org
Reason for outage:
We will upgrade the operating system from CentOS 7 to AlmaLinux 8. Unfortunately, due to an issue with how the disks were partitioned, we are unable to do in-place upgrades. This will require to do a full reinstall including re-syncing all of the FTP content on each system after reinstallation. The re-sync will likely take multiple days due to the size of the content.
This will be a multi-day outage and will only impact one server at a time. This server has been removed from the DNS rotation of ftp.osuosl.org and rsync.osuosl.org during its outage which will minimize the impact of issues with users. If projects or users are directly referencing ftp-chi, please make sure you change it to ftp.osuosl.org otherwise you will be impacted by this maintenance.
Since ftp-osl acts as a primary for our mirror infrastructure, we have set up a temporary server to minimize the impact of syncing while we reinstall the hardware running ftp-osl. This server is not in the main DNS rotation, but it will still be syncing content for the other servers.
April 1, 2024 9:00AM PDT
Active - Scheduled maintenance is starting.
April 2, 2024 8:13AM PDT
Completed - This was completed early and ftp-osl is back in rotation.
April 3, 2024 9:51AM PDT
Investigating - One of the production ganeti nodes was having resource issues and required a reset. It is currently coming back online as we speak.
April 3, 2024 10:09AM PDT
Resolved - This node is back online and all instances should be back to normal
August 5, 2024 10:41AM PDT
Scheduled - We got the following notification from Link Oregon that I wanted to pass on to you which impacts our connectivity to the Internet:
> On August 13, 2024 (08/13), Oregon State University (OSU) will be cutting both primary and backup power to its data center in the Kerr Administration building, to install new power distribution and backup equipment. This will take two of Link Oregon’s points of presence offline, resulting in a loss of all provisioned network services provided by Link Oregon at those POPs, over a period of approximately eight (8) hours (08/13-20:00 to 08/14-04:00).
To be clear, this does not affect the data center where we host all of our servers so power will remain online. HOWEVER, this data center does house our uplink to Link Oregon (our ISP).
UPDATE:
LinkOregon and OSU have figured some secondary power to keep our connection to LinkOregon online during the outage. They will be connecting some temporary rack-based UPSs that are connected to the "house power" (i.e. not generator backed). Now, if for some reason there is a power outage from the "house power", the temporary UPS systems will only last for 10-20 minutes.
August 13, 2024 8:00PM PDT
Active - Scheduled maintenance is starting.
August 14, 2024 4:00AM PDT
Completed - Scheduled maintenance is complete.