New York City, NY Cloud New York City, NY Shared
Identified - Our upstream provider has confirmed this is a facility-level network issue. We will provide further updates as they’re made available from upstream.
Update - Our upstream provider has now confirmed this issue is a facility-level power issue. They have provided an estimated resolution time of 3-4 hours. We will provide any further updates as we have them. Thank you for your continued patience during this outage.
Update – Our upstream provider has informed us this is a widespread power event impacting the entire datacenter facility. Due to this, extensive work is required before power can safely be restored. We have been informed this work will take hours to complete, and we likely will not see connectivity restored before 8:00 AM EDT / 12:00 PM UTC. We apologize for the extended downtime.
Update 07-11-2023 13:25:00 UTC – This is still an ongoing issue. We have been made aware that due to a brief fire in the UPS room at the facilty, all power to the facility was cut off while the fire marshall and relavent parties perform an inspection and approval to re-energize the site. As of this time we do not have an ETA for when power will be restored.
Update 07-11-2023 19:35:00 UTC – Our upstream provider has advised us that the facility remains powered down as per the fire marshal. There is an ongoing meeting with relevant parties to discuss the cleanup process and get approval from the fire marshal to re-energize the facility.
Update 07-11-2023 23:34:00 UTC – Our upstream provider has provided us the following update:
Evocative onsite management met with the fire marshal and electrical inspectors to review the current state of the facility. The fire marshal has reviewed the current level of progress on the cleaning and requested additional remediation actions around the fire system components. The follow up inspection from the fire marshal is 9:00 AM EDT, Wednesday. The Evocative team is bringing on additional personnel and vendors to comply completely with the additional requests by this deadline.
In preparation for being able to allow clients onsite, the fire marshal has stated that Evocative must perform a full test of the fire/life safety systems. This must be performed after utility power has been restored and fire system components replaced. Vendors have already been scheduled and are standing by to complete this work tomorrow.
If there are no significant changes in the timeline, the earliest INAP will be allowed back into the site to power up servers will be later in the day on Wednesday. We anticipate the next update of significance will be provided by Evocative Wednesday morning, and will keep you updated as information becomes available.
Update 07-12-2023 15:07:00 UTC – Our upstream provider has provided us with the following update:
The initial inspection has been successfully completed, and we are now starting the process to power up the data center.
Update 07-12-2023 17:11:00 UTC – Our upstream provider has provided us with the following update:
We have completed the entire site inspection with the fire marshal and the electrical inspector and utility power has been restored.
We are now working to restore critical systems and our onsite team has energized the primary electrical equipment that powers the site.
Concurrently, we are beginning work to bring the mechanical plant online. Additional engineers from other facilities are on site this morning to expedite site turn-up.
The ETA for bringing up the critical infrastructure systems is approximately 5 hours.
We are planning for a late afternoon/early evening time frame when clients can come back on-site.
We will send out additional information regarding access to the facility and remote hands assistance and we will notify you once client access to the facility is permitted
Update 07-12-2023 18:00:00 UTC – Our upstream provider has provided us with the following update:
Our onsite team has energized the primary electrical equipment that powers the site, enabling us to bring our mechanical plant online. We are currently cooling the facility.
As we monitor for stability, we are focused on bringing up our electrical systems. In starting this process, we have identified an issue with powering up our fire panel as well as power systems that were powered by UPS3. While this will cause us a delay, we are working with our vendors for remediation.
We are currently at 25% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 7 hours. We are still planning for an evening time frame when clients will be able to come back on-site. We will send out additional information regarding access to the facility and remote hands assistance and we will notify you once client access to the facility is permitted.
Update 07-12-2023 19:12:00 UTC – Our upstream provider has provided us with the following update:
Our onsite team is currently bringing our UPS Systems online. We have our UPS vendor onsite assisting with this as we bring up UPS-4 and then UPS-R after that.
As these systems are brought online we will concurrently work on bringing carriers up.
We are currently at 30% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 6 hours. We are still planning for an evening time frame when clients will be able to come back on-site.
We will send out additional information regarding access to the facility and remote hands assistance and we will notify you once client access to the facility is permitted.
Update 07-12-2023 22:00:00 UTC – Our upstream provider has provided us with the following update:
Our onsite team is currently bringing our UPS Systems online. We have our UPS vendor onsite assisting us with this. We have brought UPS-4 online and are currently charging its associated battery system. While bringing UPS-R online we have run into a minor issue that we are currently investigating.
We are currently at 35% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 5 hours. We are still planning for an evening time frame when clients will be able to come back on-site.
In the process of our power system re-energizing, we have been working on our fire system as well. We are currently sourcing materials to bring our fire system fully online and do not have an ETA for completion. Because of this and fire marshal compliance, we will only be allowed to have supervised escorted customer access when we finish bringing up the critical infrastructure systems. We are currently sourcing additional personnel to assist us with this escort policy.
Update 07-12-2023 23:00:00 UTC – Our upstream provider has provided us with the following update:
Our onsite team is currently bringing our UPS Systems online. We have our UPS vendor onsite assisting us with this. We have brought UPS-4 online. We resolved our issues with UPS-R and have brought it online and are currently charging its associated battery system. We will now begin work on bringing UPS-1 online. After bringing the UPS Systems online, we will then fully transfer the load to our downstream power distribution, which will enable us to start powering up individual customer cabinets.
We are currently at 40% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 4 hours. We are still planning for an evening time frame when clients will be able to come back on-site.
We are currently sourcing materials to bring our fire system fully online and do not have an ETA for completion. Because of this and fire marshal compliance we will only be allowed to have supervised escorted customer access when we finish bringing up the critical infrastructure systems.
Update 07-13-2023 01:00:00 UTC – Our upstream provider has provided us with the following update:
Our onsite team is currently bringing our UPS Systems online. We now have UPS-4 and UPS-R online. While bringing up UPS-1, we ran into an issue and we are unable to bring it online. We have engaged with our vendor and are finding a workaround to deliver power downstream to customer cabinets.
Due to the issue with UPS-1, we will not be automatically powering up all customer cabinets. We will be reaching out to you to let you know if you are part of the group of customers on unprotected/non-UPS backed utility power so that you can make the decision whether to energize your cabinets at that time.
We are currently at 50% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 3 hours. The current time frame for when clients will be able to come back onsite is approximately 10:30 PM EDT.
We are currently sourcing materials to bring our fire system fully online and do not have an ETA for completion. Because of this and fire marshal compliance we will only be allowed to have supervised escorted customer access when we finish bringing up the critical infrastructure systems. We will have additional personnel onsite to assist us with this escort policy.
Update 07-13-2023 07:00:00 UTC – The power in the facility has been restored, and all of our servers are now online. We are actively restoring service to all cloud-based services, including our shared, reseller, and semi-dedicated plans.
Update 07-13-2023 07:27:00 UTC – Everything is online at this time. If you’re still experiencing issues, please contact our support department for further investigation.
Last updated: March 20, 2024 at 2:06 PM UTC