Hypera Development - お知らせ履歴

この問題は解決されました。

This incident has been resolved.

We are continuing to investigate this incident, however we have not heard anything further from our hosting provider.

Our hosting provider has been contacted regarding this issue, as it seems to be a networking issue.

We manually started the server again. We are currently trying to figure out what caused the server to be killed.

This incident has been resolved.

It appears our server was shutdown during our hosting providers "temporary node migration", and was not started again afterwards.

We have restarted the server manually and are monitoring stability.

At 17:35 UTC our hosting provider posted an announcement:

VPS in NYC is being migrated to temporary nodes. Please bear with us. There will be a few minutes of downtime during the migration.

At 17:43 UTC we were alerted of an outage.

We are unsure whether this outage is related to the "temporary node migration" announced by our hosting provider, especially as their announcement states "a few minutes of downtime" and this has lasted over an hour.

We were also not provided with prior notice, nor a reason, for this migration.
We will be continuing to investigate the cause of this issue.

We've identified the cause of this issue and successfully rolled out a fix.

当社のホスティングプロバイダーはパフォーマンスの問題に対する修正を実装しており、現在その結果を監視しています。

依然として異常に高いパケット損失と遅延が発生していますが、改善され始めているようです。

当社のホスティングプロバイダーは、7 月 30 日 04:01 UTC に次の声明を発表しました。

この障害は東部標準時間約 9 時 30 分に解決されました。今後もネットワークの監視を続けていきます。

インフラストラクチャ全体で異常に高いパケット損失と遅延が依然として発生しており、原因を調査中です。

当社のホスティングプロバイダーは次の声明を発表しました。

私たちのラックプロバイダーは、私たちが請求書を支払ったにもかかわらず、支払わなかったと主張しています。証拠は提出されているが、動じていない。私たちの銀行はこれを確認し、ポートを増やす必要があります。報酬はFloofy Hostingのクライアントに与えられます。

当社のホスティングプロバイダーは次の声明を発表しました：「当社のラックプロバイダーが不当にポートを拒否しました。当社はこの問題を解決するために彼らと協力しています。」

私たちはこの問題をさらに調査し、インフラストラクチャの継続的な停止の数を減らすことができる代替ソリューションを検討する予定です。

ご不便おかけしてすみません。

この問題は自動的に解決されたようです。

トラフィックをプライマリ クラスターに再ルーティングし、すべての操作が通常どおり継続しています。

この障害によりご迷惑をおかけしたことをお詫び申し上げます。

API と Discord Bot (Axolotl) をプライマリ クラスターに戻します。これにより、数分間のダウンタイムが発生する可能性がありますが、切り替えはほぼ瞬時に行われ、目立ちません。

Power has been restored to the datacenter and our infrastructure is now accessible again. We will be moving traffic back from our temporary infrastructure later today.

Maintenance on our Discord Bot, Axolotl, has been completed.

We are quickly performing on our Discord Bot, Axolotl, to prevent issues from occurring when the Secaucus, NJ data-center comes back online.

INAP has posted an incident update:

We have completed the entire site inspection with the fire marshal and the electrical inspector and utility power has been restored.

We are now working to restore critical systems and our onsite team has energized the primary electrical equipment that powers the site.

Concurrently, we are beginning work to bring the mechanical plant online. Additional engineers from other facilities are on site this morning to expedite site turn-up.

The ETA for bringing up the critical infrastructure systems is approximately 5 hours.

We are planning for a late afternoon/early evening time frame when clients can come back on-site.

We will send out additional information regarding access to the facility and remote hands assistance and we will notify you once client access to the facility is permitted

We are currently looking for options to temporarily restore our Maven Repository, however as the majority of our libraries are also hosted on Sonatype OSS, we are not treating this as urgent.

We are also creating plans to make our infrastructure automated and highly available in the near future, allowing us to stay online when an incident like this occurs.

We apologise for any inconvenience this outage has caused.

INAP posted an incident update on July 11, 2023 at 16:04 CDT:

The Secaucus data center remains powered down at this time, per the fire marshal.

Evocative onsite management met with the fire marshal and electrical inspectors to review the current state of the facility. The fire marshal has reviewed the current level of progress on the cleaning and requested additional remediation actions around the fire system components. The follow up inspection from the fire marshal is 9:00 AM EDT, Wednesday. The Evocative team is bringing on additional personnel and vendors to comply completely with the additional requests by this deadline.

In preparation for being able to allow clients onsite, the fire marshal has stated that Evocative must perform a full test of the fire/life safety systems. This must be performed after utility power has been restored and fire system components replaced. Vendors have already been scheduled and are standing by to complete this work tomorrow.

If there are no significant changes in the timeline, the earliest INAP will be allowed back into the site to power up servers will be later in the day on Wednesday. We anticipate the next update of significance will be provided by Evocative Wednesday morning, and will keep you updated as information becomes available.

We have successfully deployed our application proxy and API on our temporary New Jersey infrastructure, restoring api.hypera.dev.

We have setup temporary infrastructure in New Jersey and have restored Axolotl so we can continue providing support via Discord. We will be restoring our application proxy and our API soon.

We are currently in the process of starting up temporary infrastructure to bring some of our essential services back online.

INAP raised an incident on July 10th, 2023 at 17:32 CDT stating that they received reports of loss of connectivity affecting their New Jersey (NYJ) datacenter, and that they had started working to identify the issue.

At July 10th, 2023 at 22:21 CDT, INAP released an update stating the following:

A small electrical failure occurred with one of the redundant UPS devices which began smoking and led to a small fire in the UPS room. The fire department was dispatched and the fire extinguished quickly. The fire department subsequently cut power to the datacenter and disabled the generators while they and the utility company inspected the electrical system.

The fire marshal is requiring extensive cleaning of the UPS devices and rooms before the site can be reenergized. There is a vendor onsite currently who will be performing the cleanup.

We do not currently have an ETA for when the datacenter will be back, and therefore are unable to provide an ETA for when we can restore our services.
We will be looking into alternative solutions in other datacenters until INAP's NYJ datacenter has been restored.

We've identified the cause of this issue and successfully rolled out a fix.

We are rolling out a fix for this issue and monitoring the results.

This incident has been resolved.

We have notified our hosting provider of this issue and they are working on a fix.