RPM Status

Site issue has been resolved

We apologize to our customers for any inconvenience.

Site issue 08/15-08/16 2011

We are seeing an issue with 1 of our collectors that is causing problems saving data for some customers.

Root cause: We added some disks to the array on one of our databases. We had done this twice on test systems and once on another production system with little visible impact. This particular database server is busier than the others. The added IO overhead has caused problems which manifest as flat spots in some customers’ data.

Once this process has started there is no stopping it and it must run it’s course. The array rebuild takes a long time to complete and until it is done there will continue to be customer impact. We have engineers working to mitigate customer impact till the process is done. We expect the process to complete by tomorrow morning.

Remediation:  This maintenance task was done outside of a maintenance window with the belief there would be no impact. This was a mistake and tasks like this will only be done during scheduled weekend maintenance windows in future.  Customers will be notified in advance of maintenance events like this from now on.
We sincerely apologize to the customers who have been effected.

Having trouble with a database

Some customers are unable to access rpm. Updates ASAP

Going to do some DB maintenance

One of our DBs will be off line for a short while. Some customers will be unable to access their RPM data.

Site down for scheduled maintenance

We are moving to our new Chicago data center tonight. The site will be down for several hours.

7:07pm – Site taken down

7:18pm – Broke replication between EY and Chicago

7:30pm – Snapshots of Chicago databases taken

7:45pm – Sanity checked Chicago databases

7:55pm – Start proxying agent traffic from EY to Chicago

8:10pm – Brought up Chicago environment for testing, realized that agent traffic is not getting proxied. Working on that.

8:19pm – Agent traffic from EY is now being proxied to Chicago.

8:25pm – Changing external DNS for “collector.newrelic.com”

8:43pm – New Relic engineers are fully testing the new site

9:30pm – The new site looks good. We are finished with the migration.