A denial of service (DOS) attack on our external DNS provider UltraDNS on April 30th reduced the ability of our customers to resolve our servers' DNS records from 16:00 UTC and 16:15 UTC.
Our monitoring systems started giving alerts at 16:00 UTC on April 30. Peach Payments’ services continued to run during this time, however transaction volume experienced a drop of approximately 10% from transactions in which DNS resolution was not cached.
Our internal alerts dropped significantly at around 16:15 CET, after which transaction volume returned to normal levels. After 16:15 only a few one-time alerts were triggered. As a countermeasure to reduce the impact, our incident response team increased the time to live (TTL) of our DNS service to trigger a longer caching of DNS lookups. This action decreased the chance of a failed lookup during the incident period.
From around 18:20 UTC no further DNS-related alerts were triggered by our system. The all clear however was not given by UltraDNS until 01-MAY-2014. UltraDNS stated, “As of 00:26 GMT on May 1st, DNS traffic for customers on the PDNS1-PDNS6 Name Server segment has been resolved and stable.” (PDNS1-PDNS6 are the nodes upon which the PAY.ON DNS resolution occurs)
Please be advised that the impact the situation had on our clients and their customers customers depends on the geographic origin of the transactions being sent, as well as the location and DNS configuration of our clients’ acquirers and payment partners.
Please accept our sincere apologies for any inconvenience caused. Peach Payments is actively developing a plan to prevent such a disruption of service in the future.
Once again, we apologize for any inconvenience caused.
Your Peach Payments Team