PayPal payments are failing on embedded forms

Incident Report for Beacon

Postmortem

Status

This incident has now been fully resolved and PayPal payments are processing as before, including on embedded forms.

It's my responsibility to keep Beacon fully operational at all times and I've missed the mark this week. I'm really sorry.

What happened?

We started seeing a large volume of payment failures from PayPal, as reported by our internal monitoring systems, and customers reported seeing errors when submitting payments through PayPal. From our investigations it became clear that embedded forms were especially, but not exclusively affected.

What did we do?

Given that the failures happened at form submit time which created a very poor experience for donors we took the decision to switch off PayPal as a payment option on all forms.

What was the problem?

We found that there was a problem with the authentication flow between Beacon and PayPal. Part of our authentication process involved using a secure access token generated by PayPal. We found that this access token was not being created consistently. We reached out to PayPal for guidance on how to correctly handle this authentication flow.

How did we fix the problem?

On PayPal’s recommendation, we made changes to our payment processing system to generate our own secure tokens rather than using the ones provided by PayPal. This fixed the problem immediately.

Thanks for your patience, and I’m really sorry for the disruption that this will have caused. If you have any further questions or concerns, you can contact me directly at david@beaconcrm.org.

David - Chief Technology Officer

Posted Oct 19, 2023 - 12:21 BST

Resolved

We've taken the decision to switch off the option to pay with PayPal from all Beacon forms while we resolve the ongoing issue. No action is required by you. Users will no longer be able to see the option to pay with PayPal on any forms.

This is a tough decision, and one that we do not take lightly. We've decided that it is best to ensure that we provide a smooth and consistent donation experience, even if we can't offer the full range of payment platforms for a while.

We will be working hard to bring PayPal payments back online, and we have already reached out to PayPal to report the issue and to seek their guidance.

Posted Oct 16, 2023 - 16:45 BST

Investigating

We have been investigating further and are seeking support from PayPal to resolve the issue. From our testing we are confident that, only embedded forms are affected, but not on every payment submission. Users wishing to make payments via PayPal should be directed to standalone versions of forms while we fix the issue.

Posted Oct 16, 2023 - 11:07 BST

Update

We have raised the issue with PayPal and are awaiting a response.

Posted Oct 13, 2023 - 17:29 BST

Identified

We've identified the cause of the issue and we're working on a fix.

Posted Oct 11, 2023 - 14:42 BST

Update

We are continuing to investigate this issue.

Posted Oct 11, 2023 - 13:35 BST

Investigating

We are currently investigating an issue in which PayPal payments through embedded Beacon forms are failing.

Upon clicking 'Submit' on a form, users are shown an error stating 'Failed to process payment through PayPal'. No payments are being taken from users' PayPal accounts, and no data is written to Beacon.

All embedded payment forms (donation, membership, ticketing) are affected.
Standalone (non-embedded) Beacon forms are unaffected.

Posted Oct 11, 2023 - 13:33 BST

This incident affected: Forms and Payments.