My Shopify app hides, renames and reorders payment and shipping methods at checkout. One screen in it sets up B2B: you say who counts as a business buyer, and it writes the rules for you.
This week the first store from outside my own test shops used that screen for real. It picked customer tags, created two payment rules, deleted both, and uninstalled the app 1 hour and 51 minutes after installing it. The uninstall reason said it wasn't working properly with the store.
The rule engine had no bug.
What the event log showed
The app writes one line per action, so the store's path was easy to read back:
- The setup screen created two rules from a customer tag.
- Both rules were created switched off. That's the default for anything the setup writes.
- The store was on the free plan, which runs one rule at a time.
- Nothing in the log shows a rule being switched on from the rules list.
- The only other save was one rule-editor save, ten seconds before both rules were deleted. At the time, that line didn't record on or off.
So whatever the store tested at checkout, it most likely tested with no rule running.
Replaying the rule
I still wanted to know whether the rule would have worked once it was on. The setup's tag rule hides a business-only payment method from anyone without the wholesale tag. I ran it through the engine with a few buyers.
A logged-in customer with the tag saw the method. A logged-in customer without it didn't. A guest didn't either. Tag case and a trailing space made no difference, and there's a test for the case part.
The engine did what it was told.
Where tags come from
My function asks checkout for the buyer's tags like this:
buyerIdentity {
customer {
hasTags(tags: $customerTags) {
tag
hasTag
}
}
}
The tags hang off customer, and Shopify's schema describes that field as the customer interacting with the cart. The buyer's email is a separate field. A guest who types a wholesale buyer's email address is still a guest. There's no customer object, so there are no tags, and the rule treats them as retail.
My setup screen didn't say that. The option for Shopify B2B companies started with "Logged in". The tag option said "Customers with a tag".
I can't say whether this store would have hit the guest problem next. The log points to the first one: the setup left both rules off, and nothing shows them turned on.
I also checked whether functions needed a bigger Shopify plan. They don't for apps distributed through the App Store, so that wasn't it.
What changed the same day
- The end of B2B setup now has a Turn them on now button. It switches on the new rules in order while the plan allows, and names the ones that stay off.
- The setup states the free plan's limit and how many rules are running, even when the new ones fit.
- The tag option now reads "Logged-in customers with a tag", with this note under the tag field (the rule tester got a shorter one):
Tags count only for customers logged in at checkout. A guest who types a business buyer's email is treated as retail, so test with a logged-in account.
- Results show right under the button. Pressing create again on rules that are already saved but off offers the same turn-on button.
Then I ran a real checkout on a test store. As a guest, cash on delivery was hidden, and logged in as a wholesale customer, it showed. No order was placed.
Later that day the same store installed the app again, and it now has one rule switched on. It didn't go through the B2B setup this time, and the log doesn't say why it came back.
Takeaway
A rule that's saved but off and a rule that reads a field checkout never fills look the same from the merchant's side. Nothing happens.
If I could only ship one of those fixes, I'd ship the turn-on button. It's the one the log points to.
If you've built anything on customer tags at checkout, which one bit you first: a setting that was saved but never active, or a guest checkout with no customer attached?
This post was drafted with AI help. The numbers and facts come from my own logs and repo.
Top comments (4)
'The rule engine was right' is the most dangerous sentence in product work, because it's true and it doesn't help. The store didn't uninstall your engine; it uninstalled its first hour with your setup screen. What I take from your log work: the uninstall reason said 'wasn't working properly,' and the event log said otherwise — the stated reason was a mood, the log was the fact. Every churn I've ever understood came from reading the path, not the exit survey. The fix is almost never in the part that's correct. It's in the part where a correct thing asked a new user to do something unclear at minute ten.
Agreed on where the fix lives. The awkward part is that off-by-default was deliberate. A comment in the setup code says the rules come out switched off, like presets, so nothing reaches a live checkout until the merchant has tried both buyers in the tester and turned them on.
That protects a store from a half-checked rule. It also lets a new user finish setup with nothing running. The turn-on button at the end is my attempt to keep the first part and lose the second.
When you've seen this kind of gap fixed, did the default change, or did a step get added at the end like this one?
Your event log did the important job: it separated the merchant's stated reason from the actual path. I'd also look at the first successful outcome, not just the first saved rule - could setup finish with a test checkout that visibly confirms the rule is active? That gives the merchant proof before they leave the flow.
Setup gets partway there. After the turn-on button, it offers a name check: the merchant pastes the payment method name as checkout shows it, and the app compares it with the same matcher the checkout function uses. Before anything is on, it points to the rule tester, which can compare a business buyer with a retail buyer.
Both checks run inside the app, so neither shows the rule firing at a real checkout. That's the proof you're describing, and setup doesn't have it.
Would you get that proof from a test order, or from something the function reports back after a checkout?