This post is for our Paid Subscribers. If you haven’t subscribed yet,
Clarifying Questions
Before diving in, I would want to align on scope and make sure I am solving the right problem at the right level of specificity.
Which scam category should we prioritize? Marketplace and buy-sell fraud, romance and relationship scams, and investment-opportunity scams promoted through ads all represent genuinely different fraud patterns requiring different detection and intervention approaches. Clarifying this prevents us from building a generic system that is mediocre across all three rather than excellent at one.
What does “success” mean for Meta’s business here? Is the primary driver regulatory compliance and avoiding government fines, protecting platform trust to reduce churn, or reducing direct financial liability? The answer shapes whether we optimize for recall (catching every scam at the cost of false positives) or precision (protecting legitimate users from wrongful flags).
Are we designing detection, intervention, or both? A detection-only system surfaces insights to a trust-and-safety team. An intervention system interacts directly with users in real time. These have very different latency, privacy, and false-positive risk profiles.
What data assets and ML infrastructure does Meta already have? Meta’s integrity teams already run large-scale behavioral classifiers. Knowing what exists avoids rebuilding infrastructure and clarifies whether this is a net-new product or a layer on top of existing tooling.
Are there geographic or regulatory constraints we must respect from day one? GDPR in Europe and the UK’s Online Safety Act both have implications for analyzing private message content, and some markets have specific financial-fraud reporting obligations that would affect our design.
For this answer I will assume Meta wants to tackle the romance-to-investment-scam escalation pattern, the well-documented sequence where a scammer builds fake emotional intimacy over weeks through Messenger or Instagram DMs before pivoting to a fraudulent investment or crypto opportunity. This pattern is worth targeting because it causes disproportionate financial harm per victim (FBI IC3 data from 2023 puts average romance-scam losses at roughly $50,000 per victim), it follows a behaviorally distinctive arc that machine learning can detect before money moves, and it is currently under-addressed by purely reactive, report-based moderation. I will also assume we are designing both a detection engine and a real-time, user-facing intervention layer, and that we are building on top of Meta’s existing integrity infrastructure rather than from scratch.
Product Description
Meta is the world’s largest social network by monthly active users, with roughly 3.3 billion people using at least one of its apps every day as of early 2024. Its family of apps includes Facebook, Instagram, WhatsApp, and Messenger, and the company generated approximately $134 billion in revenue in 2023, almost entirely from advertising. Trust and safety is both a regulatory imperative and a direct business input: advertiser confidence and user retention both depend on people believing Meta’s platforms are safe places to spend time and share personal information.
Romance-to-investment scams, sometimes called “pig butchering” scams after the Chinese term sha zhu pan, have grown dramatically in scale. The FBI’s 2023 Internet Crime Report recorded over $4.5 billion in investment-fraud losses in the United States alone, with romance-scam-to-crypto-investment escalation being the fastest-growing subcategory. These scams typically originate in Southeast Asian fraud compounds, operate across multiple platforms, and target adults aged 40 to 70 who are recently widowed, divorced, or socially isolated. A single scam operation can run thousands of simultaneous “relationships” and extract anywhere from $10,000 to over $500,000 per victim over a multi-week or multi-month grooming period. Meta’s platforms, particularly Facebook Groups and Instagram DMs, are frequent origination points where initial contact is made before conversations migrate to WhatsApp or other channels.
The differentiated opportunity here is proactive, pattern-based intervention before money moves. Current trust-and-safety approaches are largely reactive: a user files a report, an analyst reviews it, and the scammer’s account is removed, often after the victim has already suffered devastating financial loss. Because the romance-to-investment-scam pattern follows a distinctive behavioral arc, specifically an accelerating emotional-intimacy trajectory (high-frequency messaging, affectionate language escalation, manufactured crisis moments) followed by a sudden financial ask, it is statistically separable from genuine relationship formation at a meaningful confidence level. This gives Meta a window to intervene before the wire transfer, which is the moment that actually matters.
Define Goal
The core problem is that Meta’s platforms are a primary point of first contact for one of the most financially devastating fraud patterns in the world, and the current reactive, report-driven moderation approach means most victims have already lost money before any platform-level action is taken. Meta has the behavioral data and the ML infrastructure to change this, but has not yet built a proactive, real-time intervention product specifically tuned to the romance-to-investment-scam escalation pattern.
The goal I want to focus on is: reduce the financial harm caused to Meta’s users by romance-to-investment-scam escalation by surfacing a proactive, empathetic, and actionable warning before a payment is initiated, targeting a 40 percent reduction in completed financial transfers to flagged accounts within 18 months of full rollout.
The north star metric I would track is pre-transaction warnings acknowledged per week, measured among users whose conversation has been flagged with at least 80 percent classifier confidence within the prior 7 days. I prefer this over a simpler “scammer accounts removed per week” metric because account removal is a lagging signal that measures reactive action after harm is already possible. Warnings acknowledged is a leading signal that measures whether the intervention is actually reaching vulnerable users at the right moment. It also captures false-negative risk: if acknowledged warnings per week flatlines while new scam accounts keep appearing, we know our detection coverage is slipping.
User Segmentation
Because this is a two-sided problem, involving both victims and bad actors, I want to segment each side separately before choosing my focus.
Potential Victims
Socially isolated older adults (55 to 75): Statistically overrepresented among romance-scam victims. Often recently widowed or divorced, active on Facebook specifically, and more likely to trust a connection that has developed over weeks. Average loss per victim in this segment approaches $60,000. High churn risk for Meta if they lose trust in the platform after being scammed.
Middle-aged adults navigating major life transitions (35 to 55): Divorced parents, people relocating for work, individuals recovering from grief. Active on both Facebook and Instagram. More digitally literate than the older segment but still vulnerable to extended emotional manipulation. Average loss around $30,000.
Younger adults drawn to crypto investment narratives (25 to 40): More likely to be contacted via Instagram. Already curious about alternative investments, which makes the investment pivot in a pig-butchering scam feel more plausible. Faster-moving conversations, shorter grooming periods. Average loss lower at around $12,000 but higher volume of targets.
Bad Actors
Organized fraud-compound operators: Running thousands of simultaneous scam relationships from Southeast Asian compounds, using scripted playbooks, and cycling through newly created Meta accounts as old ones are removed. Highly adaptive to detection.
Independent opportunistic scammers: Smaller scale, less scripted, often operating in a single language and targeting a specific regional audience. Less sophisticated but harder to detect because they lack the telltale behavioral consistency of scripted operations.
Recovery scammers: A secondary exploitation category targeting people who have already been victimized, posing as fund-recovery specialists. Often contact victims directly via Messenger after the original scammer has been removed.
I will focus the rest of this answer on the socially isolated older adult victim segment, because they represent the highest average financial loss per incident, the greatest vulnerability to extended emotional manipulation, and the highest potential trust-damage to Meta if the platform fails to protect them. Protecting this segment also generates the most meaningful regulatory and media goodwill.
