Editorial note: This week, I plan to write three articles about recent research from the National Bureau of Economic Research. This is the third.

Policymaking is hard. I have experienced this firsthand when working in a local legislature and in my work as a Uniform Law Commissioner. Policymaking is hard because it inevitably involves tradeoffs, and often those tradeoffs occur under conditions of deep uncertainty. Policymaking is also hard because politics is hard. Two recent working papers examining implementation of well-intentioned technology interventions in public-welfare programs demonstrate the challenge.

In the first, Is the Cure Worse than the Disease? Unintended Effects of Payment Reform in a Quantity-based Transfer Program, the author examines a policy choice to implement Electronic Benefit Transfer (EBT) instead of paper vouchers for the Special Supplemental Nutrition Program for Women, Infants and Children (WIC). The significance of the WIC program is hard to overstate: WIC serves about half of all infants born in the United States. EBT was implemented as part of a modernization effort of the program, a “digitization” project.

The policy change for WIC had good intentions: to provide oversight of the foods purchased under the program and to prevent grocery stores from engaging in price discrimination against WIC recipients. But, the author found, the actual policy change had some unintended consequences: “the reform was successful in reducing price discrimination, but also … it increased store dropout, reduced take-up among eligible pregnant women, and caused some non-WIC shoppers to pay higher prices.”

It seems straightforward that you’d want both: less price discrimination and greater take-up among intended beneficiaries. But, in fact, there’s a tradeoff; you just don’t know how significant that tradeoff will be at the outset.

In the second paper, Identity Verification Standards in Welfare Programs: Experimental Evidence from India, the authors ran a “large-scale experiment randomized across 15 million beneficiaries to evaluate the effects of more stringent ID requirements based on biometric authentication on the delivery of India’s largest social protection program (subsidized food) in the state of Jharkhand.” Here, too, the policy change had good intentions. Under India’s policy, food was directly subsidized by the state, and a substantial amount of that publicly subsidized food was diverted to individuals who do not qualify for aid. This problem of “leakage” was significant; the authors wrote that the system had “nation-wide estimated rates of leakage at 42% as of 2011-2012.”

The authors’ conclusions were, on one hand disappointing: “authentication of transactions had no measurable benefit; it slightly increased mean transaction costs for beneficiaries, excluded from their benefits a minority who did not have ID’s ‘seeded’ to their ration cards at baseline, and did not reduce leakage.” In other words, the intervention made things worse. That said, a subsequent intervention (“reconciliation”) that was dependent on authentication had a positive impact: “[w]hen paired with the new reconciliation protocols, authentication appears to have substantially cut leakage but at the cost of concurrent reductions in mean value received by legitimate beneficiaries.” So, perhaps it worked? But at what cost?

Answering that question has a much broader implication:

The comparison of these results with our own prior work on the impacts of biometric authentication in a public employment program and a pension program in the state of Andhra Pradesh (AP), is especially illustrative. A key point to note is that both programs reduced leakage. However, in the case of AP, the reduced leakage was passed on to the beneficiaries in terms of more money received (displaced from corrupt intermediaries), while there were no savings to government. In contrast, in the case of Jharkhand, the reduced leakage…led to reduced disbursals from the government, but did not improve the beneficiary experience in any way (and worsened it in some ways). In other words, the technology of biometric authentication “worked” in both settings in terms of reducing leakage. But the question of how the benefits of this leakage reduction should be shared between the government and beneficiaries is ultimately a design question and also a political one. Thus, the biggest reason for the difference in results (in our assessment) was not because of the technology (Smartcards vs. Aadhaar) or the context (AP vs. Jharkhand) but because of differences in program design. AP emphasized the beneficiary experience, whereas Jharkhand (implementing the policy decision of the Government of India) emphasized fiscal savings - and the results are consistent with this difference in emphasis.

In other words, even when a policy “works,” meaningful evaluation of the effect of a policy requires careful evaluation of “who benefits” under that policy.

So, what lessons can we draw from these studies? I draw at least two. First, we can gain a deeper appreciation for the uncertainty inherent in policymaking. In some instances, the tradeoffs may be knowable but hard to estimate. In other instances, the tradeoffs may be unpredictable. Second, we can remind ourselves that evaluation of policymaking is hard precisely because of politics. Technology solutions and context matter, but program design nearly inevitably requires political tradeoffs.

Like I said: policymaking is hard. Tech can’t change that.