A blog by Dave Zvenyach

Protest reform

How should we reform bid-protest procedures? [hint: we probably shouldn't]

Yesterday, Jen Pahlka offered a provocation:

My initial reaction was that “protests are fine.” And then Jen and I traded references to the Death Star. But, this morning I want to give a more considered answer. Fair warning, it's much longer than 140 characters.

tldr: the answer is that, if I could wave a magic wand, it would be for (1) better empirical research; (2) more honest dialogue and humility; and (3) design of a more fault-tolerant, human-centered procurement system.

Before I do, a little history. Every generation has its attempts to reform bid-protest procedures. In our current time, there are the Section 809 Panel recommendations. The panel recommended things like: changing jurisdiction thresholds and timelines at the GAO and Court of Federal Claims. The panel even considered a DOD-specific forum for bid protests.

But, twenty five years ago, the Administrative Conference of the United States also considered bid protests and options for reform. Specifically, in 1995, the ACUS published a report entitled Procurement Reform and the Choice of Forum in Bid Protest Disputes, which discussed some of the options around protest procedures. It is a good read if you're into procurement policy. [Also, fun fact, we should expect a future report on Agency-level Bid Protests if this RFP is any indication.]

Indeed, since 1887, and through the present day, bid protests have been a controversial part of the government contracts landscape. For the civic-tech-savvy readers, procurement nerds even share with UX researchers a common enemy in the Paperwork Reduction Act (the PRA made permanent the creation of the General Services Contract Appeals Board, which handles certain contracts disputes).

The reason I bring up this history is that, 25 years ago (and before that even), the problems were really the same as they are today. Policy suggestions about choice of forum, timelines, jurisdictional thresholds, burdens of proof, and scope are part of a long tradition of tweaking the system at the margins. But, as the ACUS report suggests, all of these recommendations derive from a more central question:

The most important policy decisions to be made about the protest status quo flow from the answer we give to the [question of how much discretion should government procurement officials be given to carry out their responsibilities]. The extent and form of protection given to offerors hinges on one's assessment of the incentives that guide purchasing agency officials. To address this fundamental issue we have distressingly weak empirical knowledge. There has been little systematic study about whether the modem protest system has improved the quality or reduced the cost of public procurement. A necessary first step toward evaluating the protest system is to perform empirical tests of the effect of the protest process. Such empirical tests should include an evaluation of past protest effects and experiments with more austere protest oversight.

In other words, how we feel about protests is largely driven by how much we trust contracting officers to do their jobs effectively. The reason I say feel and trust is that we still don't have very good data or evidence about whether protests affect purchasing behaviors or decisions. Anecdotally, protests affect acquisition strategy. But do we believe that the absence of protests would meaningfully affect evaluation decisions? Or would it simply change the means of documentation?

By contrast, as I wrote recently, there is evidence to suggest that greater autonomy in procurement might lead to better outcomes. But even that experiment has limited explanatory weight.

The best we have is anecdotal evidence from 2018. In response to a sudden congressional interest in bid protests, the RAND Corporation issued a report entitled Assessing Bid Protests of U.S. Department of Defense Procurements, which surveyed DOD contracting officers about their decisionmaking process. And its findings track my own experiences:

DoD and service contracting officers do not consider the prospect of receiving a bid protest to be a top priority as they develop RFPs. Their primary focus is on ensuring that requirements and evaluation criteria are clearly defined to minimize bid protests. When asked whether the fear of a bid protest would limit acquisition and contracting options, these personnel responded that they were not afraid of receiving a bid protest and that, in general, they believed that acquisition and contracting options were not being thwarted by the fear of bid protests.

However, contracting officers also noted that the possibility of a bid protest did affect the type of contract or contract vehicle they selected—usually prompting them to favor a price-related choice or existing task/delivery order–type contract if appropriate. They added that possible bid protests also affect the scrutiny that source-selection documentation receives from legal counsel, as well as the amount of time required to award the contract, which can result in programs missing key milestones or losing funds.

Do protests affect decisionmaking by the COs? Nope. Not really. No. Sure, it changes which contract vehicles they use, but doesn't really change what they buy or from whom.

And yet, protests really are just fine. As Professor Schooner, a scholar of public procurement law and policy, observed in his writeup of the RAND report: “One significant take-away from the study is that, although disappointed offerors do not always win their bid protests, they appear to protest, by and large, for legitimate business reasons—including, as noted above, poor agency debriefings.” True to form, Stan Soloway offered a view that tracks this perspective:

Where does that leave us? Government and industry don't trust each other, and the protest system we have in place both responds to this distrust and perpetuates it.

So, what is to be done? First, I think some research is in order. Rather than adopt a policy that attempts to tweak procurement policy based on our feelings, I would advocate for some controlled experiments. Perhaps we have some already with regard to IDIQ thresholds? I haven't examined it enough to know whether we could draw policy recommendations, but I'd sure be more excited about reforms grounded in data than my hunches about what contractors or contracting officers may or may not do.

Second, if I truly had a magic wand, I would make leaders of government and industry bury some hatchets and work to rebuild trust and open communication. As much as I love my old colleagues at GSA and in industry, the rhetoric we all too often hear in open discussions follows the same tropes. Industry wants better communication and documentation. Government says industry games the system. Absent a magic wand, I don't have great suggestions on what specific steps our institutions must to repair trust, but it needs to be a priority.

The third, and perhaps most complex of my thoughts, is that we need a more fault-tolerant system. Consider the fact cited by RAND that “in almost half of all procurement protests, GAO either sustained the protest or the agency took corrective action to fix a flaw in the procurement.” Half of the protested cases had mistakes. On the one hand, yikes. But on the other hand, to err is human nature. As Don Norman wrote in his classic text The Design of Everyday Things:

If the system lets you make the error, it is badly designed. And if the system induces you to make the error, then it is really badly designed. *** Why do people err? Because the designs focus upon the requirements of the system and the machines, and not upon the requirements of people. Most machines require precise commands and guidance, forcing people to enter numerical information perfectly. But people aren't very good at great precision. We frequently make errors when asked to type or write sequences of numbers or letters. This is well known: so why are machines still being designed that require such great precision, where pressing the wrong key can lead to horrendous results?

Is it any wonder that in the last quarter of the fiscal year, when contracting officers are slammed with trying to get contracts out the door that “mistakes were made?” Pop quiz: when do most protests come? If you guessed August, September, and October, you're right! Protests are a lagging indicator of the fiscal year.

If pushed to answer what I think is wrong with the bid-protest system, I'd probably argue that we treat all errors conceptually equally. Even though our system requires a showing of “prejudicial error” because corrective action, we inevitably conceptualize all errors as being “mistakes,” when they're often more likely “slips.” For a lawyer, it might not matter. In litigation, lawyers rightly focus on the impact of a slip on the contractor. But for managers of an acquisition system, neglecting a focus on what led to the error in the first place is a problem. Having multiple layers of reviews can't help when everyone is trying to cram to meet deadlines. Involving legal often makes timelines worse. Our current patterns of acquisition management just don't work very well at scale.

In the end, bid-protest reform would likely be better served by conceiving of it as a service-design problem. How might we ensure that contracting officers capture the decisions that are important for potential protestors as part of a debrief? How might we consistently write our evaluation criteria in a simpler, more explainable, more routinized manner? How might we reduce the number of internal reviews on contracts that don't actually need them so that we can free people up to more critically evaluate acquisition packages? [Here's a provocative one] How might we encourage the sort of risk taking that will speed up a procurement that is inevitably going to bid protest anyway?

You can see where this goes. Bid-protest reform, at bottom, requires answering the question of “how much discretion should government procurement officials be given to carry out their responsibilities.” To me, answering this turns on designing a system that places discretion where humans are good at this sort of decisionmaking, and removing it where humans are less capable. By focusing on “the requirements of people” rather than the “requirements of the system and the machines.” Bid protests, for all of their flaws, can be reframed as good signals for opportunities to improve.

In the end, my sense is that the best way to make procurement better is to strive to make it more human.