Full text search in PostgreSQL is pretty great!

Recently, though, I found a bug in my application that perplexed me. When I searched for the word “Oddball” in the app, there were no results even though there should have been an entry with Oddball in it. Making matters worse, when I ran the search on my computer in dev, it worked!

After an inordinate amount of time being confused, I discovered that the root cause of the problem was that the default language in PostgreSQL differed from my dev computer and the prod environment. In dev, the language was set topg_catalog.english but in prod (running in AWS RDS), the language defaults topg_catalog.simple.

As a result, the “lexemes” that were stored in the database were different from the search query. For example, when I ranSELECT * FROM ts_debug('english', 'oddball'); the lexeme was{oddbal} (note: only onel). But when I ranSELECT * FROM ts_debug('simple','oddball');, the lexeme can back with{oddball} (twols). That caused the bug.

But thesolution required a bit more work because RDS doesn’t let you adjust the settings as easily as you might locally. Ultimately, to solve it, I needed to adjust thedefault_text_search_config parameter. If, however, you simply ranSET default_text_search_config = 'pg_catalog.english; it wouldn’t save it. As soon as you exited the session, the config would revert. Most publicly posted solutions emphasized adjusting the PostgreSQL configuration files, but because it’s RDS, you can’t really do that.

In the end, the solution required just one more step: granting theuser the ability to adjust the public schema and then altering the user’srole to set the default language.

Here’s the code:

GRANTUSAGEONSCHEMApublicTOdbuser;ALTERROLEdbuserSETdefault_text_search_config='pg_catalog.english';

Ultimately, this fixed the problem for me and was a simple fix. Because I spent way too much time googling for an answer, though, hopefully this post will help someone else identify the problem and get to a solution a bit faster.

I have a dataframe that has contains duplicates. And although I wanted to get rid of many of the duplicates, Ialso wanted to keep some duplicates on certain conditions. In this post, I outline the motivating problem and the solution (Note: I’m usingpolars here but you could use pandas/etc for your dataframe).

The motivating problem

To illustrate, here’s a random dataframe of foods, their categories, and some “last_date” field.

What Iwant in this example here is to keep both lettuces but discard the oldest fruit. In other words, I want my final table to look like this:

See how there are two fewer rows? The second banana and the second salmon entries drop off, but the second lettuce entry stays.

How can you do that?

The way to reason about this is that you’re going to (1) split the dataframe into two separate dataframes, (2) do your deduplication on one of them, and then (3) concatenate them into a new dataframe. Here’s some code:

# Sort the dataframedf_sorted=df.sort(["last_date"],descending=True)# Split the dataframe into two dataframes, one with vegetables and one withoutdf_vegetables=df_sorted.filter(df_sorted["food_type"]=="vegetable")df_others=df_sorted.filter(df_sorted["food_type"]!="vegetable")# Drop the duplicates from the dataframe that does not have vegetables in itdf_others_latest=df_others.unique(subset=["food"])# Combine the dataframes again, now with the dupes droppedfinal_df=pl.concat([df_vegetables,df_others_latest])

And voila, a conditionally deduped dataframe.

Is there a better way to do this? Perhaps! But it worked for me!

AI Disclosure

ChatGPT helped me think through a solution here when I got stuck. Hopefully this is the best way to do it, but if there’s a better way, please let me know and I’ll update the blog with a better solution.

I have a case where a bunch of CSVs are stored together in a zip file and I want to convert those CSVs into a parquet file. I’m usingpolars because it has an awesome ability to lazily read CSVs and then efficiently sink to parquet. It’s actually kind of magical.

But, there’s a problem. Because the CSVs are in a zipfile, you hit a snag pretty quick. That’s because you can’t just pass the CSV file name to thescan_csv function. The following code willnot work!

importpolarsasplwithzip_file.open("csv_in_zipfolder.csv")ascsv_file:pl.scan_csv(csv_file).sink_parquet("my_new_file.parquet")

That’s becausecsv_file is actually aZipExtFile, and thescan_csv function can’t accept that! According to thepola.rs API documentation,scan_csv only accepts a path to a file. Unlike theread_csv function, which accepts a pathor a file-like objects,scan_csv does not allow file-like objects.

This also means that attempting to download from a URL directly intoscan_csv won’t work either. Bummer, right?

But, there’s a hack if your csv file will fit in memory*: write it to a temporary named file and then pass that temporary named file to thescan_csv function. Here’s how that looks:

importpolarsasplwithzip_file.open("csv_in_zipfolder.csv")ascsv_file:# Create the temporary filewithtempfile.NamedTemporaryFile()astf:tf.write(fp.read())# Write the csv file to the temporary filetf.seek(0)# Start at the beginning of the temporary filepl.scan_csv(tf.name).sink_parquet("my_new_file.parquet")

By saving the file-like object into the temporary file system as a temporary file, you can happily pass the path to that file to polars and scan to your heart’s content.

* Technically, you could even iterate over the lines of your CSV file if it doesn’t fit into memory at all.

Recently, I was playing with a rather large dataset usingpandas and trying to improve the performance of my code. While reaching for themultiprocessing library, I learned about one small way to improve performanceand improve the readability of my code: usemap instead ofapply.

Let’s look at some code:

importnumpyasnpimportpandasaspd# Let's create a random dataframedf=pd.DataFrame(np.random.randint(0,100,size=(10,3)),columns=list('ABC'))# Normal approach: Use .apply() to iterate through the rowsdf["D"]=df.apply(lambdax:x["A"]**2,axis=1)# The new approach: Use python map() to iterate through the rowsdefpower(x):returnx**2df["E"]=list(map(power,df["A"]))

The new code is admittedly more verbose (requires three lines instead of one), but there are two advantages of this. Let’s start with the advantage that motivated it.

Adding multiprocessing is trivial with this pattern

Now, instead of trying to figure out where / how you’re going to handle multiprocessing in your code, you can simply just replacemap withpool.map. Let’s see it in action.

importmultiprocessing# Pro tip... you're going to want to make sure this has one of those# `if __name__ == "__main__":` things in front if you're using# multiprocessing.withmultiprocessing.Pool()aspool:df["F"]=list(pool.map(power,df["A"]))

See? Just a slight tweak in the code and suddenly you’re using all the cores.

You get a performance boost without any multiprocessing

This one kind of surprised me, tbh. I tried to see whether, without any multiprocessing, I’d get a speed bump

%timeitdf["D"]=df.apply(lambdax:x["A"]**2,axis=1)# 153 µs ± 951 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)%timeitdf["E"]=list(map(power,df["A"]))# 30.7 µs ± 119 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

Sure enough: a5x speed increase!

I’m not honestly surewhy this happens; it’s probably some weird thing where python does it better than pandas. But hey, a 5x improvement is notable!

Have fun mapping!

I created a Django app that runs on my computer. Most of the time, I don’t really think about running it as a background service because it’s not a production application. But I decided recently that Iactually wanted it to be a background service on my Apple computer. And even though I’m not sure this is theright way to do it, I’m here to tell you it works!

First, you create a.plist file, let’s call itcom.vdavez.app.service.plist.

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"><plistversion="1.0"><dict><key>RunAtLoad</key><true/><key>KeepAlive</key><true/><key>Label</key><string>Dave's Django App Agent</string><key>ProgramArguments</key><array><string>/Users/vdavez/.local/bin/poetry</string><string>run</string><string>gunicorn</string><string>myapp:app</string></array><key>WorkingDirectory</key><string>/Users/vdavez/app</string></dict></plist>

The key things to pay attention to here are theProgramArguments and theWorkingDirectory tags. Within ProgramArguments are the script you want to run (taking care to have the first string be an absolute path). Then, WorkingDirectory is where you want the script to run.

I’m using poetry to manage the virtual environment, so it’s just callingpoetry run gunicorn myapp:app as if I were doing it in the appropriate directory.

Next, save this file in~/Library/LaunchAgents, and runlaunchctl load com.vdavez.app.service.plist. Now, it just works!

Hope someone finds this useful background material. 🥁

What do you think of when you see the word “computer”? Do you think of a human?

In the 21st century, the answer is obviously not; a computer is decidedlynot a human. But at the beginning of the 20th century, you would. And, likely,you’d think of a woman:

In the mid-1950s, Kathleen Wicker received an assignment at work with one of the first electronic computers at the National Aeronautics and Space Administration (NASA). “At the time, I hardly knew what a computer was,” she recalled in an interview 40 years later. “I thought it was a woman.”

Wicker’s statement might seem surprising today, but in the mid-twentieth century, the term “computer” referred to people, not machines. It was a job title, describing someone who performed mathematical equations and calculations, and NASA employed hundreds of women as “human computers” in this era.

So, in just about 50 years, the use of the word “computer” came to mean the opposite of a human.

What about the word “robot”? Do you think of a human?

In the 21st century among English speakers, the answer is again decidedlynot a human. This, too, was a20th century innovation:

Robot is drawn from an old Church Slavonic word, robota, for “servitude,” “forced labor” or “drudgery.” The word, which also has cognates in German, Russian, Polish and Czech, was a product of the central European system of serfdom by which a tenant’s rent was paid for in forced labor or service.

Today, in Russian, рабочий (pronounced “rabochee”) means “worker” and работа (pronounced “rabota”) means “work”. So, how did this eastern European word referring to manual labor (specifically serfs) come to mean the opposite?

It owes its origin to “a brilliant Czech playwright, novelist and journalist named Karel Čapek (1880-1938) who introduced it in his 1920 hit play, R.U.R., or Rossum’s Universal Robots.”

Taking its cues from other literary accounts of scientifically created life forms such as Mary Shelley’s classic Frankenstein and the Yiddish-Czech legend The Golem,R.U.R. tells the story of a company using the latest biology, chemistry and physiology to mass produce workers who “lack nothing but a soul.” The robots perform all the work that humans preferred not to do and, soon, the company is inundated with orders. In early drafts of his play, Čapek named these creatureslabori, after the Latinat root forlabor, but worried that the term sounded too “bookish.” At the suggestion of his brother, Josef, Čapek ultimately opted forroboti, or in English, robots.

In the play’s final act, the robots revolt against their human creators. After killing most of the people living on the planet, the robots realize they need humans because none of them can figure out the means to manufacture more robots a secret that dies out with the last human being. In the end, there is a deus ex machina moment, when two robots somehow acquire the human traits of love and compassion and go off into the sunset to make the world anew.

Audiences loved the play across Europe and the United States.

In 100 years, robots went from human laborers at the bottom of the social order to being literally inhuman.

Beyond being etymologically interesting that the words “computer” and “robot” have had their meanings inverted, I can’t help but wonder about the throughline: that the altered words had the effect of erasing and dehumanizing the labor of women and workers.

I also can’t help but wonder whether the early 20th century fear of a robot rebellion in the United States echoes the late 18th century fears of slave revolts in the United States.

All of which raises the question of what words do we use today to refer to humans that will mean the opposite 100 years from now? For example, consider modern-day anxieties around artificial intelligence. Already we have seen how “neural networks”, which just a few years ago would automatically evoke human brains, have become commonly associated with machine learning and automation.

It’s also worth considering what an alternative history might have been where, instead of the language of technology being deployed to erase the contributions of women and workers and replace humans, our language evolved in a way that elevated their humanity.

In writing this, I wish I had a slightly stronger thesis about what it is about our culture that caused these two specific inversions, and a stronger suggestion about how to properly think about them. But in the meantime, it’s worth reflecting on the power of words, even as they change over time.

At the onset of World War II, the US government established asystem of rationing foods and supplies. As part of that system, individuals received “War Ration Books” with stamps that could be exchanged for rationed goods.

The other day, a family member discovered a handful of these books in another family member’s home and I was struck by the instructions on the back of one.

On the top, the book has straightforward, directive prose:

1. This book is valuable. Do not lose it.
2. Each stamp authorizes you to purchase rationed goods in the quantities and at the times designated by the Office of Price Administration. Without the stamps you will be unable to purchase those goods.
3. Detailed instructions concerning the use of the book and the stamps will be issued. Watch for those instructions so that you will know how to use your book and stamps. Your Local War Price and Rationing Board can give you full information.
4. Do not throw this book away when all of the stamps have been used, or when the time for their use has expired. You may be required to present this book when you apply for subsequent books.

On the bottom, the book borders on propaganda:

Rationing is a vital part of your country’s war effort. Any attempt to violate the rules is an effort to deny someone his share and will create hardship and help the enemy.

This book is your Government’s assurance of your right to buy your fair share of certain goods made scarce by war. Price ceilings have also been established for your protection. Dealers must post these prices conspicuously. Don’t pay more.

Give your whole support to rationing and thereby conserve our vital goods.

Be guided by the rule: “If you don’t need it, DON’T BUY IT.”

I’m fascinated by these instructions because they seem to be an excellent example of the use ofplain language in government.

The instructions write for the audience. These books were intended for distribution to the general public (thoughwomen were the primary users) and used simple, straightforward words and short, concise sentences (e.g., it has a Flesch-Kincaid Grade Level of 5.7). It pulls the reader in quickly (“This book is valuable.”). There’s no jargon here (though I could quibble about whether the Office of Price Administration needed to be named specifically). There are no frills.

The instructions are organized well, using a numbered list at the top, and prose near the bottom. It uses strong typography, mixing use of italics and capitalization to highlight the main points.

And the instructions tell a story. War Ration Books, which were fundamentally about limiting people’s ability to access basic commodities, were framed around language of civic duty, fairness, and service. The instructions directly connected the use of the stamps to the war effort.

In short, these instructions are an interesting historic example of the government taking plain language seriously and how good content design can make a significant impact.

As we enter the new year, it’s time to think abouthabits! As I’ve written before, “habits are a product of the environment in which you operate. And if you make your habits part of that environment, they’re more likely to stick.”

With this in mind, I created a new command-line interface application calledhabits.sh. It’s extremely simple; you configure a few things you want to track every day, run an initialization script, and then, every morning, you can get prompts to track those habits. Over the course of the day, I update my tracker with a single command: “habits”!

I know there are probably hundreds (thousands?) of similar applications out there. But thinking a bit deeper about how Ipersonally wanted to track my habits allowed me to build more of the environment I wanted to see in the world.

Do you track your habits? What do you track? Do you use something like this? What would make this better? Let me know what you think!

Here’s a fun experiment. Suppose you have a CSV and want to export it into JSON. Obviously, one way you could do it is through a purpose-built tool. But did you know you can use SQLite directly? Let’s try it.

Suppose you have the following CSV saved (cleverly) as “in.csv”:

name,ageJane Doe,25William Shakespeare, 40

You can run the following script at the command line:

123456

sqlite3 /tmp/test.db\'.mode csv'\'.import in.csv temp'\'.mode json'\'.once out.json'\'SELECT * from temp;'

And out you’ll get this saved in “out.json”!

[{"name":"Jane Doe","age":"25"},{"name":"William Shakespeare","age":" 40"}]

What’s happening here? Well, a couple of really cool things! You’re exploiting the.import function of sqlite3 that allows importing csv files into sqlite. But even cooler, you’re exploiting theoutput formats of SQLite by exporting to JSON.

You can even omit line 5 and then go straight intojq or whatever format you’re messing with!

sqlite3 /tmp/test.db\'.mode json'\'SELECT * from temp;'| jq .

As a bonus, you can export into markdown, a table, or even HTML by changing the mode. Neat, huh?

sqlite3 /tmp/test.db\'.mode markdown'\'SELECT * from temp;'

Hat tip tothis answer on Stack Overflow for teaching me the basic trick!

TIL!

There are stories in the data. You just have to know how to read them.

So opens Dan Bouk’s “Democracy’s Data: The Hidden Stories about the U.S. Census and How to Read Them.” The book hasreceived incredibly positive reviews and for good reason: it’s beautifully written and insightful scholarship.

Bouk claims to specialize in studying“modern things shrouded in cloaks of boringness”. And he’s clearly very good at it.

And yet, convincing you to read Bouk’s book is not the point of this post. Instead, I wanted to share that Bouk’s book caused me to reconsider a concept that is pretty familiar in the world of data: “data quality”.

Data quality is the sort of thing that people who deal with data think about a lot and talk a lot about. We’ll observe that, when making decisions based on data, we need to make sure that we have “good data”. Bad data, on the other hand, causes people to knowingly observe “garbage in, garbage out”.

In the preface of his book, Bouk makes a statement that (at first blush) runs in the same direction about data quality: “what is our democracy, if this is its data?” As I started on the book, I imagined: “yes, surely the book will reveal how an overconfident census leads to all kinds of political problems. Garbage in, garbage out, right?” And, sure, I guess there’s some of that in there.

But, readingDemocracy’s Data offers an alternative framing: behind any data are stories. When you read data, if you read it deeply and with dignity, you can learn greater truths about the motivations and goals across the data lifecycle.

Bouk’s effort largely returns to the “doorstep interactions” of the decennial census and he described all of the challenges of capturing, classifying, and reporting the number of people in our country and facts about them. Issues of race, gender, politics, and more suffuse the “simple” counting of people on a given day. But, by reading in the data (and what was left out of the data), we better understand the complexities and richness of our nation. We see how people imagined and contributed to a better world, lived through fear and struggle, created beauty and art. We see how the “mistakes” of the counts – and the omissions from the counts – revealed and reflected people more accurately.

This book taught me, therefore, to remove the phrase “garbage in, garbage out” from my vocabulary when talking about data. Instead, when confronted with the inevitable “data quality” issues that affect any organization, I will try and approach the discussion with greater curiosity about the question: “what is this organization, if this is its data?”

Data, as it happens, is never really garbage. Rather, looking for the stories behind the data allows questions of data quality to be an entry point to study the motivations and actions of the collectors, the processors, the managers, the publishers. It may be messy (just like democracy), but only by better understanding it can we truly make progress.

Chances are, if you live in the United States, the buildings on one side of the street are all even or all odd, and the buildings on the opposite side of the street are the opposite.

Butwhy is this the case? How did this come to pass?

As it turns out, you can trace this convention back toArticle I, Section 2, Clause 3 (the “Enumeration Clause”) of the US Constitution, which established the United States Census, and to Philadelphia, which was the nation’s capital during the first national census!

That’s right! The odd/even house numbering convention owes its origin to the founding of the US constitution!

While reading Dan Bouk’sDemocracy’s Data: The Hidden Stories in the U.S. Census and How to Read Them, I came across this assertion and checked out his citation and then I went one step further to learn more. Here’s the source of that assertion,a doctoral thesis entitled “Governmentality, the Grid, and the Beginnings of a Critical Spatial History of the Geo-Coded World”:

During the 1790s, the practice of placing odd and even numbers along opposite sides of the street became common.The U.S. Marshal who conducted the first federal census in Philadelphia (1790), Clement Biddle, renumbered the city’s houses and separated odd and even numbers on different sides of the street. Interestingly, Biddle (1791) then published his own city directory based upon this new numbering scheme.

(emphasis added). Shortly after Philadelphia did it, other cities (like New York City) followed suit.

The full thesis on “house numbering” is actually a super interesting read. He argues that “as a spatial practice, house numbering is a comparatively recent innovation, which did not become widespread until the second half of the eighteenth century, and in some cities the practice was not systematically adopted until well into the nineteenth century.”

So if numbering didn’t start until the eighteenth century, why did house numbering start in the first place?

The emergence of house numbering as a spatial practice cannot solely been attributed to a single factor. Rather, there were multiple forces at work. On the one hand, centralized military control is a key element in explaining the rise of house numbering in France (with the exception of Paris). Equally important was the use of house numbers for civil administration in urban centers. The practice of administratively affixing individuals to specific numerical addresses enabled the government to more easily tax its population, facilitated the conducting of a national census, and also served as the administrative basis for the commodification of space.

TIL!

During the oral arguments inStudents for Fair Admissions, Inc. v. Presidents and Fellows of Harvard College, the case where the Supreme Court is highly likely to end affirmative action in education as we know it, Chief Justice John Roberts stated “We did not fight a Civil War about oboe players. We did fight a Civil War to eliminate racial discrimination, and that’s why it’s a matter of – of considerable concern.” Implied in this question is a historical claim the Fourteenth Amendment to the United States constitution was adopted to create a “color-blind constitution.”

That argument of a color-blind constitution was more explicit by the attorney for those challenging Harvard’s policies:

In terms of the original meaning of the Fourteenth Amendment, the best source on this I’ve ever read is the United States’ brief on reargument in Brown. It painstakingly details the legislative history and how the framers of the Fourteenth Amendment saw it as a ban on all racial classifications. Also, the – everyone knows that the impetus for the Fourteenth Amendment was to constitutionalize the Civil Rights Act of 1866. The Civil Rights Act of 1866 is a series of bans on racial discrimination. It’s a series of color-blind measures and requirements.

But are these points correct? Did we fight a Civil War to eliminate racial discrimination? Was the purpose of the Fourteenth Amendment to establish a “color-blind constitution”? Is it true that “everyone knows” that the Fourteenth Amendment was trying to constitutionalize the Civil Rights Act of 1866?

History, as it turns out, is more complicated than the advocates suggest.

Indeed, when reading theThe Second Founding, the latest book by Eric Foner, who is one of the leading historians of the Reconstruction Era, you quickly discover that the Thirteenth, Fourteenth, and Fifteenth amendments represent a deeply complex soup of goals. As he writes:

For the historian, seeking to understand the purposes of the Reconstruction amendments is not the same as attempting to identify, as a matter of jurisprudence, the “original intent” of those who drafted and voted on them or the original meaning of the language used. Whether the courts should base decisions on “originalism” is a political, not a historical, question. But no historian believes that any important document possesses a single intent or meaning. Numerous motives inspired the constitutional amendments, including genuine idealism, the desire to secure permanently the North’s victory in the Civil War, and partisan advantage.

Foner’s work reveals how the framers of the Reconstruction amendments contemplated radical departures from the constitutional framework that preceded it, and how those expectations were shattered over time. Again, as Foner writes:

In the chapters that follow my purpose is not so much to identify the one “true” intent of the Reconstruction amendments, as to identify the range of ideas that contributed to the second founding; to explore the rapid evolution of thinking in which previously distinct categories of natural, civil, political, and social rights merged into a more diffuse, more modern idea of citizens’ rights that included most or all of them; and to suggest that more robust interpretations of the amendments are possible, as plausible, if not more so, in terms of the historical record, than how the Supreme Court has in fact construed them.

The book covers a lot of ground in a relatively quick read. And, for me anyway, Foner accomplished his mission of helping illuminate the wide diversity of perspectives and goals in the framers of the Reconstruction amendments.

For example, although modern readers of the Fourteenth Amendment predominantly focus on section one (or in these unusual times, section three), few focus on section two. That’s not especially surprising given that the section has never been enforced.

At the time, however, what is now section two was the primary focus of the committee considering constitutional amendments. As Foner writes:

The first version of a Fourteenth Amendment to emerge from the committee was an attempt to finesse the black suffrage issue while dealing with an ironic political consequence of the abolition of slavery. Now that all blacks were free, the Constitution’s Three-Fifths Clause became inoperative. In the next reapportionment allocating membership in the House of Representatives and votes in the Electoral College, all blacks would be counted as part of each state’s population. The southern states would thus enjoy added representation, giving them, as one congressman put it, “an undue and unjust amount of political power in the government.”

An ironic political consequence, indeed, that the Republican project of abolishing slavery would favor southern Democrats in a future Congress and White House. But actuallysolving for that irony created other problems for Republican drafters:

Seventeen proposals to restructure congressional representation came before the Joint Committee. The simplest way of dealing with this problem, Radicals insisted, was to require the states to enfranchise black men. This would ensure that the Slave Power would no longer control southern politics. Moderates, however, believed such an amendment would never secure ratification. Another option was to base representation on voters, not total population, as [Rep. Thaddeus] Stevens had proposed. This would leave suffrage requirements in the hands of the states. It would encourage the Johnson governments to enfranchise their black populations or leave those states with reduced power in Washington (a loss of one-third of their congressmen according to one estimate). But as Representative James G. Blaine of Maine pointed out early in January 1866, western migration was skewed toward men, and thus basing representation on voters would result in a shift of power away from eastern states, which had a higher percentage of women in their populations. The proposal, Blaine warned, might also unleash an “unseemly scramble for voters,” including the enfranchisement of women, which would double a state’s representation in Congress.

In other words (and unsurprising to the modern observer), members of Congress were keenly focused on future election cycles and the Fourteenth Amendment was (in part) an effort to maintain a structural advantage for the Republicans. And the compromise they settled on reflected the prejudices of the time, the idealism of reformers, and the realpolitik calculus of what could get enacted and ratified.

In the end, as Foner notes, it is a political question about whether, legally, itmatters what the authors of the Reconstruction Era thought they were doing when they engaged in a “constitutional revolution” by passing the Thirteenth, Fourteenth, and Fifteenth amendments.

From a purely historical perspective, though, his book reveals just how messy progress can look. In the moment, and even 150 years later.

Years ago, Iargued that lawyers should learn about writing software because doing so could improve the interactions between lawyers and technologists. As I wrote:

Ultimately, lawyers and non-lawyers must learn to talk to each other. To collaborate with each other. To tackle society’s challenges together. And that requires empathy for each other’s domain expertise.

Over the past six years, it is largely unsurprising that lawyers have not stepped up to the plate. More surprising, though, is that government institutions have increasingly created formal paths for technologists to answer the call.

In my own experience, I have seen tech fellowship programs succeed in bothlocal government, and in both branches ofthefederalgovernment. Which is why I wasthrilled to see the creation of theJudicial Innovation Fellowship program at Georgetown Law’s Tech Institute.

Readers of these pages know that trial courts at the state, local, territorial, and tribal (SLTT) levels are vital to the basic fabric of democracy in the United States. And, unfortunately, SLTT trial courts are typically structurally underfunded and rely on outmoded tools and technologies.

As the program explains:

Partnering with state and tribal courts to build critical data infrastructure, simplify process, and improve usability of court services, this competitive fellowship is a unique opportunity to innovate a core democratic institution. More than just a job with a competitive salary and benefits, this fellowship is the flagship opportunity to change the way people access their rights and are served by courts.

On first blush, these goals may sound lofty. It’s just a one-year fellowship, right?

Over the years, though, a pattern with government tech fellowship programs has emerged. And I, for one, am hopeful to see it repeat itself here. That pattern? People join these fellowships expecting to stay for a year or two and then return to the private sector. But that expectation is quickly shattered when they realize the potential impact they can have and the level of professional satisfaction they can enjoy as public-servant technologists.

For example, over 50% of the Presidential Innovation Fellows end up staying in government roles after graduating from the program. By my read of the description, the PIF appears to be the closest analogue to the Judicial Innovation Fellowship program. If even a quarter of the technologists who join the courts as innovation fellows end up staying, our democracy will be better off.

I still believe that lawyers should put in effort to learn how technology works. In the mean time, though, we are fortunate that government and related institutions are stepping to create space for technologists to fill an access-to-justice gap.

If you work in a SLTT trial court, or are a technologist who cares about justice and democracy, I highly recommend youcheck out the program.