An RSS feed for LIMS
Syndicating data for better legislation tracking
Recently, I built an RSS feed for LIMS.
The URL for the RSS feed is here: https://esq.io/lims-rss.xml, and the source code is available here: https://gist.github.com/vzvenyach/757fa97fd99c3a14e798. This post explains my reasons for doing it.
What is LIMS?
One of the most important resources for tracking legislation in the District of Columbia is the Legislative Information Management System, known as “LIMS,” and found at http://lims.dccouncil.us. LIMS provides information on all legislation before the Council, and allows issue advocates, Council staff, and others to easily filter or search through the database to find legislation, sorted by type (bill, resolution, contract, etc.), status (pending, enacted, official law, etc.), and other criteria.
LIMS is a great resource for most users. And it is a huge improvement from earlier iterations. But, on its face, it's not ideal for web developers. For example, there's no documented API. There are few obvious URL endpoints. It uses certain web-design choices–such as sessions–that make scraping particularly difficult. And, relevant to this post, there's no RSS feed or sitemap. (If you don't know what any of that stuff means, that's ok.)
So what, you might be wondering? To answer that question, I need to describe a particular “use case” for LIMS: tracking new legislation. As General Counsel, I am responsible for tracking all legislation pending through the Council. In practice, there are different ways to doing this:
- Check the LIMS site every day.
- If you work at the Council, check your paper inbox every day.
- Check the DC Register every Friday for “Notices of Intent to Act.”
All of these are fine solutions, but they require you to actually do something. Repeatedly. And that's not great.
It would be better if there was a way to get automatic notifications of new legislation.
RSS and IFTTT
One way that web developers thought to address the problem of automatic notification of changing content was through the introduction of RSS. When a website includes an RSS feed, a user can get automatic updates of new and changed content on the site. You can use an RSS reader to keep track of all of the sites that you care about (I personally use Digg Reader, but many other options exist).
And, once you have an RSS feed, you do other cool things. For example, you can use IFTTT, which stands for “IF This Then That,” to get notifications in other places once you have an RSS feed. Personally, I have set up my RSS feed to send a text message to my phone anytime there is new legislation introduced at the Council. But you can imagine many other options. So, RSS and IFTTT would solve the problem of automatically tracking Council legislation.
As discussed above, LIMS does not offer RSS feeds. And even though it is not well-documented, the underlying architecture of LIMS allows it to be developer friendly. Specifically, it has a “hidden” API, which allows a user to send an HTTP request to the server, and receive a JSON response. [At some point, I'll set out to actually document the API.]
Using this hidden API, you can get data about all of the laws that LIMS has identified as being “New” (LIMS explains that all legislation introduced in the last 7 days is considered “new”). Once I accessed this information, I wrote a Python script to convert that data into an XML file, using the Atom Syndication Format, to “look” like an RSS feed. Then, I set up a “cron” job to tell the computer to do this every hour, so that any new information on LIMS would be included in the XML file, thus making it a functional RSS feed.
Obviously, some of this might look like wizardry. And, to a certain extent, it is not the most basic of programming. But, it's also not too hard once you have some comfort with Python and web development.
The whole thing is only 27 lines of code, but it will make sure that I never have to bother going to LIMS every day, or checking the Register, or looking for things in my paper inbox. I'll just get a text message anytime there's a new measure introduced in the Council. And, if you set up an IFTTT recipe, you can too.
Finally, if you want to follow this blog, the link to the RSS feed is: https://esq.io/feed.xml. Thanks!