This is not a stereotypical post about my artwork, instead it’s a post about appreciation of some of my open source work!
Over the past couple of years I have been lucky enough to be apart of some some amazing projects and meet some really cool people, and its funny how you never know where it will end up taking you.
In 2014, the open source project that is now known as Scrapy Cluster began as a trial to determine the capability to do open source distributed web crawling. There were many different projects that claimed to do web crawling like we needed to do, but none really seemed to fit the style of architecture that we needed.
Fast forward through years of internal use, and probably close to hundreds of deployments in virtual machines, servers, and customer applications, we end up where we are today with a project that is trying to lay the groundwork for a tried and tested distributed crawling framework built upon one of the most popular open source projects for web crawling, Scrapy.
While the git commit log shows primarily my account, in reality there are many great minds behind the facade of the log, and an awesome organization helping us advance it. After all, it is part of the Darpa Open Catalog, which spans many different programs, each of which have a large amount of open source work associated with them. With that comes the pleasure to share back with the world the breakthroughs, research, and trials that we have faced in order to mature and advance the project, in hopes that others could benefit from it.
Over the past 3 or so months I have been in contact with a pair of developers named Liam and Jeremiah who are trying to fit Scrapy Cluster for their use case. I am very happy to not only hear about our work being used in other ways, but to also be a bit involved in answering questions or giving recommendations on how to best utilize the project. It has been a really good learning experience and hopefully will make the project that much better.
Out of appreciation for my help, they decided to send me a “Thank you” I just received it last week!
Neat! A package with a bit of weight to it. But what is inside?
A note! But now what is inside the bubble wrap?
Whiskey! I can’t express how thankful I am for this, and am happy that the hard work we have been putting in for so long can be appreciated by others. This post is really just a giant Thank You to Liam and Jeremiah for the Maker’s Mark, and I am always happy to help as best I can those who are trying to use the project for their needs.
While I don’t know where the project will be in the future, and how much time both myself and everyone else who works on the project will have, I hope that it helps get us closer to a better web scraping world.
Cheers.
