Hartley Brody

A few days ago, the Facebook Messenger team launched version 1.4 of their platform. The most prominent changes for most bot developers are the enhancements to the persistent menu options.

A lot of commentators noticed that the updates seem to steer bot developers away from building conversational, chat based interfaces. Indeed, there is now the ability for bot developers to completely disable the text composer for users, and requiring them to tap or click on buttons for every interaction.

Many people are already heralding this as the beginning of the end for chat botsif Facebook seems to be discouraging bot developers from using the primary chat interface, maybe chat bots aren’t really living up to the hype!

The truth is… right now, they’ve got a point.

Not all Facebook Messenger Bots are that great right now.

web-scraping-ebook-coverAfter four years, I figured that my book, The Ultimate Guide to Web Scraping was due for an upgrade.

The internet has changed, websites have gotten more advanced, and I’ve learned a lot of new skills and tactics over the years that I wanted to incorporate into the book.

I’ve also gotten a lot of feedback from some of the over 1,300 readers that helped to make the second edition of the book even more thorough and complete.

The book has been largely re-written and re-organized to focus on simple concepts, show you them in action, and then build on them for more advanced web scraping use cases.

There are over 40 new pages with many more python code samples as well as better coverage of more advanced topics. The book also includes a discount code for my online web scraping video course: Scrape This Site (a $50 value).

Once you’ve put together enough web scrapers, you start to feel like you can do it in your sleep. I’ve probably built hundreds of scrapers over the years for my own projects, as well as for clients and students in my web scraping course.

Occasionally though, I find myself referencing documentation or re-reading old code looking for snippets I can reuse. One of the students in my course suggested I put together a “cheat sheet” of commonly used code snippets and patterns for easy reference.

I decided to publish it publicly as well – as an organized set of easy-to-reference notes – in case they’re helpful to others.

Web Scraping Cheat Sheet Graphic

While it’s written primarily for people who are new to programming, I also hope that it’ll be helpful to those who already have a background in software or python, but who are looking to learn some web scraping fundamentals and concepts.

This past summer, I launched a web scraping class targeted at total beginners.

After talking to a lot of people who were having web scraping challenges over the past few years, I realized that many of them weren’t coders and didn’t have any technical backgrounds. Many people who were looking to learn web scraping had never coded before.

So, when I was making the material and lessons for the course, I had to keep this in the front of mind. I couldn’t expect students to know what a for-loop is or how to spot syntax errors in their text editors.

I really tried to hold students’ hands through all of the class material, writing the simplest possible code, explaining things line by line as I went, and intentionally running into common mistakes to show them how to power through.

Debugging Tips

But even with all of that assistance, the real world is a lot messier. Students would begin to stumble when they’d go out on their own. Their code “wouldn’t work” and they’d feel hopelessly stuck pretty quickly.

I realized that the process of troubleshooting and fixing the bugs in your code isn’t intuitive yet to anyone who hasn’t spent a long time learning to code already. So I decided I’d collect some of my most common tips for beginners who feel stuck when their code is giving errors or isn’t doing what they want it to do.

While these tips are generally geared towards beginners, many of them are actually the tips I still follow day-to-day when debugging issues with my own code. Even as someone who has been coding full time for over 7 years, I use these steps myself constantly (especially #9).