This is an adaption of an article I wrote on the engineering wiki at Buffer, where I work as a growth hacker.

The Zen of Python

Run import this from a python interpreter to see the official “zen of python”

• """
   Beautiful is better than ugly.
   Explicit is better than implicit.
   Simple is better than complex.
   Complex is better than complicated.
   Flat is better than nested.
   Sparse is better than dense.
   Readability counts.
   Special cases aren't special enough to break the rules.
   Although practicality beats purity.
   Errors should never pass silently.
   Unless explicitly silenced.
   In the face of ambiguity, refuse the temptation to guess.
   There should be one-- and preferably only one --obvious way to do it.
   Although that way may not be obvious at first unless you're Dutch.
   Now is better than never.
   Although never is often better than *right* now.
   If the implementation is hard to explain, it's a bad idea.
   If the implementation is easy to explain, it may be a good idea.
   Namespaces are one honking great idea -- let's do more of those!
  """
  

PEP8

tl;dr: Whatever IDE or text editor you’re using probably has a plugin that will notify you when the code you’re writing doesn’t match these styles. Installing one was the best decision I ever made as a python programmer. (Besides reading “Dive into Python“)

Ideally, there should be One Correct Way or writing a given statement or expression. If everyone decides their own conventions for spacing, variable naming, etc. then code becomes harder to read and maintain. The PEP8 style guide is the most commonly used convention for addressing this problem in python. Written by Guido van Rossum — the creator of python — it recommends the following conventions:

• Use 4 spaces for indentation, not tabs
• A single space after every comma and between every operator:

  # BAD
  v1,v2=0,0
  my_dict={"key1":1,"key2":2}
  
  # GOOD
  v1, v2 = 0, 0  # much easier to quickly scan and read
  my_dict = {"key1": 1, "key2": 2}
  

• Two blank lines between classes and top-level function definitions, one blank line between class methods.

  def function():
      pass
  
  
  class MyClass(object):
  
      def __init__(self, arg):
          super(MyClass, self).__init__()
          self.arg = arg
  
      def get_arg_offset(self):
          return self.arg * 10
  

• Imports should be on separate lines, and should be explicit:

  # BAD
  import sys, time, os
      
  # GOOD
  import sys
  import time
  import os
  
  # this is okay, multiple imports from same package
  from os import environ, chdir
  
  # NEVER do wildcard imports. Lots of badness ahead.
  from somereallybigpackage import *
  
  # Explicitly import the things you need. Avoids
  # collisions and makes later code way more readable.
  # Ie: "Where is this function defined? Oh, I see it 
  # was imported from somereallybigpackage"
  from somereallybigpackage import func1, func2, func3, class1, CONSTANT
  
  # If you really need everything, then just
  # import the module and use it as a namespace.
  import somereallybigpackage
  ...
  somereallybigpackage.func1()
  somereallybigpackage.func2()
  

• Variables are where you give your code meaning and context. Variable names should err on the side of being long and explicit. It may take a few extra seconds to type, but it saves other engineers hours of trying to figure out what you meant. If your variable names are shorter than five characters, they’re probably too short.
  P.S. They say one of the hardest problems in programming is naming things.

  # BAD, what do these variables mean?
  t = datetime.datetime.strptime(strdate, "%Y%m%d")
  ts = int(time.mktime(t.timetuple()))
  t2 = t - timedelta(1)
  ts2 = int(time.mktime(t2.timetuple()))
  
  # GOOD, very explicit
  current_cohort_user_query = build_cohort_query(joined_at__gte=current_cohort_month, joined_at__lt=next_cohort_month)
  current_cohort_users = UserMetrics.objects.raw_query(current_cohort_user_query)
  

• Comments can be super useful when there’s a lot of “magic” going on or if you’re worried that there’s a lot of logic that might be hard for a new reader to understand. But beware the out-of-date comment! Whenever you change code, be sure to change any nearby comments if they’re no longer true. Comments that contradict the code are worse than no comments.

  def function():
      """
      Any function that's longer than a few lines
      should have a docstring that explains, at minimum:
        1. what params the function takes, if any
        2. what it should be expected to return
      Docstrings should be immediately after the function
      definition and are usually multi-line comments that 
      use the triple-quote syntax, not hashtags.
      """
  
      # If you need to use in-line comments, you can either put them
      # above a big block of code, like this...
      while True:
         counter_var += 1  # or put them inline, with two spaces between the code and the hashtag
  
      # Note there's always a space between the hashtag and first letter of the comment.
  

Final Thought

It may seem like splitting hairs, especially if the code still functions regardless of whether it follows the style guide. But having clean, easy-to-read code is a huge investment in your code base and saves you from piling up mountains of technical debt down the line. If any engineer can open a file and quickly figure out exactly what’s happening and make edits, you become a much faster engineering team.

Every single citizen should know CPR and how to handle major blood loss events. If you don’t, please please read up and take a class. Spending a few hours learning these things now could give you the skills that end up saving the life of a loved one or a stranger.

It only takes a few hours to learn, and then you can go on with your life. But when a moment of potential tragedy catches you off-guard, you won’t become a passive, helpless observer. You’ll be a life saver. An unsuspecting hero.

Here are some excellent ways to get started:

The American Red Cross offers a class called “CPR, First Aid, and AED for Lay Responders.” It takes five hours and is an excellent way to spend a Saturday morning. Everyone should take this class, or a local equivalent.

NOLS Wilderness First Responder Course is probably the most comprehensive first aid course you can take, short of becoming an EMT. These classes usually take a solid week and are focused on backcountry first aid, when getting someone to a hospital isn’t an immediate option. This is more for people who find themselves in remote situations.

Many larger schools offer student-run ambulance associations which you could either join or take classes with.

Read articles like this one on How to Use a Tourniquet or some of the questions on the First Aid tag on Quora to build out more of an intuition for how our bodies work and things to be aware of when administering first aid.

I’ve seen lots of people posting statuses about wanting to help in some way, maybe donating blood. My advice: take a first aid class as soon as you can. God forbid you ever need to use your training, you’ll have the opportunity to be someone’s hero.

-

While getting people into your product is obviously A Very Good Thing, it isn’t time to pat yourself on the back just yet. To build an awesome product experience and an engaged user base, you really need to carefully focus on what happens after a user has signed up.

Once they’re in the door, you want to show them value quickly. Help them avoid common pitfalls and show them how to get the most of your product. Walk them through the process of moving from beginner to power user to evangelist.

Not only does this make for happy users, but it also make it easy to ask for referrals, product reviews and up-sells, while simultaneously avoiding churn and abandonment. That all sounds pretty awesome, right?

I’ll walk you through a few basic ideas and then we’ll look at how some of the hottest startups have built out their post sign up funnels.

What Is All This Talk About Funnels?

The concept of “the funnel” is very common in business-to-business (B2B) marketing, where potential customers make slower, deliberate decisions — often taking weeks or months — before deciding to sign up for a product.

In that context, the funnel helps businesses realize that there are stages consumers go through before signing up. In order to market and sell successfully, businesses need to move people through those stages, and not just jump right to the end by trying to sell, before the potential buyer is ready. Very rarely does someone show up on a B2B website for the first time and whip out their credit card, ready to purchase.

The funnel tapers because you expect to lose people along the way: not everyone who visits your website will become a good lead for your business, and not every lead will end up signing up. That’s totally expected. But you want to minimize the loss between each stage, so that a higher percentage of visitors become leads, and a higher percentage of leads become customers.

An Example Post Sign Up Funnel

For products that are tailored to consumers (so-called “B2C”), sign up is often much faster and less calculated. But there are still stages that a user goes through between when they first find out about your product and when the become an evangelical customer.

Let’s look at what an example post sign up funnel might look like for a typical B2C product:

Stage 1: Receives Tour
We start by giving them a tour so they understand how things work. Hopefully, every user that signs up will go through the tour so we get a chance to show them all that our product can do. There shouldn’t really be any drop off at this early of a stage.

Stage 2: Performs Primary Action
Next, we drive the user to actually use our product. If there are multiple things a user can do with our product, we should drive them to the action that proves the “stickiest,” which we’ll talk about later.

If there’s a lot of drop off between stage 1 and 2 — that is, lots of people go through the tour but don’t perform the primary action — then you probably means you should tighten things up and make the tour shorter. Maybe even skip the tour altogether and drive them right into the primary action.

Stage 3: Receives Positive Feedback/Value
This is the most important stage you want to drive a user through — the moment where they experience the value of your product first-hand. If a new user doesn’t make to the stage quickly, you’re likely to lose them forever.

Maybe they started uploading their information, but it was taking too long so they never ended up getting your sweet analysis. Or maybe there were too many steps between taking the picture and sharing it with friends.

If someone starts performing your product’s primary action but doesn’t make it all the way, this is very bad. Sure, you might lose a few people for random reasons, but most people who start using your product should get to see the value for their effort. If they’re not, then you have a product issue and should probably address that before anything else.

Stage 4: Comes Back Later
At this stage, a user has already taken your product for a spin once. Some users might only ever use your product once, and never come back. And for some businesses, this might not be an issue. But ideally, the user liked it so much that they come back and use it again later. Encouraging regular, habitual use of your product is a great way to build a strong, loyal fan base.

Some startups only report on “active users” — that is, users who have come back to the product recently — rather than total users, since it gives a clearer picture of how valuable users find the product. In fact, looking at the percentage of “daily active” or “monthly active” users to total users is a great way to understand how loyal your user base is.

You’re going to have a hard time trying to encourage referrals or reviews or up-sells if people aren’t coming back to use your product regularly.

Stage 5 & 6: Invites Friends & Write Review
If a user is so excited about your product that they’re using it regularly, the next stage is to get them to share their experience. Encouraging them to invite their friends or leave a review somewhere is a great way to leverage your current user base to help bring in new users. This is where the “viral loop” kicks in and new users lead to more new users.

But, just like the B2B marketing funnel, you shouldn’t jump to this stage right away or it could come off as pushy. Make sure you’re walking your users through all the necessary steps from curious first-timer to dedicated power user.

Finding Your Own Post Sign Up Funnel

The above is just an example funnel, but I think it’s roughly accurate for a lot of B2C products. Ultimately though, you need to figure out what the funnel looks like for your product. Maybe it ends with referring friends, or maybe it’s simply getting the user to come back on a regular basis.

For a lot of freemium products, one of the main goals is to get a user to upgrade and start paying for premium features. But even once they do that, it doesn’t mean that the user has come all the way through the funnel. You could still potentially drive more upgrades if you have multiple price points, or encourage users to try new parts of the product.

A good way to come up with your own funnel is to find the “stickiest” feature in your product — that is, the feature that’s mostly strongly correlated with low user churn — and drive the user to that. A sticky feature is one where, once the user starts using it, they’ll have a hard time stopping.

It could be that the value from the feature is unavailable elsewhere, or it could be that the switching costs of that feature are so high that users would rather stay put than move to another product.

Some people bristle at the idea of locking a user in, but it’s not necessarily about keeping their data. It’s about creating value they can’t find elsewhere.

For example:

Even if I found a better photo sharing site than Facebook, I still want to tag my friends in all my photos. Since my friends are all on Facebook already, I’m pretty much locked into Facebook for photo sharing.

Even if I wanted to switch from Gmail, I’ve given out my email address to so many people that the switching costs of moving to a new email address would make it nearly impossible.

I could export my data from either service, but where would I take it to get the same value? Once I start using those services, it’s very hard to stop. Photos and email are both sticky features for Facebook and Google, respectively.

The Growth Hacking Tool Belt

Just like with the marketing funnel, you want to constantly be helping your users move from one stage to the next, losing as few along the way as possible. Drive them from sign up to first use, then to regular use, then to becoming evangelists.

At any given point in time, you’ll have users that are at every stage of the funnel. This means it’ll be tricky to manage things manually, so you should think about ways to build encouragement into your product so that it happens automatically.

There are a few categorical ways to build growth into your product. We’ll go through some of them and then look at examples of hot startups that have successfully used these techniques to drive huge user growth.

Lifecycle or “Drip” Emails
Lifecycle emails are a series of pre-written emails that are automatically sent to users based on certain triggers. The triggers are either related to how a user is using the product, or simply keyed off a time delay where users receive an email every few days.

The time-delayed emails are the easiest to setup. Every user receives the same series of emails based on when they signed up. Here’s an example of a time delayed — or “drip” — campaign:

• Day 0: Thanks for signing up!
• Day 2: Here are some tips for getting started.
• Day 5: Here’s what other user are doing.
• Day 8: Advanced tips for our product.
• Day 12: Let us know if we can help.

To be a bit smarter about it, you can send these emails based on how users are interacting with your product. If you noticed they signed up but haven’t performed your product’s primary action with a day or two, you might send an email to see if they have questions.

If they haven’t logged in in awhile, let them know what they’ve been missing. Or you can use email to encourage users to keep doing what they’ve been doing.

Twitter does a great job of this with their “Suggestions similar to” emails. Clearly, they’ve discovered that a user is more likely to be active on Twitter if they follow lots of other accounts — following new accounts is a sticky feature for Twitter because it creates a ton of value (seeing what friends and celebs are saying) in a way that a user can’t find elsewhere.

When a user first sign up, Twitter is constantly suggesting other accounts to follow on their website. But once the user has followed a certain number of accounts, they’ve moved to the next stage of the funnel, and the on-site recommendations start to be less intrusive.

However, whenever when a user follows someone new, Twitter sends them an email with suggestions for similar accounts they might want to follow. I personally get a lot of these emails, but I don’t mind because they’ve helped me discover a ton of interesting people. It helps me get more value from the service as a user, and it helps Twitter ensure that I’m coming back again and again.

In fact, Twitter has a number of email notifications that are all designed to pull users back to the site. They offer tips on getting more out of the product, things a user has missed since the last time they logged in, as well as top tweets and stories from around Twitter.

Lifting Restrictions
One of the best ways to encourage behavior is to reward it. If your product has a freemium version that comes with certain restrictions, lifting those restrictions is a super easy way to incentivize users to perform an action you’d like them to take.

There are tons of great examples of this but I think Dropbox’s is the most well-known. They give away free storage space for literally every single part of their post sign up funnel:

• 500MB for everyone I invite to Dropbox, up to 16GB
• 250MB for taking a product tour
• 125MB for hooking up both my Twitter & Facebook accounts
• 125MB for following them on Twitter

By completing all of these steps, a user is not only unlocking more storage space, but also becoming both a power user (product tour + following Dropbox on twitter for updates) as well as an evangelist (inviting all their friends).

Performing each of these steps helps the user get more out of Dropbox, and helps Dropbox ensure that they have a well-educated user base full of evangelists.

Gamification
If your product has any game mechanics built into it, you can use those to incentivize behavior that drives users through the post sign up funnel.

A site that does this well is Q&A site StackOverflow. While you can obviously win badges for asking good questions and providing helpful answers, they have a number of other badges that simply encourage users to become competent, regular users.

• Analytical (bronze): Visited every section of the FAQ.
• Autobiographer (bronze): Completed all user profile fields
• Informed (bronze): Read the entire about page
• Enthusiast (silver): Visited the site each day for 30 consecutive days
• Fanatic (gold): Visited the site each day for 100 consecutive days

These badges don’t really involve any participation beyond just setting up your profile, reading about the site, and then checking back regularly. A simple but effective way to keep users coming back, even if they’re unsure about participating in the community just yet.

Have Your Active Users Do The Heavy Lifting
If your product has a strong social component, you can ask your active users to do the hard work of pulling other, less-engaged users back in.

Every once in awhile, Facebook shows an area on the side of the news feed that suggested friends to “reconnect with.” While the feature is ostensibly designed to help you refresh old relationships, it has the curious property of only showing profiles of users who hadn’t been active on Facebook in a few months.

Facebook is essentially encouraging its most active users to help pull back in the least active users. Presumably, the old friend will receive an email saying that you have written on their wall or want to reconnect, and they would want to click through to see what you had said — by logging into Facebook for the first time in awhile.

Very clever, and I’m sure it is very effective at pulling unengaged users back in.

Using the Metaphor

Getting a user signed up for your product is really only the beginning of your relationship with them. In order to build a viable business, the product needs to show value, pull users back, and create evangelists.

How each product does this varies widely, but the post sign up funnel provides a great metaphor for thinking about the relationship you want to have with new users, and how you want to grow that relationship over time.

Not every user will end up inviting all their friends, paying you money, or otherwise making it to the “bottom” of your funnel. But if you really think about each of the funnel stages and how you can gently nudge your users between them, you’ll build an awesome product experience and a highly-engaged user base.

Now that you now how to retain them, ready to attract new users? Check out my eBook Marketing for Hackers, the step by step guide to turning your weekend project into paying customers.

Discussion on Hacker News.

With an extra hour or two of setup, you can have a well-optimized, secure blog that pulls in new visitors and keeps out intruders. Most of these things are one-time setups that you won’t have to think about again.

These are the tricks I’ve learned after setting up dozens of WordPress blogs for myself and for friends — some tiny niche sites and some massive content destinations attracting 10k+ visitors a day.

Before you get started, make sure you’re running the latest version of WordPress by going to Dashboard > Updates and checking for any upgrades.

Since the WordPress software is the most used CMS on the internet, it’s a huge target for hackers and spammers, and new security updates come out frequently. You wanna make sure you’re starting with the latest, most secure version.

Optimize for Search Engines

You want your website to be easy for search engines to find, crawl and understand. Despite its reputation, SEO isn’t all that scary. A little bit of structural work up-front will help your content rank more highly and drive search traffic to your blog, forevermore.

URL Structure
The first thing you need to do is change the default permalink structure that WordPress comes with. Unless your content is really time-sensitive, you probably don’t want all those date numbers pushing the keyword-rich part of your URL further to the right.

Search engines generally give preference to pages where keywords appear closer to the left in both the URL and the page title (which we’ll adjust in a minute). Go to Settings > Permalinks and make sure you choose the “Post Name” permalink option that doesn’t add the unnecessary year and month numbers.

Now that you’ve got that setting out of the way, you’re going to install two of the more popular WordPress plugins: All in One SEO Pack and Robots Meta.

All in One SEO Pack
Once you’ve installed the All in One SEO Pack, go to Settings > All in One SEO and make sure “enabled” is selected. There are a lot of options in here and most of the defaults are pretty good, but there are a few things you should consider:

• Home Description This is the snippet of text that someone sees when your site shows up in search engine result pages (SERPs). Use it to briefly explain your site (160 characters max) in a way that entices people to check it out.
• Google Plus Profile Sitewide Default If you have a Google+ account, you can link your site’s content to your Google+ account and benefit from authorship snippets in search results. If you’re writing a personal blog where you’re the only author, this is a great idea.



• noindex for Categories, Archives and Tags These pages will almost always create duplicate content issues for your site. Unless you really use Categories or Tags to create special sections of your site with unique content that doesn’t show up on your homepage, you probably don’t want these to be indexed by search engines.

The other cool thing that All in One SEO Pack does is provide additional space for you to provide a custom meta description and title tag for each one of your posts.

If you go to Posts > Add New and click “Screen Options” at the top, make sure “All in One SEO Pack” is checked, and then you can scroll down to add that information to each article.

You should consider updating the “Title” — or page title tag — for each post. This can, and often should be, different than the actual title of the article that visitors see on your blog. The title tag has a big impact on a post’s ranking in search results, so it’s important to pick a good, keyword-rich title.

The “Description” — or meta description — is the text snippet that shows up in SERPs for each post, just like what you filled out for the Home Description earlier for your homepage. It’s also the preview text that Facebook grabs when someone shares a link to your content. Unlike the title, the meta description has no impact on search rankings for an article.

If you’d like to learn a bit more about things you can do on your site to optimize it for search engines (so called “On-Page SEO”) check out this awesome article by SEOmoz.

Robots Meta
The Robots Meta plugin gives you tons of control over how search engines crawl and index your content. These can get a bit advanced, but we’ll just set up some sensible defaults. Once you’ve installed the plugin, go to Settings > Robots Meta and make sure the following options are checked:

• Prevent Indexing
  • This site’s search result pages
  • All Admin Pages
  • Author Archives
  • Date-Based Archives
  • Category Archives
  • Tag Archives
• Archive Settings
  • Disable the author archives (only if your site is a single-author site)
  • Disable the date-based archives
  • Redirect search results pages when referrer is external

This should take care of all of the major duplicate content issues that WordPress sites often face. If you’re a particularly savvy webmaster, you can also integrate the plugin with your existing Google Webmaster Tools account. If not, not worries.

Build an Engaged Following

Now that your site is well-optimized for search engines, you want to think about how you can start to retain visitors, and keep them coming back to see your latest content. The easiest way to do this is to start building a following of email and social subscribers.

Set Up a Free Email Subscription Service
It’s common to think that it’s not worth setting up an email distribution list, but you’d be surprised by how many people like to get subscriptions in their inbox. Maybe it’ll just be your mom at first, but over time, that email list will grow if you promote it well. Plus, it’s surprisingly easy to set one up.

To get started, create a free account with MailChimp and then create an RSS-Driven campaign. Just tell them where your blog’s RSS feed is (by default, it’s <blog_domain>.com/feed/), pick a cool email template, and then it’ll automatically send out an email to your subscribers whenever you publish a new article.

Once you’ve gotten that configured, make sure you make it easy for readers to give you their email address and subscribe to your content. MailChimp has their own WordPress plugin that lets you put a “subscribe” box in the sidebar of your website — just like the one you see in my sidebar, above.

You should also build out a dedicated “blog subscription landing page” on your site that you can send visitors to if they want to subscribe. This gives you more room to explain a bit about the content you write, how frequently you post, and what the visitor should expect by subscribing. I’ve written about effective landing pages before, so check that out for more info.

Optimize for Social Sharing
If you’re already active on Twitter and Facebook, you probably don’t need to build out a dedicated Facebook page or Twitter profile just for your blog. If you’re only writing articles once or twice a week, you should simply encourage your blog’s visitors to subscribe to your updates on Twitter or Facebook, so that they’ll also see later articles you publish.

If you want to make sure your content looks good when it’s shared on social media, you might consider adding Facebook Open Graph tags or Twitter Cards to your post. These are special markup tags that ensure both social networks can understand your posts and provide proper previews when people share links to your content. There are plugins can help with that, if you’re interested (Facebook, Twitter).

Security & Spam

There are two broad classes of undesirables on the internet — hackers and spammers. As mentioned earlier, WordPress sites are widely targeted by hackers and spammers because of how common the software is.

Even if you’re just creating a small site in a tiny corner of the internet, there are all sorts of robots spidering the web and looking for sites with vulnerabilities. Don’t think you’re too small to be a target, you will be.

Fortunately, you can set up some basic, simple defenses that’ll go a long way to keeping your site safe and spam-free.

Keeping out the Hackers
If an attacker is able to login to your dashboard, that’s obviously a really bad thing. While they could do something dramatic like delete all your articles or post a bunch of spam on your blog, they’re more likely to do something subtle to try and remain undetected.

They might add some links to the footer of your site, or change the content of older posts. Don’t assume that you’ll notice if your site is compromised. Make sure your dashboard has rock solid defenses by installing the following two plugins: LimitLoginAttempts and Google Authenticator.

LimitLoginAttempts is a crucial WordPress plugin that does exactly what it says — it prevents attackers and robots from trying to repeatedly guess your password, a common technique known as brute forcing.

If the plugin detects too many failed login attempts from a particular computer, the plugin will automatically block that machine from trying to login again for a set period of time. The defaults on this are pretty good, but you can make them stricter by going to Settings > Limit Login Attempts.

Installing Google Authenticator will add an extra field to the login screen making it even harder for a hacker to get in. In order to login, you need to provide a username, password AND a random code generated by an app on your phone. This means that even if someone were to guess your password, they still wouldn’t be able to login unless they also had access to your phone.

This is a super secure login method known as two factor authentication and it’s a no-brainer to add another layer of security.

With those two plugins installed, it’s going to be nearly impossible for someone to gain unauthorized access to your blog.

Keeping Out Spammers
The other group of undesirables are the spammers. If you don’t put any anti-spam measures in place, you’ll end up with hundreds of useless blog comments that link back to spammy websites, which are a huge pain to deal with.

By default, WordPress ships with Akismet, which is a basic spam prevention service that’ll flag comments that seem spammy and let you review them before posting them on your site.

The alternative that I prefer is to use the Facebook comments plugin. This requires visitors to be logged into Facebook in order to post a comment, and ties their identity to their message. While it’s not explicitly designed to reduce spam, I haven’t seen a single spam comment since I switched my blog over to it a few years ago.

For sites that prefer to allow anonymous comments, this might not be the best option, and you might consider Disqus instead.

Hosting & Monitoring

At this point, you’ve set up a well-optimized website that’s ready to grow a loyal audience over time. Hackers and spammers are no match for your defenses.

But even if all of the software runs properly, some things are out of your control. Servers crash and files get corrupted. Networks go offline and can bring down your website. Here are a few tips to handle common problems every website will run into eventually.

Automatic Nightly Site Backups
Just like backing up your music and movies, you want to keep a fresh copy of all your website’s content stored in a safe place. The best plugin I’ve found for this is WP DB Backup.

Once you install the plugin, go to Tools > Backup and go to the “Schedule Backup” section. Make sure you schedule the backup for “Once Daily” and have it emailed to yourself, then click “Schedule Backup.” By automating nightly backups, you’ve just proven yourself smarter than the vast majority of website owners.

Now, if your site every crashes or gets hacked, you’ll always have a spare copy of your data that’s no more than a few hours old, making it really easy to get your content back up and your site running again in no time. Without a fresh backup, you stand to lose a lot.

Free Website Monitoring
Websites go down for all sorts of reasons. I cancelled my credit card when I lost my wallet a few years ago, and forgot to update my billing information with my hosting provider. After a few failed charge attempts, they took my websites offline.

Fortunately, I had set up free website monitoring from Pingdom, so I was notified via text message within a minute of my site going offline. Pingdom checks your website every minute from data centers around the world and notifies you as soon as something goes offline.

You’ll know as soon as there’s a problem and can take steps to get your site back online quickly. In my case, I updated my credit card information and my sites were back up after about 5 minutes of downtime. Not bad!

Totally Managed Hosting
If you’re looking for a secure, reliable, fully-managed WordPress hosting service, I switched to WP Engine about a year ago and haven’t regretted it at all. They’re the experts on keeping a WordPress site running fast and secure.

So far this year, my blog has only had 20 minutes of downtime (99.98% uptime) and is averaging less than half a second page response times (351ms), which is pretty good considering I didn’t have to lift a finger to manage any of that. They also gracefully handle huge traffic spikes if (…when ) an article becomes popular.

You give up some control versus hosting and managing a blog yourself, but I’ve found the peace of mind and reliability to be worth it. Plus migrating your existing site over to them is super easy. Definitely worth checking out.

—

Once you’ve gotten all of this set up, the final step is to start writing! Growing a blog audience takes a bit of patience and consistency, but it’s a great investment and can be really rewarding.

If you’re thinking about blogging to grow interest in a product or business, you should check out my eBook called Marketing for Hackers. It’s a step by step guide to turn your weekend project into paying customers.

I’ve been an avid Buffer user for a long time, and I’ve been following their company blog for all of the life hacking and interesting social psych Leo writes about.

They’re a small team that’s made a huge impact already — 10 employees, 550k users, $100k MRR — and are a fairly well known name in startup and social media circles.

But beyond their product, I’m really interested in being a part of their company’s culture. Even as an outsider, it’s clear that Joel and Leo have nurtured an environment that trusts team members to find what makes them happy and effective, and to constantly be learning from each other.

I also love their focus on living smarter, not harder.

Buffer culture 0.1 from Joel Gascoigne

I see them as the poster child of how most startups will operate in 10 years.

The team is distributed all around the world, so I’ll be working remotely and staying in the Boston area for the foreseeable future, which I’m also really excited about.

Boston has a great community of startup peeps and I’m going to focus more on giving back to the community that’s given so much to me these past few years.

Thanks to everyone I’ve worked with — directly and indirectly — over the past two years at HubSpot. It’s been an awesome journey, and I’m looking forward to the next saga!

One of the rookie mistakes first-time entrepreneurs often make is to assume that visitors on their website are ready to buy their product or service right away. I’ve made this mistake myself.

But in reality, the average visitor to your website isn’t ready to make a buying decision… yet.

In order to grow a successful business, you need to understand the concept of “the funnel” and the stages that fall between “casual website visitor” and “evangelical customer.”

This concept of a funnel is something that applies especially well to B2B products, since your prospects will tend to move more slowly and make more deliberate decisions.

Stage 1: Top of the Funnel

At this stage, a visitor might be hearing about you for the first time. Maybe they saw a link to your blog article on Twitter or found your homepage through a non-branded search query.

They’re probably not ready to buy yet, they’re just checking things out. They don’t know much about you and your product yet, so your goal at this stage should be to start a relationship and build trust, not to sell them.

There are a few specific things you can do at this stage to begin a relationship:

1. Get them to subscribe to your blog. You are blogging, right? You want them to trust your brand and think of you as a thought leader. Get them clicking through your content and encourage them to subscribe to get the latest articles. This gets you in their inbox regularly, and you become seen as a source of information, not just a company trying to sell.
2. Suggest your free tool. If they’re on your site, they’re probably looking for help with something. Having a basic, free tool is a great way to help them solve their problems without requiring them to open their wallet. HubSpot does this well with our free Marketing Grader tool — it gives you free analysis of your website and has run millions of reports since it was launched.
3. Offer a free eBook or webinar. The goal is to help them while only requiring some basic contact info — like an email address — in exchange. Use a landing page to collect their email address and other follow-up information, and then send them a copy of your eBook, or invitation to a webinar.

Stage 2: Middle of the Funnel

At this point, the prospect knows who you are, and has exchanged their contact information for some useful content or service you’ve provided them.

At this point, they may be ready to buy. You’re not 100% sure how long it’ll take them to be ready. But you keep nurturing the relationship by regularly following up and sending them more useful, targeted information.

Usually, this stage of the relationship happens via email since it ensures you’re getting in front of them again. It’s sometimes known as “lead nurturing” or “marketing automation” but the general idea is the same. You should have a pre-planned series of emails you can use to follow up with someone once they’ve provided their email address.

Send them a few of your top-performing blog articles, or a new eBook you’ve written. It’s got to be something useful that will continue to help them solve their problems. You don’t want them to open the email and see a sales pitch — they’ll likely never open another email from you again.

But you also want to make it easy for them to get in touch with you when they’re ready to buy, so that you don’t lose a sale. Allow them to respond to emails with feedback (don’t use a “noreply” email address), or direct them to a “get in touch” landing page where you can answer their questions more directly to begin the sales process.

Stage 3: Bottom of the Funnel

At this point your prospect knows who you are, has come to see you as a trusted source of information, and has raised their hand to say they want to get in touch. They might have signed up for a free trial, or requested a demo of your product.

Their behavior indicates that they’re seriously in the process of evaluating your product and want to learn more about it. This is the easiest kind of sale to make, since they’re an inbound lead and have demonstrated that they’re sales-ready.

—

You’re much more likely to acquire customers when you set up a sales and marketing funnel that’s focused around your prospects’ needs. Get in front of them with shareable content, help them with free tools and more useful information, and then nurture the relationship to keep your company top of the mind for them.

This stuff is all super high level. If you’re looking for more tactical ways to execute some of this, check out the eBook I’m working on called “Marketing for Hackers.”

But I’ve found it’s also really easy to get stuck overthinking some things. What tie should I wear? Which apartment should I live in? What do I want to do with my life?

Sometimes there are so many options that we’re overwhelmed. Sometimes we get competing feedback from our trusted advisors. We waste so much time thinking about it, that we never pull the trigger. Opportunity passes us by.

I stumbled across an apt analogy the other day that really framed the issue of overthinking things.

A friend of a friend saw my article about the weather balloon I sent to space. He reached out with a few questions and we started a good discussion via email.

At one point, he mentioned that he needed to do a bunch of calculations to figure out the amount of helium he would need in order to generate enough lift to send the balloon into the atmosphere with all of its cargo.

It struck me that I had never even given any thought to those calculations when I sent up my balloon. At the time, I wanted to make the launch happen quickly. My intuition simply said to keep things light, and use as much helium as I could.

If I did that, it would just kinda go upwards. And it did.

But suddenly I panicked that I had missed something. There must have been a bunch of science that I totally skipped over that would help me figure out exactly what I needed. So I dug out my Ideal Gas Law notes from my high school AP Chemistry class, and started coming up with a model for the balloon’s flight.

Okay, let’s see. PV = nRT

P is for pressure and I’m sure we can calculate that based on how tight the balloon is. Not exactly sure how we’d measure that thought. But wait, won’t the pressure on the outside of the balloon change as it rises into thinner air? Hmm, we’ll skip that and come back to it.

V is volume and that should be easy to measure on the ground. Get the diameter of the balloon, then cut it in half and plug it into 4/3 π r 3. I’d have to make sure I keep track of my units.

T is for temperature and I definitely know that’ll be changing from the surface going upwards. Would wind chill factor into this? What about winds that might blow the balloon up or down? Hmm… We’ll come back to this one too.

n is the amount of gas, measured in… moles? Oh boy.

What am I trying to do? Calculate lift? In pounds?

That’s when I stop scratching my head and took a step back. If I really wanted to, I could spend hours diving into this problem and doing a bunch of calculus. But even if I did all that, there would still would be a lot of variables that would be hard to measure and predict at all!

So even if I did all that work to try and come up with an approximation, I’d still wouldn’t learn much more than I already knew — that I need the highest helium:weight ratio in order to maximize success. Even without all the math and consternation, the balloon would still rise if I went with my instincts.

Why overthink it, if I already knew the answer?

—

One of the greatest ironies of life is that often, the answers we spend so long looking for end up being right in front of our faces the whole time.

Don’t overthink things. Trust you instincts.

The first product was a social network where students could share outfits and fashion ideas [archive].

The second was a music distribution service for artists that want to tap into how listener behavior has shifted to blogs and free downloads [archive].

Unfortunately, as of this writing, both startups are no longer functioning.

While I was mostly on the technical side of things, I learned a ton of business lessons about startups that I wanted to share.

You’re Probably Over Engineering It

It’s good to read articles about how Facebook and Netflix and Reddit solve their scaling and hosting challenges. It’s bad to think those lessons apply to your startup.

When you’re constantly devouring those articles on Hacker News, it’s easy to get caught up in architecting some super-advanced system. When we were building the music distribution service, I came up with this elaborate architecture that required multiple technologies and instances and CNAMEs:

I built up all this complexity, and ended up stripping it back to two ec2 micros and an s3 bucket, which held up just fine. A simple stack will get you surprisingly far.

We did start to grow traffic to the point where scale became a sporadic issue, but the problems were always bugs in my code that I hadn’t anticipated. A complex hosting environment wouldn’t have helped, and would probably have made issues harder to debug.

You ain’t gunna need it. Plus, if you get to the point where scaling your infrastructure really becomes an issue, you’ll probably have gotten some money which you can use to hire someone smarter than yourself to fix it.

Always be Validating Your Idea

The basic premise behind validation is that you shouldn’t waste time building something that nobody wants. It sounds simple enough, but it’s easy to get caught up in the excitement of building something and end up doing a poor job on validation.

Find Strangers to Validate Your Idea
Both startups began essentially the same way — first there was an intriguing idea, then we kicked it around for a few weeks, and finally decided it had enough promise to dive in and start building. Fast forward a few month and all we had was a big pile of code and no customers to show for it.

How could this happen! we thought. We did customer validation: our friends all said they loved the idea! Wasn’t that enough?

In hind sight, it’s obvious that I made two key mistakes, both times:

1. Our friends weren’t our target customers.
2. Our friends didn’t really understand what the fuck we were doing so they’d just smile and nod to make us feel better.

The takeaway from these mistakes is that you need to find people outside your friends and family that actually like your idea.

Find People Who Need Your Product
Which brings me to the next point. Validation doesn’t mean that people “like” your idea. People, even strangers, will usually say they like it because it’d be rude to say otherwise.

You want people who need your idea. People who need it so much they’re willing to put up with downtime and a bad UI and constantly changing features and all the other issues that nascent products often face.

Paul Graham says it well in an essay called How to Get Startup Ideas:

When a startup launches, there have to be at least some users who really need what they’re making—not just people who could see themselves using it one day, but who want it urgently.

Usually this initial group of users is small, for the simple reason that if there were something that large numbers of people urgently needed and that could be built with the amount of effort a startup usually puts into a version one, it would probably already exist.

Which means you have to compromise on one dimension: you can either build something a large number of people want a small amount, or something a small number of people want a large amount. Choose the latter. Not all ideas of that type are good startup ideas, but nearly all good startup ideas are of that type.

Even if we had looked a little further outside our friend group, we should have noticed the lack of raving fanatics. If people are kinda “meh” on your idea, you’re going to have a hard time getting them to pay you for it. Find someone who wants to put their money where their mouth is before you consider the idea totally validated.

Selling is Really Hard

Until recently, sales always seemed like an unnecessary soft skill. If I had a strong product, customers would find me. If you build it, they will come. The product sells itself. How hard could it be?

Selling Sausage
While I was spending my time thinking about features and implementation, I got bogged down in the messy details. Certain things didn’t work out how I thought they would. I found myself making arbitrary deicions cause I didn’t have enough data to go off of. Once we had a few users, it seemed like they were always finding new bugs or turning up edge cases I hadn’t planned for.

As a result, most of the time both products felt like sloppy, creaky messes full of mistakes and bugs just waiting to fall over. I was embarrassed to ask for money for them.

I found solace in the fact that this is a rather common phenomenon for startups to be in. Reid Hoffman (LinkedIn co-founder) famously said:

Reid Hoffman: “If you’re not embarrassed by your first product release, you released too late.” #PandoMonthly

— PandoDaily (@PandoDaily) August 10, 2012

And while it is comforting to know that other entrepreneurs have felt the same way, no one mentions how hard it makes it to actually sell the damn thing you’re building.

I’d go from monkeypatching some javascript compatibility issue to jumping on a call to talk about how stable and secure our product was. Inevitably, I had to defend it against some well-entrenched competitor with a huge engineering team that’s already attracting top-notch talent.

Seeing how messy and haphazard the prototype of a product is makes it almost impossible to sell. If you’re the one making sausage, you probably shouldn’t be the one selling it.

Benefits vs Features
It’s really easy to fall into the mindset that your product is simply a bundle of features. After all, that’s how you’re building it — going down the list of things you need for your first prototype.

But features don’t sell a product, benefits do. Unfortunately, you don’t built benefits, you build features. Hence the disconnect.

At one point, I typed up a one-page summary of all the features we had built or were planning to build (mostly the latter) and sent it off to a hot prospect. She was the CEO of a music label whom I knew personally. She was excited about our idea and had asked for a proposal before she started getting all of her artists using our product. We figured it’d be a layup sale.

She said no. In retrospect, sending someone a one-page list of features and expecting that to convince them to spend hundreds of dollars a month on our service seems extremely naïve.

Selling is now a skill I hope to work on this coming year.

Startup Incubators are Like College

You can waste a ton of time filling out paperwork and applying to them or you can focus on getting shit done. If you get into a top tier one, it will add to your brand and look good on a resume, which helps people take you seriously. But other than that, they’re mostly useless.

Everyone seems to think they’re a good idea even though most have questionable value. But, if you’re good, you probably don’t need to waste your time in one.

I built that extended metaphor after applying and failing to get into Mass Challenge for the first startup. Actually filling out the applications was a good experience since it forced us to think about a lot of questions we hadn’t directly answered yet.

But the application process took up several days that would’ve been better spent working on the product and talking to potential customers (did I mention how important validation is?). We should have waited till we had more traction before applying. There probably would have been more value for us if we had a clearer sense of what we really needed.

I don’t think I’d do it again.

My New Year’s Resolution for 2013 is actually “no new startups.” I want to commit more time to working with and learning from great people this year, so that I can hopefully raise my batting average and take another swing in 2014.

If you have an idea, here’s what you need to know to get started.

Discuss on HN

PS: I’m also planning on spending this year finishing up Marketing for Hackers, a step-by-step guide to turn your weekend project into paying customers, which you should check out right now.

But a few weeks ago, I switched to the Marketing team, where I now work on SEO and driving leads from organic search and referral traffic.

My path is an unusual one, and it puts me in quite a hybrid position. My day-to-day work is much different than it was on engineering — there are no UIs to build or deploys to coordinate. But it also looks much different from the other members of the marketing team — none of them spend all day in Sublime Text and terminal windows.

So I’m not exactly a marketer, but I’m probably not what you’d call an engineer. I am: a growth hacker.

A Clearer Definition of “Growth Hacker”

I believe it was Andrew Chen’s famous article from last spring that really crystallized what it meant to be a growth hacker. But kudos if you can survive reading through the buzz words:

Growth hackers are a hybrid of marketer and coder, one who looks at the traditional question of “How do I get customers for my product?” and answers with A/B tests, landing pages, viral factor, email deliverability, and Open Graph. On top of this, they layer the discipline of direct marketing, with its emphasis on quantitative measurement, scenario modeling via spreadsheets, and a lot of database queries. If a startup is pre-product/market fit, growth hackers can make sure virality is embedded at the core of a product. After product/market fit, they can help run up the score on what’s already working.

Unfortunately, a lot of people drowned in the slew of topics and channels he lists and ended up missing the point of the article. Growth hacking isn’t just a new channel for marketers to use. And it isn’t another worthless buzz word for startups to throw around.

Growth hacking represents the fusion of two fields that were previously discrete. A true growth hacker possesses two separate sets of skills and finds ways to combine them to drive marketing metrics.

What Isn’t A Growth Hacker?

I think Andrew’s definition would have been much clearer if he had stopped at: Growth hackers are a hybrid of marketer and coder.

There’s been a lot of confusion and misinformation around the position, which weakens the community’s understanding of what makes a growth hacker valuable, and why it’s a new position and not just a buzzword for a plain-old marketer.

I think most of the confusion stems from competing definitions of the word “hacker”. Paul Graham has a great essay about hackers where he starts by saying:

…the noun “hack” also has two senses. It can be either a compliment or an insult. It’s called a hack when you do something in an ugly way. But when you do something so clever that you somehow beat the system, that’s also called a hack.

A lot of people are using the “beat the system” part of the definition, presenting clever ways to secure guest blogging opportunities or get more SlideShare views as “Growth Hacking.”

One article even made a list of ways to “growth hack” PR that says:

1. Be creative.
2. Build Relationships as you pitch.
3. Create interesting content
4. Take advantage of Social Media.

To anyone who has actually spent time in marketing, it’s obvious that these aren’t new ideas. It certainly doesn’t take a new type of employee to execute them. It’s dubious whether they’d even be considered “hacks” at all — it sounds more like marketing best practices, dressed up with a new name.

While the internet has ushered in tons of new channels for acquiring customers, marketers as a whole have responded fairly well. The new concept of inbound marketing centers around ways that non-technical marketers can leverage increasingly online buyer behavior to get leads and customers.

Optimize your website for search engines. Interact on social media. Blog and create interesting content. Use email intelligently. Track everything and do more of what’s working. These are things that savvy marketers can do thanks to a whole bunch of marketing tools that have sprung up in the last decade. But it’s not necessarily growth hacking.

So What Does a Growth Hacker Actually Do?

So how does the combination of marketing and engineering manifest itself in a day-to-day sense? I see my role as a growth hacker as three-fold:

1. Build tools that drive visits, leads and customers.
2. Dive into our data and find interesting trends and statistics.
3. Automate boring stuff so my coworkers have more time to focus on being awesome marketers.

These are all different ways to use code to drive marketing metrics.

I should mention the caveat that I work at a 400+ employee company that sells a complex B2B software product. Our marketing team is focused on generating qualified leads that our sales team can follow up with. There is no concept of a user-to-user “viral loop” that most growth hacking seems to focus on.

Building Tools (Look, Engineering!)
Besides our blog, the biggest source of referral traffic and new leads on hubspot.com is our free Marketing Grader tool.

A visitor puts in their website and we do some custom scraping and analysis to give back their site’s “Marketing Grade.” Optionally, we ask for an email address, and allow them to sign-up in order to unlock more analysis.

The tool has proved extremely popular, driving hundreds of thousands of visits to our website each month. It’s also very targeted — we’ve seen that referral traffic from Marketing Grader converts very highly into leads and customers.

If a visitor provides an email address, we email them regularly with updates to their grade, in order to pull them back to our site. Each recommendation in the report has a call-to-action that encouraged visitors to check out our main website and download our content, which would get them in front of a sales person.

It’s a great example of writing code that helps generate leads and customers — growth hacking at its finest.

Moonlighting as a Data Scientist
As you’re probably aware, the profession of marketing has become very data driven over the past decade. This is especially true for businesses that drive most of their leads and customers through online channels, where everything can be measured.

All of our tracking and analytics tools hold tons of insights, but it can be non-trivial — and sometimes impossible — to build reports that surface the most interesting information if all you have is an overwhelmed javascript UI and some pivot tables (although I’ve learned to never underestimate the power of a marketer who knows pivot tables…)

As an engineer, I think in terms of data — how it’s organized and structured, and how to break down complex relationships into simple, reusable pieces. Thinking about programming as data manipulation was one of the big breakthroughs for me as a Python programmer.

By combining and manipulating data from APIs, I’ve discovered all sorts of interesting insights that were largely inaccessible to my coworkers.

We not only use these insight to help measure our marketing, but we package the data and use it to create content that helps us drive even more leads and customers. Some of our highest performing blog articles and content offers center around stats that we’ve discovered by diving into huge data sets and using code to pick out patterns.

Being a Super Hero (Automating Mundane Stuff for Others)
I’m a big believer that everyone should learn how to code. That doesn’t mean that I think everyone should build software — just that everyone should understand how powerful raw computation can be, and how much easier it can make your life.

From automating college research assignments to looking up interesting data correlating GDP and Olympic Medal counts, I’ve used code to solve lots of interesting problems that have nothing to do with building software.

Unfortunately, not everyone has the time or patience to learn how to code — so I try to find ways to use my skills to help my coworkers be more productive.

A great example is a small app I wrote for a coworker that saved her a few hours each week. She was spending a ton of time exporting data from several different tools, reformatting it in excel and then uploading it into another system every week.

For me, it was only a few hours to chain together some webhooks, reformat the data on the fly, and automatically insert it into the desired system through a simple API call. Now the data was flowing between systems in real time, and she had a few extra hours in her week.

It was a huge win that saved someone a bunch of time, and it felt great.

Growth Hacking Isn’t a Meme

I take a lot of pride in my role. I love trailblazing a new type of career, and I believe the message that “Growth Hacker is the new VP of Marketing” will become increasingly true in the coming years.

With all the new excitement around growth hackers, the term has been tossed around with a bunch of other buzz words in the tech press recently, causing some people to dismiss it as Yet Another Startup Meme like “Coding Ninja” or “Social Media Guru”:

Sorry to ruin the fun, but growth hacking is here to stay.

According to a Gartner report from earlier this year, marketing departments are expected to spend more on technology than IT departments by 2017. And if you follow the startup space, you’re constantly hearing (and dreaming) about “viral growth” and “user acquisition” and the importance of building that into your product.

Marc Andreessen wrote that “Software is Eating the World” but I think it’s even bigger than that. People who can write code are eating the world. And right now, we’ve got our sights set on marketing.

—

I love to do marketing, and the fact that I can write code means that I can do marketing in new and exciting ways. Ways that are fundamentally different from how it’s been done in the past.

I know that I have a fairly unique background, but I believe it will become more ubiquitous in the next 3-5 years. As more non software engineers learn how to code, we’ll see the field mature into a more widely recognized profession, with its own best practices and literature.

I’m excited for the day when every marketing team has someone who can write code.

Discuss on Hacker News.

PS: I’m working on an eBook called Marketing for Hackers that takes the lessons I’ve learned and helps you market your side project, maybe even turning it into a business. Check it out…

That’s right, we’re talking about pulling our data straight out of HTML — otherwise known as web scraping. Here’s why web scraping is awesome:

Any content that can be viewed on a webpage can be scraped. Period.

If a website provides a way for a visitor’s browser to download content and render that content in a structured way, then almost by definition, that content can be accessed programmatically. In this article, I’ll show you how.

Over the past few years, I’ve scraped dozens of websites — from music blogs and fashion retailers to the USPTO and undocumented JSON endpoints I found by inspecting network traffic in my browser.

There are some tricks that site owners will use to thwart this type of access — which we’ll dive into later — but they almost all have simple work-arounds.

Why You Should Scrape

But first we’ll start with some great reasons why you should consider web scraping first, before you start looking for APIs or RSS feeds or other, more traditional forms of structured data.

Websites are More Important Than APIs
The biggest one is that site owners generally care way more about maintaining their public-facing visitor website than they do about their structured data feeds.

We’ve seen it very publicly with Twitter clamping down on their developer ecosystem, and I’ve seen it multiple times in my projects where APIs change or feeds move without warning.

Sometimes it’s deliberate, but most of the time these sorts of problems happen because no one at the organization really cares or maintains the structured data. If it goes offline or gets horribly mangled, no one really notices.

Whereas if the website goes down or is having issues, that’s a more of an in-your-face, drop-everything-until-this-is-fixed kind of problem, and gets dealt with quickly.

No Rate-Limiting
Another thing to think about is that the concept of rate-limiting is virtually non-existent for public websites.

Aside from the occasional captchas on sign up pages, most businesses generally don’t build a lot of defenses against automated access. I’ve scraped a single site for over 4 hours at a time and not seen any issues.

Unless you’re making concurrent requests, you probably won’t be viewed as a DDOS attack, you’ll just show up as a super-avid visitor in the logs, in case anyone’s looking.

Anonymous Access
There are also fewer ways for the website’s administrators to track your behavior, which can be useful if you want gather data more privately.

With APIs, you often have to register to get a key and then send along that key with every request. But with simple HTTP requests, you’re basically anonymous besides your IP address and cookies, which can be easily spoofed.

The Data’s Already in Your Face
Web scraping is also universally available, as I mentioned earlier. You don’t have to wait for a site to open up an API or even contact anyone at the organization. Just spend some time browsing the site until you find the data you need and figure out some basic access patterns — which we’ll talk about next.

Let’s Get to Scraping

So you’ve decided you want to dive in and start grabbing data like a true hacker. Awesome.

Just like reading API docs, it takes a bit of work up front to figure out how the data is structured and how you can access it. Unlike APIs however, there’s really no documentation so you have to be a little clever about it.

I’ll share some of the tips I’ve learned along the way.

Fetching the Data
So the first thing you’re going to need to do is fetch the data. You’ll need to start by finding your “endpoints” — the URL or URLs that return the data you need.

If you know you need your information organized in a certain way — or only need a specific subset of it — you can browse through the site using their navigation. Pay attention to the URLs and how they change as you click between sections and drill down into sub-sections.

The other option for getting started is to go straight to the site’s search functionality. Try typing in a few different terms and again, pay attention to the URL and how it changes depending on what you search for. You’ll probably see a GET parameter like q= that always changes based on you search term.

Try removing other unnecessary GET parameters from the URL, until you’re left with only the ones you need to load your data. Make sure that there’s always a beginning ? to start the query string and a & between each key/value pair.

Dealing with Pagination
At this point, you should be starting to see the data you want access to, but there’s usually some sort of pagination issue keeping you from seeing all of it at once. Most regular APIs do this as well, to keep single requests from slamming the database.

Usually, clicking to page 2 adds some sort of offset= parameter to the URL, which is usually either the page number or else the number of items displayed on the page. Try changing this to some really high number and see what response you get when you “fall off the end” of the data.

With this information, you can now iterate over every page of results, incrementing the offset parameter as necessary, until you hit that “end of data” condition.

The other thing you can try doing is changing the “Display X Per Page” which most pagination UIs now have. Again, look for a new GET parameter to be appended to the URL which indicates how many items are on the page.

Try setting this to some arbitrarily large number to see if the server will return all the information you need in a single request. Sometimes there’ll be some limits enforced server-side that you can’t get around by tampering with this, but it’s still worth a shot since it can cut down on the number of pages you must paginate through to get all the data you need.

AJAX Isn’t That Bad!
Sometimes people see web pages with URL fragments # and AJAX content loading and think a site can’t be scraped. On the contrary! If a site is using AJAX to load the data, that probably makes it even easier to pull the information you need.

The AJAX response is probably coming back in some nicely-structured way (probably JSON!) in order to be rendered on the page with Javscript.

All you have to do is pull up the network tab in Web Inspector or Firebug and look through the XHR requests for the ones that seem to be pulling in your data.

Once you find it, you can leave the crufty HTML behind and focus instead on this endpoint, which is essentially an undocumented API.

(Un)structured Data?

Now that you’ve figured out how to get the data you need from the server, the somewhat tricky part is getting the data you need out of the page’s markup.

Use CSS Hooks
In my experience, this is usually straightforward since most web designers litter the markup with tons of classes and ids to provide hooks for their CSS.

You can piggyback on these to jump to the parts of the markup that contain the data you need.

Just right click on a section of information you need and pull up the Web Inspector or Firebug to look at it. Zoom up and down through the DOM tree until you find the outermost <div> around the item you want.

This <div> should be the outer wrapper around a single item you want access to. It probably has some class attribute which you can use to easily pull out all of the other wrapper elements on the page. You can then iterate over these just as you would iterate over the items returned by an API response.

A note here though: the DOM tree that is presented by the inspector isn’t always the same as the DOM tree represented by the HTML sent back by the website. It’s possible that the DOM you see in the inspector has been modified by Javascript — or sometime even the browser, if it’s in quirks mode.

Once you find the right node in the DOM tree, you should always view the source of the page (“right click” > “View Source”) to make sure the elements you need are actually showing up in the raw HTML.

This issue has caused me a number of head-scratchers.

Get a Good HTML Parsing Library
It is probably a horrible idea to try parsing the HTML of the page as a long string (although there are times I’ve needed to fall back on that). Spend some time doing research for a good HTML parsing library in your language of choice.

Most of the code I write is in Python, and I love BeautifulSoup for its error handling and super-simple API. I also love its motto:

You didn’t write that awful page. You’re just trying to get some data out of it. Beautiful Soup is here to help.

You’re going to have a bad time if you try to use an XML parser since most websites out there don’t actually validate as properly formed XML (sorry XHTML!) and will give you a ton of errors.

A good library will read in the HTML that you pull in using some HTTP library (hat tip to the Requests library if you’re writing Python) and turn it into an object that you can traverse and iterate over to your heart’s content, similar to a JSON object.

Some Traps To Know About

I should mention that some websites explicitly prohibit the use of automated scraping, so it’s a good idea to read your target site’s Terms of Use to see if you’re going to make anyone upset by scraping.

For two-thirds of the website I’ve scraped, the above steps are all you need. Just fire off a request to your “endpoint” and parse the returned data.

But sometimes, you’ll find that the response you get when scraping isn’t what you saw when you visited the site yourself.

When In Doubt, Spoof Headers
Some websites require that your User Agent string is set to something they allow, or you need to set certain cookies or other headers in order to get a proper response.

Depending on the HTTP library you’re using to make requests, this is usually pretty straightforward. I just browse the site in my web browser and then grab all of the headers that my browser is automatically sending. Then I put those in a dictionary and send them along with my request.

Note that this might mean grabbing some login or other session cookie, which might identify you and make your scraping less anonymous. It’s up to you how serious of a risk that is.

Content Behind A Login
Sometimes you might need to create an account and login to access the information you need. If you have a good HTTP library that handles logins and automatically sending session cookies (did I mention how awesome Requests is?), then you just need your scraper login before it gets to work.

Note that this obviously makes you totally non-anonymous to the third party website so all of your scraping behavior is probably pretty easy to trace back to you if anyone on their side cared to look.

Rate Limiting
I’ve never actually run into this issue myself, although I did have to plan for it one time. I was using a web service that had a strict rate limit that I knew I’d exceed fairly quickly.

Since the third party service conducted rate-limiting based on IP address (stated in their docs), my solution was to put the code that hit their service into some client-side Javascript, and then send the results back to my server from each of the clients.

This way, the requests would appear to come from thousands of different places, since each client would presumably have their own unique IP address, and none of them would individually be going over the rate limit.

Depending on your application, this could work for you.

Poorly Formed Markup
Sadly, this is the one condition that there really is no cure for. If the markup doesn’t come close to validating, then the site is not only keeping you out, but also serving a degraded browsing experience to all of their visitors.

It’s worth digging into your HTML parsing library to see if there’s any setting for error tolerance. Sometimes this can help.

If not, you can always try falling back on treating the entire HTML document as a long string and do all of your parsing as string splitting or — God forbid — a giant regex.

—

Well there’s 2000 words to get you started on web scraping. Hopefully I’ve convinced you that it’s actually a legitimate way of collecting data.

It’s a real hacker challenge to read through some HTML soup and look for patterns and structure in the markup in order to pull out the data you need. It usually doesn’t take much longer than reading some API docs and getting up to speed with a client. Plus it’s way more fun!

PS: I’m working on a free guide called Marketing for Hackers which you should totally check out right now. Click here to check it out now…

Disclaimer
While scraping can sometimes be used as a legitimate way to access all kinds of data on the internet, it’s also important to consider the legal implications. As was pointed out in the comments on HN, there are many cases where scraping data may be considered illegal, or open you to the possibility of being sued. Similar to using a firearm, some uses of web scraping techniques can be used for utility or sport, while others can land you in jail. I am not a lawyer, but you should be smart about how you use it.

Coming to terms with that is a huge part of growing up.

But I’m starting to see the other side of that coin.

As a young adult, I have the autonomy to be relentless in my pursuit of happiness. I don’t have to go to class. I don’t have to mow the lawn or do the dishes. There’s no systematic frustrations I must put up with or tribulations I must survive.

It is 100% up to me to make sure that I’m enjoying every single day. And if I’m not, it’s my own fault. The only constraint is that I provide for myself. Past that, I’m essentially unencumbered.

“Choosy” vs “Durable”

And so I have to start being more choosy with all of my decisions, to make the most of my time. This is something I’ve had to re-learn.

Do I really enjoy my career? Do the people I surround myself with make me happy? Am I learning new, interesting things? Am I getting enough exercise? If the answer is “no,” that might still be okay if there are no better options, but it should certainly give me pause.

It’s not selfish or egotistical, it’s just good living.

Growing up, I was constantly operating under what I’ll call a “durable” mindset: “This isn’t what I want to be doing, but I must do it so I might as well learn to enjoy it.”

Sometimes, you consciously put up with discomfort in the short term for some longer term benefit. But when you’re younger, it’s mostly because you “have to.”

You have to go to school and get good grades! they say, or else you won’t get a job. (Not that this is even true anymore.) Just tough it out and things will magically get better!

But when? And how?

Growing up, life didn’t always work out how I wanted it, but I became mature enough to put up with a decent amount of discomfort. At some points, I became numb to it, and put a lot of my life decisions on auto-pilot. I stopped being choosy about important things in my life.

However, I would notice the discomfort with some of my bigger commitments, like my high school swim team. From a Facebook note I published in the fall of 2006 (I was a junior in high school):

I’ve [swam] for so long that I’ve gotten so used to all that it entails: smelling like chlorine, shitty hair, a pretty good body, constant appetite, no free time, shaving your head and legs, being on one of our school’s best sports teams but getting absolutley no recognition. My recent label as a ‘slacker’ has gotten me thinking as to whether I ever will get back in the pool on a regular basis.

I’ve been taking for granted all these things that define a swimmer, never stopping to ask myself if I was doing something I wanted to do. This might sound corney, but people have always asked me, “Why do you do it?” and I’ve laughed it off, never really being able to answer. Its time for that to change.

Adulthood

I bring all this up because I see a lot of other young adults who aren’t being choosy about important things. They’re still operating under the durable mindset, sacrificing their quality of life now for some vague future payoff:

I don’t like this job but I need to start my career somewhere.

Now, maybe the transition from durable to choosy isn’t college graduation for everyone. But it should happen at some point. You shouldn’t have to put up with your life.

There’s a famous Steve Jobs quote that says:

I have looked at myself in the mirror every morning and asked myself, ‘If today were the last day of my life, would I want to do what I am about to do today?’ and whenever the answer is ‘no’ for too many days in a row, I know I need to change something.

The Hard Part

I think the hard part for most people is knowing what to be choosy about, and when to grit your teeth and be durable.

Gary Vaynerchuck had a great quote in his keynote at Inbound 2012 when he said:

People nowadays put more effort into planning their wedding than they do their marriage.

We optimize the shit out of some things, and let other more important things go to waste. We’re choosy when we don’t need to be, and durable when we could have it so much better if we tried.

There’s a similar concept from Ramit Sethi in his awesome financial advice book “I Will Teach You To Be Rich“. The book is a great primer for young adults on how to be financially responsible. One of my favorite takeways from the book is what he calls “Conscious Spending“:

Don’t listen to experts telling you that you should stop [buying mp3s] or how you should brown bag a lunch. Think about your goals. Ask yourself if you’d rather spend $10 on lunch, or save $10 towards a house or car. If you would rather spend the money on lunch, by all means enjoy lunch! You save money so that you can spend it later on things that make you happy. You don’t save money just to watch your account balance grow.

In other words, be choosy where it makes you happy, and then suck it up and cut costs where you don’t mind it so much.

—

Of course, being choosy doesn’t suddenly make me immune to adversity. Tragedies still happen, there are obligations I must meet, some days just suck. I still need to be durable in many situations.

But it’s important to be aware of whether those situations are endogenous or exogenous, and make deliberate changes when things are going south.

My Career

This past week, I moved from the engineering team at HubSpot back to the marketing team.

A few weeks ago, I detected that my passion for my work was starting to wane. Things were progressing one way, when I had hoped they’d take a slightly different path. I wasn’t as excited about HubSpot as I used to be. Once I noticed it, it was impossible for me to ignore.

I talked to friends and mentors about it and pretty much everyone said the same things: give it some time.

Maybe you haven’t hit your stride yet. But you’re learning so much! It’s normal to hit a slump after 6 months in a position. It looks fishy on a resume if you don’t stick with things.

In a bunch of different ways, they were all encouraging me to be durable about the situation.

And I listened. I talked to my managers and tried to keep an open mind. Maybe the only thing that needed to change was my mindset. I could learn to love it.

But a few weeks later, things still hadn’t fully clicked. Should I keep trying to just stick it out? A lot of people were telling me yes. But I’ve found it’s usually better to zig when all signs are telling me to zag.

So I talked to my managers and coworkers about it and was able to keep my career from slipping into something that required me to be durable.

Fortunately, I had chosen to work with really awesome people who care about my happiness and career growth, which made the transition very smooth.

Now, I feel reinvigorated about my work. My job is something I spend half of my waking life thinking about. I didn’t want to have to put up with it.

—

On the one hand, life is short and you shouldn’t waste it with a routine that doesn’t make you happy. But on the other hand, you don’t get to live in your ideal world all the time, and you need to be durable if you’re going to survive life’s ups and downs.

The first 20 years of our life, society teaches us to be durable, but it’s up to us to move on from here now and be choosy. Always remember your durability when they doesn’t work out, but be relentless in choosing the things that make you happy.

When I woke up, my inbox was full of comments from people all over the world showing support and offering to help. Apparently, the post made it to the #3 spot on the front page and drew almost 10,000 views to the article.

One email was from a HN’er who goes by wavewash, or Mo in real life. He told me his company was offering to sponsor the search for the missing footage, using a remote control airplane with a camera to fly over the forested area and conduct an aerial search.

He and I went back and forth and, two weeks later (waiting out hurricane Sandy) we were on the road together with his girlfriend, driving up to Maine to continue the search.

The Search, Take 2

I brought along my trusty Garmin 60CSx as well as the new SPOT Personal Tracker I had just purchased, in preparation for a second balloon launch.

We used the second map I had made as our reference area. The map combined four different checkin points from the tracking device in the balloon’s capsule, as well as my own guesstimates of where it might have drifted due to the wind.

Unfortunately, the remote control airplane wasn’t quite ready for the trip, but we set out into the woods again, hoping that the lack of foliage would make it easier to search the tree canopies.

It took us about 5 minutes to get to the location on my map, and then we started to spread out.

Within about 90 seconds, I spotted a giant white flap of latex blowing in the wind, about 2/3rds of the way up a tall pine tree. I actually wasn’t sure if it was the balloon at first, or whether it was some sort of giant gross mushroom thing, but then I spotted the capsule and the lens of the camera dangling below.

I couldn’t believe it — I had finally found it!

Before I could even start planning how we’d get it down — it was about 40ft off the ground — Mo came crashing through the forest and started scampering up the tree to retrieve the capsule.

It took a minute or two for him to dislodge the parachute and what remained of the weather balloon and then drop it back to us on the ground.

Thanks to the Community

As we were walking back to the car, a homeowner came out to his porch and asked if that was the balloon he had kept hearing about. He said several groups had come by asking if he’d seen anything, and he was really excited that we recovered it.

It was awesome to know that the community had gotten so involved.

Huge shout out to Mo and his girlfriend for making the three hour trek up to Maine and back, and thanks to everyone in the Bowdoin and Brunswick communities that organized search parties or told their friends, and everyone on the internet who offered to help in some way. You all kept me dedicated to keep searching.

The Footage

The GoPro was actually pretty tricky to pry open. I’m guessing the waterproof seal tightened in the vacuum of the upper atmosphere. Eventually we got the SD card out and found a suitable laptop to play the video on (thanks Toph!)

While the balloon was in the air for over 4 hours from take off till landing, the camera was only able to recorded about 2 and a half hours before the battery died.

Unfortunately, this meant that it didn’t capture its highest moments in space or its crash landing into Brunswick.

But there’s still some magnificent footage:

It was a beautifully clear day so there were no clouds to shroud the view from the camera’s soaring heights.

The camera was pointed down at an angle in order to capture both the edge of space, as well as the view of the earth below.

I’m not entirely sure how high the camera was in its final moments, but I’m sure it could be calculated based on the distance between landmarks in the video. Let me know if you want to lend a hand with those calculations!

Do’s and Don’ts

I was inspired by Felix Baumgartner’s record breaking jump from the edge of space to finally follow my dream and try to reach space myself. Out of excitement, I rushed to put the launch together in a single week. Here’s what I learned:

You need redundant systems.
Tracking and video recording are the two crucially important systems on your capsule. If either one of them fails, you might not get full video, and you might even lose all your equipment.

I was lucky that my GPS receiver thawed out enough to check in a few times before the battery died. And because of the battery issues with the GoPro, I wasn’t able to record the entire trip…

It’s worth putting in a few extra pounds to ensure you can find your capsule and that it’s able to record footage the entire time. I’ll definitely be doing this for the next launch.

The upper atmosphere can be a pretty harsh environment for electronics.
It’s not as easy as you might think to track an object that’s tens of thousands of feet off the ground in temperatures well below freezing. The Garmin device I tucked into the capsule was pretty much useless — it only checked in 5 times during the entire 200mi trip.

For people that want to build their own hardware, there’s lots of information for hobbyists available. Check out the UK High Altitude Society for more.

To keep things warm, I wrapped the inside of the capsule with thick air conditioner insulation, and tucked in several hand warmers to keep the batteries from freezing. I should have included a lid on the capsule to keep more heat locked in.

You have to think about the safety of others.
I had several people point out that I made no mention of FAA or FCC regulations regarding this sort of activity in the last article. I didn’t include them cause I thought they’d be boring, but they’re extremely important to be aware of.

IANAL, but according to my reading of the rules, if the capsule is under four pounds and doesn’t use a cell phone for tracking, you should be in the clear legally.

That said, I also took care to avoid airports and other airspace that might cause issues as the balloon was being launched. One of my coworkers is an amateur pilot and he showed me http://skyvector.com/ which shows airports as well as other restricted airspace warnings. I didn’t want my balloon getting tangled with an airplane or causing any other issues.

While the sky might appear to be wide and vast, there are thousands of people flying over our heads every day, and it’s important to keep their safety in mind.

Be prepared for a long chase.
In all of my excitement and anticipation, I didn’t really know what was going to happen once we actually launched the thing. There were a few hours of radio silence that would’ve driven me crazy if my girlfriend hadn’t suggested we go for a hike and enjoy the foliage — leaving the tracking device in the car.

We ended up driving almost 400 miles that day from the launch site to crash site and back home. Prepare yourself for this mentally.

—

As always, reach out to me on twitter and I’d be happy to answer any more specific questions. Thanks for all the support!

For me, one of the best parts of the stream was the moment when the cabin finally depressurized and the door slid open. Suddenly, you could see both vastness of space and the bright blue curvature of the earth over his shoulder, and it took my breath away.

It had always been a dream of mine to see the earth like that, and watching Felix’s live stream got me so amped up, I couldn’t hold off any longer.

The Setup

I knew I wouldn’t actually be able to travel to outer space, but I still wanted to somehow capture the awe of it myself. I decided a video camera rigged to a weather balloon would be a great start.

I found a big weather balloon on Amazon for less than $50, a parachute for $30 and then assembled some supplies from the local hardware store to build the capsule. The video recording device was a GoPro HD Hero that I got for about $130.

But getting the rig to float across the sky and record video along the way was the easy part. Getting it to land safely and then recovering the camera would be the hard part.

I knew the balloon was probably going to ascend to almost 100,000ft (about 19mi up) before it would pop in the vacuum of space and the parachute would carry it in for landing. But I expected the rig to drift pretty far from wherever it was launched.

My biggest concern was that it would land somewhere in the Atlantic Ocean and I’d never be able to recover the video. Since the prevailing winds generally blow west to east, I knew we couldn’t launch from the Boston area or it’d end up in the sea for sure.

But knowing it wasn’t in the ocean wasn’t good enough. I found a balloon trajectory forecast site that gave me a pretty good sense of which way it would drift, but that still wasn’t enough precision to actually track it down.

It turns out it is very hard to track a device that is tens of thousands of feet off the ground, but the best solution I could find was Garmin’s GPS tracking device, which uses cellphone service to broadcast its GPS location, which could then be viewed from Garmin’s website or on a mobile app, in real time.

With all that and a bit of duct tape for good measure, we were ready to go.

Launch Day

Sarah and I hit the road at 6am so that we could get west to launch early in the morning, when the winds were calmest.

The launch site was a small Hannafords store in North Brookfield, which I chose because it opened early and had a “Bring Your Own Balloon” policy for their helium tank.

Most stores require you to buy the balloon from them — but I already had my balloon, I just needed helium. Their stated policy was 79 cents per balloon for the helium, but I offered to pay 20x since this wasn’t an ordinary-sized balloon…

We ran through the pre-flight checklist, and then, at 7:54am, released the balloon with its video-recording payload. It ascended quickly and disappeared in about a minute. And then, we waited.

And waited.

The Garmin device stopped updating its location in the app several minutes after it took off, either because it had risen too high to reach the cellphone towers, or else it was in an area with bad service. This was very frustrating since I had hoped to be able to watch it travel most of the way and try to follow along underneath it.

Sarah and I went for a hike while we waited, enjoying the fall foliage.

Afterwards, we decided we’d start to drive back east, towards Waltham in case the balloon showed back up on our tracking system.

I had heard the balloon trips often last between 2 and 3 hours before the rigs touch down again, so when lunchtime rolled around and the balloon still hadn’t shown up, I was ever more frustrated. I figured it had probably blown into the ocean and it was all a waste.

But then, at 12:05pm, an email: “You Garmin device is running low on battery” That means it had checked in! And if it checked in, it must have updated its location. I quickly pulled off the highway and checked the app.

It turns out, the rig had travelled up to Maine — almost 200 miles northeast — and was currently floating across the Brunswick High School baseball field.

Wait, Brunswick High School?? Like, 3 minutes down the road from Bowdoin, where I went to college? Yep.

It was pretty incredible that that’s where it’d end up of all places, and I knew exactly how to get there so we jumped back in the car and raced 2 and a half hours north, to Brunswick, Maine. We pulled off the road just across from the high school.

Right before the battery in the tracker died, it updated its locations say it had drifted over into the woods across the street, before coming to a rest (several check-ins in a row in the same spot).

Sarah and I headed into the woods, eager to find our precious payload and recover the video.

Alas, after 3 long, agonizing hours, it was starting to get dark, so we had to pack it up and head back to Boston.

You Can Help

Unfortunately, that means we never recovered the actual video that the camera shot on its 200 mile trip through the sky and into the edge of space.

Here’s as much information as I know about its whereabouts.

Nearest Address: 183-207 Maquoit Rd, Brunswick ME
Coordinates: N43.880485, W69.980874
Accuracy 16ft

I wrote my cellphone number on the side of the capsule before launch, in case anyone found it. But this thing is buried in the woods, so I doubt anyone will stumble across it for quite some time.

There’s also a chance that the video captured the meteor shower that took place over Massachusetts this morning, which would be absolutely incredible footage from that vantage point. I really really don’t want that footage to get lost in the woods.

If anyone is willing to search for it, please, please shoot me an email at hartley.brody@gmail.com I’d really love to publish the footage and give you a huge shoutout for your assistance.

And if you have any questions about sending your own mission to space, I’d be happy to answer with a few “dos” and a lot of “don’ts”: @hartleybrody.

Update: I’ve gotten a ton of interest from students and community members who are interested in helping me locate the capsule, which is really exciting!

Here’s some more information I put together to help searchers:

The orange ballon shapes are the places where the GPS checked in. The pink marks are the landmarks I noticed while hunting around, and the white area is my forecast of where I think the balloon might be. It’s possible that the rig hit a tree, knocking the GPS receiver out, and then kept travelling a little farther east.

Sarah and I searched the area around the 3 orange markers pretty thoroughly, and ventured into the ravine briefly, but didn’t go much further east, or down the ravine.

The parachute is a dark red color, about 2 feet in diameter. The capsule is a 1 quart cardboard bucket that’s got a blue and yellow and green design on the outside. It’s strung together with yellow parachute cord and yellow duct tape.

I chose the bright colors, thinking it’d help it stand out, but it just ended up blending into the fall colors :/

Update 2: More maps! I haven’t been able to find a good trail map for the area, but I found some cool stuff on the Town of Brunswick’s GIS website:

Get in touch if you’d like to help! hartley.brody@gmail.com

Update 3: WE FOUND THE BALLOON!!!

Back in the 1980s, Richard Stallman started the Free Software Foundation as a reaction to the growing commercialization of software. He framed the issue not as a commercial one, but as a moral one:

Thus, “free software” is a matter of liberty, not price. To understand the concept, you should think of “free” as in “free speech,” not as in “free beer”.
— Free Software Definition

To this day, many people still think of FOSS as a matter of liberty. Quoting again for the GNU project’s answer to “What is Free Software“:

When users don’t control the program, the program controls the users. The developer controls the program, and through it controls the users. This nonfree or “proprietary” program is therefore an instrument of unjust power.

Clearly the FOSS movement is grounded in morals and philosophy.

But for me, I don’t tend to moralize about software too much. Instead, I work with open-source projects and technologies for more practical reasons. I’ve found they tend to be more well-documented, efficient, secure and easily extendable than their close-sourced counterparts.

If you’re starting a project or learning a new language — or just choosing software as an end-user — those are all great reasons why you should consider an open-source option.

—

The basic tenet of open-source is that anyone can see the underlying source code that makes a program run. This means that anyone can not only read and study it, but also (usually) modify it and redistribute it as they wish.

Thus, open-source software is exposed to everyone’s eyes. And — like a rock eroded by millions of grains of sand over time — this exposure helps polish it off and smooth over many bumps.

Everything is Accessible

With proprietary software, you’re relying solely on the original authors of the application to give you a full and accurate description of what it is and how it works. But even at a well-meaning organization, documentation often gets out of date or is incomplete — or sometimes flat out wrong (real-world example discussed later).

If you encounter strange or inconsistent behavior in a platform like that, your only option is to file a bug or see if anyone else has posted the same issue in a forum somewhere and gotten an answer.

With open-source software however, you can roll up your sleeves and dive in yourself. This empowers you to not only study the software and fix bugs, it also allows you to change existing functionality if it doesn’t suit your needs out-of-the-box.

Documentation is Reliable

As a corollary to the last point, open-source projects usually have excellent documentation. If someone discovers an issue or a feature that isn’t well described in the project’s documentation, they can usually go into the docs and update them to be more accurate.

This is a great example of how increased scrutiny, combined with the ability for anyone to edit the project, makes open-source software much nicer to work with than closed-source projects.

Features are Mature

With open source projects, you rarely have to worry about small bugs and edge cases that the original application developers might not have tested for. Since the source code has been exposed to the public, buggy or insecure behavior is quickly rooted out and corrected.

Even for larger projects that might have several dozen core contributors, those contributors aren’t always specialists in the minutiae of the project. By releasing the source code publicly, people with more experience and specialization have the ability to review the code and even correct issues the original developers might not have noticed.

Anyone can essentially audit the project, which ends up making the code more efficient and secure.

Addons and Plugins are Plentiful

Open-source projects are easily extended by third parties to add on functionality that the original authors didn’t include.

WordPress is a great example of this — they have over 1,600 themes and 21,000 plugins available for their system, while closed-source blogging systems like Posterous and Tumblr have only a few hundred themes and a very limited plugin selection.

It’s worth pointing out that the movement towards Service Oriented Architectures and exposing public APIs is allowing third parties to extend even closed-source systems. However, in these situations, third parties are relying on the closed-source provider to maintain those APIs consistently over time. And as we’ve seen, this isn’t always a realistic expectation.

No Sales People or Commitments

Obviously, one of the best best benefits of using open-source software is that it’s almost always free (“free” as in “free beer”). This means there are no contracts to sign, no commitments to enter, and no sales people to wrestle with.

Just find the project’s website, download the software or clone the source code, and you’re on your way. You can get up and running quickly.

Support is There, But Not Required

For larger, more complex open-source projects, you can usually pay if you need help or support, but it’s not required to use the software.

This means it’s still possible for a profitable business to form around fundamentally free software. Red Hat is the most obvious example of a company doing this right. They offer premium support for Linux, and a bunch of other open-source technologies, and have revenue of over $1Bn.

—

A Counter Example

I work with open-source so often that I often take all of these wonderful benefits for granted. I usually build projects using Python, Django and MySQL — all open-source — and so I’m used to great documentation (or tutorials where the docs are lacking) and libraries and apps I can use to extend core functionality when I need them.

Recently however, I’ve had to work on two projects using Google’s closed-source App Engine.

App Engine is Google’s cloud-computing platform for hosting web applications. However, in order to run your application on their platform, you need to build it specifically around their APIs.

Now, you might think that a company like Google, with some of the best engineers in the world, would be able to develop a fantastic set of APIs and offer world class tools to help you monitor, manage and debug your application.

But even with all that tech talent working on the project, I’ve found developing for App Engine to be slow and much more frustrating than any of the open-source technologies I use more regularly.

Their documentation is poorly organized and vaguely written. I’ve even seen code samples in the documentation with incorrect syntax — they wouldn’t even run if you tried using them! I’m sure they were typos by the engineers who wrote them, but those sorts of issues get fixed very quickly in an open-source environment.

Using common Python libraries like simplejson is unsupported. Finding proper tutorials is tricky. Querying for objects can lead to surprising results (although that might just be my nascent familiarity with NoSQL). Getting support from Google is all but impossible.

Plus, there’s the vendor lock in. If I ever wanted to take my application and host it somewhere else, I’d have to rewrite huge parts of it in order to make it work with more standard databases.

Working with closed-source App Engine has helped me realize all of the wonderful benefits that I was accustomed to when working with open-source alternatives.

—

What it comes down to is this: no developer is perfect. No matter how many bright engineers you have under one roof, they’re never going to be able to test every use case, add every features and support and explain the software exhaustively.

Just as the rough rock gets polished into something smooth and beautiful by exposure over time, scrutiny from a huge community of dedicated developers and part-time hobbyists can turn a project into something really elegant and smooth to work with.

That exposure helps to make open-source projects easier to learn and generally more efficient, secure, easily extendable than their close-sourced counterparts. To me, that makes open-source technologies the most obvious choice.

You’re a lot less active.

On campus, I was always walking or biking around. Not only cause I enjoyed it, but because I had to. Every class, every meal, and every study group or party meant walking at least a few hundred yards. And I’d rarely be in one place for more than two hours, so I’d end up walking or biking a pretty decent amount every day, without meaning to.

Now, on an average day, I only have to bike to work (about a mile) and then back home. I spend hours sitting at a desk during the day, and then again sitting on a couch when I come home. This is actually something I was warned about by a recent alum. It’s easy to get stuck in one spot for hours.

Some people try to compensate for their new found sluggishness by joining a gym. These are never cheap, and usually not as good as the world class facilities we had available to us as students, which makes them feel like an even worse deal. Personally, I do calisthenics in the morning and sometimes go for runs or bike rides on the weekend.

You have to really go out of your way to stay active, or else you feel like shit.

—

The housing process sucks.

No more complaining about getting stuck in Chambo or having your housing block break up so you couldn’t get that quad you wanted. Even if you end up with so called “shitty” housing on campus, you know it’s still halfway decent. The real world has no such guarantees.

First, you have to pick a neighborhood, which means getting to know a large area well. Is it safe? Is it noisy? Is it easy to commute? Are there grocery stores nearby? Where can I do laundry around here? How expensive is it compared to other areas? Do my friends live in the area?

Then the hard part — finding an actual apartment and roommates who also want to live there. People disagree over cost and location and a bunch of other things, so rather than optimizing for living with people I actually knew, I optimized for living in the best spot I could find, and then found roommates through Craigslist.

It was an extremely stressful process, even though I already knew where I wanted to live. My new found Craigslist roommates and I spent weeks going back and forth over who was paying what rent, and people threatened to pull out and not sign the lease, which would have left the rest of us stuck covering their rent.

The stars have to align just right for you to find a nice place, in a good area, at the right price, with great roommates. It’s very hard to get it right.

Your friends get married and have babies.

Until recently, I had never actually known another young adult as a peer who then went on and got married or pregnant.

It was very strange at first to watch a friend transform into a spouse or parent and be responsible for another person. But, in an office that has a lot of young adults, it seems like these transformations are happening every other week. I think I’ve gotten used to it by now, but it was weird to experience the first few times.

You’re totally responsible for your own livelihood.

I’ve always been someone who likes to save money and am generally responsible with it. But it’s a whole new ballgame when you are totally responsible for 100% of your own expenses. Everything from groceries and rent to vacations and unexpected bike maintenance — it’s totally up to you to cover those bills and save responsibly.

To grow your savings, you need to either earn more, or spend less.

This transition is harder for some people than others.

Getting drunk is expensive.

While bars might seem exciting at first, they get expensive really fast.

Last summer while living in Boston, there were months that I’d spend upwards of $200 at bars, and a few times where I’d drop that in a single weekend. Most people consider it a deal when you can get a single beer for $4. Fortunately, most places in Boston don’t charge for cover, but some do, and those little charges add up quick.

Plus, add on an additional $20 for the Uber fare you’ll need to get home after the T is closed (past 12:30). Drinking is a much more expensive habit than it was in college, and I’ve found myself doing a lot less of it as a result.

No anxiety in the month of August.

For as long as I can remember, the month of August meant the end of summer and back to school. This meant summer reading, back to school shopping, and saying bye to summer friends.

Now, the month of August just means — enjoy yourself! It’s almost time for fall. August no longer brings huge living shifts like it has for the past 17 years.

Groupons are a lifesaver.

You want to do fun things, and have an interesting life, but you don’t have a ton of cash to burn. You absolutely need to subscribe to Groupon and any other daily deal sites. They’re a great way to explore a city and try new things without blowing through your entire paycheck.

So far I’ve gotten handgun lessons, a dinner cruise, salsa classes, and a few laps driving a Lamborghini — all for more than 50% off thanks to daily deal sites. It’s a great way to keep yourself busy for cheap.

—

There are all sorts of other things like learning the true meaning of “work-life balance” or finding out that the liberal arts degree you worked so hard for is essentially worthless in a Get Shit Done™ economy.

What are some things you didn’t know coming out of school?

It’s been a while since HubSpot has made a fun video. The last one I was a part of was actually one of my first projects at HubSpot last spring.

Huge shout out to HubSpot’s awesome video team and to all the background dancers (both the volunteers and the unwitting).

I’m glad I get to work for a company that has this much fun.

.

Now, you don’t want to go too far down this route, or you’ll end up designing yourself into oblivion. As Donald Knuth famously said:

Premature optimization is the root of all evil.

But it’s still good to have a sense of where your system might get bogged down under heavy load. For example, database calls are often the first place you’ll start noticing slowness in your app. Or if you have to make a network request — say, hitting a third party API — that can add more than a full second to the amount of time it takes your application to serve a response to a visitor.

Break It Down

I’ve heard it said that good code should only serve a single, specific purpose. If you can take a given process and break it down into two or more simpler processes, that’s usually the best way to go.

I always kind of knew that was a good design goal, but I didn’t realize how powerful it was until I started playing around with message queues.

Analytics for Noobs

I recently started working on a new side project and one of the core parts of the application is an analytics system, that shows artists how their fans are interacting with their music. That might not sound too challenging — after all, it’s just writing lines to a database.

But the reason this challenge seemed particularly daunting was because the service would likely receive hundreds or even thousands of bits of analytics information coming in each second during peak load.

I had never designed a system that operated under that type of environment.

I had learned a few lessons in scalability when my blog had made it to the front page of HN for a few hours, but that was expensive and resulted in lots of downtime.

If this was going to be a serious service that paying clients were relying on, I had to do much better.

“Log Now, Process Later”

At HubSpot, the analytics team has built an incredibly robust analytics pipeline that aggregates information from all of our 7000+ customers’ websites. I talked to a few engineers who work on the system to see how it was built.

At a high level, we log every analytics request immediately into a log file. Then, there are separate processes that constantly chew through the log files, pulling out the data and saving it into our analytics database.

This ensures that we never lose any data — if something goes wrong with the processors or they get overwhelmed with a sudden burst of traffic, we keep on logging and simply spin up some new processors to chew through the logs faster and get everything up to date. Data is never lost because of a bug.

This separates the serving of our customers’ websites from the actual tracking of that traffic. This allows us to serve the sites as soon as possible, and then process the analytics data asynchronously, so that visitors aren’t waiting for our analytics system before they can see the page.

Do as little as possible to serve the page, and do the rest asynchronously.

The Speed/Reliability Tradeoff

This all sounded great. It allowed for fast page load times with no need to hit a database at all during the request/response cycle. The only problem with it was that the processing of analytics data was slow.

While we never lost any data for our customers, it would take minutes — occasionally, hours — before the data our customers were seeing in their dashboard was in line with the actual traffic patterns on their websites. Chewing through logs and rotating them constantly was slow and surprisingly error prone, so our system was always a bit behind.

Again, I don’t actually work on these systems at HubSpot, so I can’t speak to their actual complexity, but this is just what I observed from talking to a few people and keeping an eye on some of our internal dev dashboards.

This is all in line with the trade-offs proposed in the CAP Theorem. Conjectured in 2000 by a UC Berkeley professor, the theorem states that distributed computer system can only provide 2 of the 3 following things:

1. Consistency: data that shows up in one part of the system is immediately available to all other parts
2. Availability: the system always responds to every request
3. Partition tolerance: the overall system doesn’t lose data, even if parts of it fail

HubSpot’s system is essentially focused on #2 & #3. And this is in line with the business goals of the software: traffic and leads information is crucial for our customers — we can’t lost any of it — but our customers don’t necessarily need that data in real time. So we have developed a system that is slower, but reliable.

But it got me wondering — what if I conceded #3 in favor of #1? That is, I was willing to lose some data here and there if it meant I could report analytics to my customers in soft real-time. What would that system look like?

Message Queues

In a stoke of impeccable timing, the CEO of Crashlytics was scheduled to give a presentation called “Scaling Crashlytics: Designing Web Backends for 100 Million Events per Day” at the local Boston Tech meetup later that week.

I attended eagerly, and listened as he explained how his company uses RabbitMQ to process millions of events each day, and send those crash reports to their customers in real time.

I had heard of RabbitMQ before, but it had sounded complicated and scary. But after his talk, I checked out the well-written tutorials and realized how powerful RabbitMQ — and message queuing in general — could be.

Essentially, RabbitMQ is an open-source Message Queuing service. You can take a “message” (any arbitrary data), put it in a queue, and then have one or more “workers” that pulls messages from the queue and processes them — putting them in a database, for example.

RabbitMQ is written in Erlang — a language designed for fault-tolerant, ultra-reliable telecommunications networks. The CEO described how — in his experience — it had extremely low memory and processing requirements. One box could easily handle the volume of requests I was expecting.

But the best part about using RabbitMQ is how easily it scales.

Queues can support an unlimited number of workers, and each worker goes off and does its work independently of the other workers — they can be on totally different machines. This has a number of awesome benefits:

• Tasks are done in parallel. Rather than doing 10 slow operations in a row, you can have 10 workers each doing the operation at the same time for a 10x speed increase.
• Workers are isolated. If there’s an unhandled exception while one of the workers is doing it’s thang, the other workers can keep chugging right along, pulling messages out of the queue as if nothing’s happened.
• Messages aren’t lost. If something goes wrong while a worker is processing a task, then that message gets rotated back into the queue and passed along to the next worker. The queue can even be setup so that if the machine it’s running on dies, the messages that were in the queue at the time can still be restored.
• The number of workers is flexible. If there’s a big burst of activity or you’re not processing through your queue fast enough, you simply bind more workers to the queue and your throughput will increase.

For the analytics pipeline I’m building, these are all fantastic benefits, especially that last one. The fact that I can easily dial the number of workers up or down means I can start out small. I don’t need to have lots of machines running workers when there is little activity on the site. I just need one or two for most of the time, and then I can add more as the site grows over time, or if a sudden burst of activity starts to fill up the queue.

Using a message queuing design for my analytics pipeline means that the analytics processing and storage happens asynchronously, freeing up my web servers to focus exclusively on serving web pages.

By dividing up the services like this, it’ll be easier to work on, test, and scale each component of the project separately, in accordance with its own specific resource needs.

A One-Off Project

With all of its awesome benefits, you might think that RabbitMQ is hard to setup or configure. Fortunately, it’s not.

Last week, I was working on a project in the office where I needed to lookup several thousand companies’ websites and run them through our Marketing Grader API. At first, I dashed off a quick python script that took the list of companies, looked up their websites and then ran the websites through Marketing Grader one at at time.

Twelve hours after it started running, the script was only about half way through the list when it stumbled on a weird character in one of the company names and crashed.

It dawned on me that the slow, potentially buggy API calls would work much better if they were setup as a message queue system. I setup some workers that would take a company name and lookup the website, and other workers that would take that website and look up the Marketing Grade.

If the API calls timed out, or returned some unexpected responses, a worker could be restarted gracefully without having to restart the entire process from the first company.

With 12 workers running, my desktop got a little hectic, but the script ran through the full list in under an hour. It could have gone even faster if I had spun up more workers.

Using message queues to separate out each step of the process was not only much faster and more reliable, but also really simple to setup.

You can checkout the script I used here: get_inc_5000.py

Asynchronize All The Things

A well designed web application should do as little as possible to serve the visitor’s request, so that it can return a response quickly. Any processing, database hits or network calls that aren’t necessary for serving a response should be done asynchronously with a message queuing system like RabbitMQ.

And even outside the context of web applications, message queues allow multiple jobs to be run in parallel and are massively horizontally scalable. Whenever you find your code getting tripped up on a slow bottleneck, it’s worth thinking about whether it makes sense to pull that process out and run it asynchronously.

They don’t necessarily make sense for every task, but it’s a great tool to have in your tool belt, and one that I’m glad I’ve picked up.

And while NBC has done a notoriously bad job at keeping the production fair and balanced, it’s still pretty easy to tell that some countries are pulling in way more medals than others.

Currently, the United States has the most gold medals while China has the most overall medals and Great Britain is third in both categories. But these totals are changing every few hours, so check out the latest numbers here.

Seeing the same countries on the podium made me curious: are there patterns between the countries that bring home the most medals?

Specifically, if you controlled for the size of a country’s economy, would you be able to predict the number of medals they’d win?

Could Olympic prowess be a leading indicator of a country’s financial success? Or maybe it worked the other way, and a growing economy was a sign of increasing athletic competitiveness?

The questions were piling up and I needed to find answers.

—

Collecting Data

I knew this wasn’t going to be as rigorous as an academic paper — it was just scratching an itch for me — so I started my data collection on Wikipedia.

I build a python scraper that pulled data from Wikipedia’s “Summer Olympics medal table” for every year back to 1980. I collected both the total number of medals, as well as the number of gold medals for each country in each year.

I decided to skip Winter Olympics since many countries are at a decided disadvantage because of their climate. The games of the Summer Olympics didn’t seem to favor any particular regions.

Also note that were Olympic boycotts in both 1980 and 1984, so the data set had some holes.

Once I had that data, I needed a way to control for the size of a country’s economy. I needed time-series data, but I couldn’t find it on Wikipedia, so I turned to the World Bank instead. They had exactly what I needed: Real GDP for every country and year back to 1980.

It took a bit of work to re-arrange the data in a way that I could plot it for each country over time, but once I had it all laid out in excel, I started making graphs.

Results

The raw numbers were hard to draw meaningful conclusions from — imagine a graph whose Y-axis has numbers in the teens as well as the tens of billions — so I converted everything to percentages and tried again.

For countries that occasionally won a medal or two, there was too much variance to really notice anything meaningful.

In an Olympics games, about 900 medals are awarded (300 of which are gold), so a country who picked up six medals one year and only three the next would see their total medal percent drop from 0.67% to 0.33%. However, most GDPs for countries in this range were way more consistent, fluctuating by only tenths of a percent.

But when you start looking at some of the bigger countries with more consistent medal winnings, some patterns emerge.

I originally imagined the percentage of gold and total medals would trend up or down for a given country over time, however it was fairly consistent, even across decades.

Unfortunately, this meant that it was hard to detect any interaction between GDP and Olympic dominance — with the usual caveat that correlation wouldn’t even imply causation anyways.

However, what I did notice was that some countries continually over-performed at the Olympics, while others continually under-performed.

By graphing the percentage of medals a country wins against their percentage of world GDP, it was easy to see which countries were more dominant athletically than they were economically, and vice versa.

Surprisingly, the United States has actually been under-performing at the Olympics, bringing home only 10%-15% of the medals, compared to our 25% of the world’s GDP.

China has been performing just about as expected, bringing home more and more medals each year as their economy grows.

Other countries to note were Australia, who has grown increasingly dominant at the Olympics while their economy has remained largely stagnant.

Also Kenya, who have consistently brought home about 1% of the medals despite having a teeny tiny economy.

Further Research

At this point, there’s still a lot than can be done with this data. For example, what if you also looked at trends in population, or land mass? Would larger countries bring home more medals, or is GDP more closely related to Olympic performance than size?

It would be cool to find some metric that seemed to trend closely with Olympic medals winnings. Then you could potentially make predictions about that metric based on how a country performs every 4 years, or conversely, predict a country’s medal winnings based on that metric.

But as you can imagine, the athleticism, determination and skill required for each Olympian to bring home a medal is largely determined by their own personal circumstances, and not necessarily those of their country in aggregate.

What do you think, does this surprise you?

You should download my collected data here and tweet me if you find anything interesting!

According to Wikipedia a bigot is (emphasis mine):

a person obstinately or intolerantly devoted to his or her own opinions and prejudices; especially : one who regards or treats the members of a group (as a racial or ethnic group) with hatred and intolerance

Chick-fil-A

Today, Boston’s mayor — Tom Menino — made quite a stir on the internet when a letter surfaced that he had written to the President of the Chick-fil-A restaurant chain.

The story goes something like this: The President of Chick-fil-A, Mr. Dan Cathy, had recently conducted an interview with The Baptist Press where he had espoused homophobic beliefs, saying (among other things) that

“We are very much supportive of the family — the biblical definition of the family unit.”

In response, Menino wrote an open letter essentially saying that the Chick-fil-A franchise was no longer welcome in the city of Boston.

Now — let me be clear from the outset — I believe that any kind of discrimination is fucked up. And I think that most of my friends would agree.

But this piece has nothing to do with my personal opinion on gay rights, or anyone’s rights. Instead, I want to talk about tolerance.

Tolerance

Some time ago, a good friend of mine was falling into a romantic relationship with someone that he had grown to care very much about. She was a smart girl from a great background and they made each other very happy.

One evening, they were talking about formalizing their relationship and being more than just friends, when she hesitated. “My parents would be really disappointed if I told them I was dating you… I mean, someone who wasn’t Jewish.”

He was taken aback. He couldn’t believe they would reject him simply because he didn’t share their religion. Here were very intelligent people actively discriminating against him because of their beliefs.

Did that really just happen?

An Ideal World

Every person holds their own opinion of how the world should be. But problems can arise when those idealistic worlds run contrary to someone else’s.

If I believe the world should be one way, and you believe it should be the exact opposite, then we both have two courses of action to choose between, in a sort of prisoner’s dilemma. (Can you tell I was an econ major?)

We can either choose to try and correct the other person’s way of thinking, or we can choose to be tolerant of their beliefs and go about our business.

	My Choices
Your Choices		I tolerate you	I persecute you
	You tolerate me	me: you	me: you
	You persecute me	me: you	me: you

The problem is, if I choose to be tolerant and you choose to persecute, then I lose. And so, to correct for this, we as a society have come to make intolerance very taboo.

It’s extremely awkward now for someone to make a claim against a group of people. Some demographics are protected by stronger taboos than others — but in general — there is strong backlash whenever someone tries to publicly persecute a group of people.

This attempts to steer all players towards the most efficient outcome — where everyone is tolerant and can go about their business, free from any sort of persecution.

The problem is, when that backlash goes too far, we as a society actually end up moving to the least efficient outcome, where everyone is persecuting everyone. We end up fighting intolerance with intolerance.

If you say that I’m not welcome in your restaurant, and I say you’re not welcome in my city, who is really better off? No one.

As human beings, we try to form communities with other like minded people, since that’s where we feel most comfortable. But from within those communities, it’s very easy to lose perspective on other people who hold different ideals.

I’m certainly not advocating for intolerance, and I think it’s pretty indisputable that society would be better off if everyone was more tolerant of everyone else.

But fighting intolerance with more intolerance only escalates the feeling of superiority both groups hold, increasing the conflict.

“Sorry but you don’t hold the right beliefs to operate in our city. Bye!”

That seems pretty intolerant to me.

Now I admit, when I first saw the letter on Facebook, I got really excited and shared it on my wall. It seemed like all of my friends reposted it or liked it or supported it in some way.

Here was an icon of our city giving a big middle finger to people who are philosophically intolerant to many of my friends’ lifestyles. It was cool to show support and do the right thing to fight intolerance.

But were we really fighting intolerance? Or perpetuating it?

Suddenly, I’m not so sure…

—

Now I admit that this is a bit of a knee jerk reaction to this particular situation, and I haven’t spent a ton of time formalizing this argument, so I’d love to hear your thoughts and feedback. Tolerance — by definition — deals with sensitive subjects, so let’s try extra hard to keep things civil and constructive.

Pretty obvious, right? So if we want to accumulate more money, we have to either earn more of it, or spend less of it. Still with me?

The Power of Wealth

In my college economics classes, we rarely dealt with the topic of “money” directly, it was always just a worthless medium of exchange. People don’t really derive value from money itself, but from the things it helps them buy.

But in the real world, that’s not exactly true. Having a small fortune gives you peace of mind if you lose your job or want to take time off. It allows you to purchase things when you want, rather than having to bounce from pay check to pay check. And if you accumulate money for awhile, you can purchase big ticket items like a house or a car.

We all know that we should be saving money, but there are all sorts of reasons we don’t.

As a recent college graduate myself, I still vividly remember getting my first pay check and the feeling of seeing a few extra Gs in my bank account. I suddenly had all sorts of spending impulses.

I could go out for dinner every night. I could open a tab at the bar and put all of my friends drinks on it like a boss. I could get a new bike or a bunch of new clothes.

Fortunately, it was easy to rein in those impulses and put money aside — all without having to turn into an obsessive penny pincher.

Here’s how I do it.

Spend Smart

The first thing that’s important to realize about saving money is that most people suck at it. That’s not just me and you, that’s your friends and your boss and your coworkers and maybe even your parents.

Fact is, the average American has a negative savings rate. And if you read the news, it seems like every week there’s some new financial scandal that hurts “average” Americans.

If you take your spending cues from the people around you every day, chances are you’ll get pulled towards a lifestyle of poor savings. Especially if you’re trying to punch above your weight class and regularly hang out with people who are older or more senior to you, and who probably make more money than you. If you spend as much as them, but make less income, I guarantee you’ll be heading down the wrong savings path.

If you want to save up your first million dollars, you need to commit to being far above average, and that means learning to say no when the people around you are buying lots of thing.

Now you might complain and say “But Hartley, I have this money that I worked so hard for. Don’t I deserve to spend it? Isn’t that what I earned?”

And my answer is — you’re absolute right! Find the things that you care about, and by all means, spend money on them. Life is short and you shouldn’t waste your early adult years feeling like a hermit just so you can save up a small fortune.

But in order to live a fulfilling life while still saving money, you need to cut spending aggressively on the things you are kinda “meh” on.

For me, I know that I’m not a huge foodie, so I’m super cheap with groceries and I avoid restaurants like the plague. I could probably afford my own place, but I share a small room in a great part of town, and I’m totally happy saving a few hundred dollars on rent each month.

If you actually sit down and look at every dollar you spent over the last month, you’ll find lots of places where you are casually spending $50 here or even $10 there that didn’t really make you happy. Do you really need that Netflix subscription, or would you pay less by renting 1 or 2 movies a month on iTunes? Are you really using that gym membership?

Those are perfect places to start cutting spending.

As I mentioned, personal finance all comes down to the money we make, the money we spend, and the money that’s accumulated in between. If you want to accumulate more money, you either need to make more, or spend less. That’s it.

I’ll probably write another post about ways to make more money if you really wanna hustle for it. But for now, start trying to find ways to decrease your spending, and you’ll already be thinking smarter than 90% of your friends.

Saving The Right Way

If you can find ways to cut $100 of spending each month, that’s fantastic. You’re already doing very well.

Now the trick is to make sure that you’re doing the right thing with that money, and not simply wasting it on something else.

For this, I’d highly recommend opening a savings account — or several savings accounts — with your bank. Personally, I use ING Direct because they have a great online banking interface, high interest rates, and it takes literally 15 seconds to setup a new savings account.

It’s important to put money aside in a separate savings account — rather than your main checking account — for several reasons. Note that I won’t say it’s because you earn higher interest with a savings account — the interest on most savings accounts barely keeps up with inflation, so don’t think that you’re actually “investing” when you use one.

The first reason to use a savings account is that it makes your savings harder to spend. The money is still there and you can get it if you need to in an emergency, but you can’t withdraw from an ATM or write checks against it (usually). This helps protect your hard work from sudden impulse purchases or regrettable late night spending decisions.

Using a separate savings account also gives you the benefit of being able to watch your savings grow over time. If you know that you’re being frugal and cutting costs, but all you see is a single nebulous checking account, it can be easy to lose motivation and slip into old habits. By breaking your savings out into their own separate account, you get to watch it grow over time. And if you’re a money nerd like me, that’s pretty exciting.

It’s even more exciting when you set savings goals. ING Direct allows you to give each account a nickname, and I’ve labeled mine things like “Travel and Adventure” so that I know what I’m actually saving for in each account.

I know that when I get enough in that travel account, I get to reward myself by booking an awesome vacation that I know I’ve earned. I don’t have to feel bad about it since I’ve been saving regularly. It’s guilt free spending at it’s finest!

Don’t Even Think About It

So now you’ve cut your spending and maybe even come up with some savings goals, here’s the linchpin of this entire strategy: you need to automatically transfer a consistent amount of money to your various savings account each month.

Don’t tell me you’ll do it yourself. You won’t. You’ll forget, or you’ll find something you just had to spend it on, but you’ll ‘catch up’ next month. And then you won’t.

Managing your money sucks, so the less time you have to spend doing it, the better. And if you can automate your ideal habits, you’re not only saving yourself time each month, you’re ensuring you stick with you plan, which is 95% of the battle.

If your paycheck comes on the 15th and 30th of each month, you should setup your savings account to automatically pull in money from that checking account on, say, the 1st and the 16th. That way the money is whisked off to your savings before you even think about spending it, and you can ensure you’ll always be saving, without even thinking about it.

Note that this assumes you have direct deposit setup with your employer — which you should. You’re not still going to the bank to deposit checks, are you??

It also assumes the money arrives on time, every time. If your pay isn’t as consistent, then maybe wait a few extra days between when the check is expected to come and when the money is transferred to savings, to ensure your bank doesn’t start a transfer when the money’s not there yet.

Thanks to Jonathan Kim for pointing this out in the comments

Again, ING Direct rocks at this, making it super simple with their “Automatic Savings Plan”. (Have you still not signed up for an account yet??)

If you’re worried about running out of money, start with a very small amount. Maybe only $25 each transfer (ie $50/mo). But just get it going so that you get used to the system.

Over time, you can adjust the amount that’s transferred if you want to save more aggressively. But if not, that’s cool too. Just start somewhere.

In the words of some ancient Chinese philosopher dude:

A journey of a thousand miles begins with a single step.

Save a little bit now, and find ways to cut even more spending over time. As you watch your savings pile up, it becomes almost like a game. How much can you save?

—

So now you’ve found a way to cut spending and start accumulating more money each month. The next step is deciding what to do with that money once you’ve set it aside, but I’ll get to investing in another post.

Subscribe via email so you don’t miss it!