Moving from Amazon S3 to Netlify

I recently began using Netlify for some of my side projects. Netlify is a modern web hosting provider that integrates a mix of offerings like a fast CDN and customizable build system. You specify how your website or project should be generated (ideal for static site generators like Jekyll) and all you do is run a git push when you're ready to publish.

Netlify will automatically generate and deploy your site, optimize assets and images, provide access control and more. It also brings some desired functionality to static sites like built-in form handling and server-side analytics.

Netlify website settings

While I was only using it for a basic landing page, I quickly began to admire how convenient it made the publishing process. Long story short, I moved this website to Netlify from my previous setup utilizing Amazon AWS S3 with CloudFront.

My previous AWS S3 + Cloudfront setup

I have been running this site for over 15 years now. The first few years I had a WordPress blog that I served on a bunch of devices ranging from a G4 Mac Mini in my dorm to a shared server and then to a VPS. Several years into that I wanted to change how I was publishing. I hated constantly updating the software due to security issues, keeping my VPS operating system and packages up to date and dealing with MySQL database backups (one of my early articles was about writing a bash script to save a WordPress MySQL database dump to Amazon S3).

I migrated my WordPress blog to Jekyll in late 2010 as part of my desire to simplify some of those performance and maintenance concerns. With Jekyll I had a site made up of fully static files and no database to worry about. I hosted my Jekyll site with Apache on an Ubuntu VPS for a while then landed on a Heroku setup with Rack::Static for a few years.

I eventually ended up hosting my site directly on S3 with CloudFront and AWS Route53 for DNS. I favored this setup for its pure speed and lack of a real server I would have to manage in any capacity. Having everything on your site served at the edge directly from a CDN is a big deal. I remember receiving a few emails from readers outside the U.S. saying they noticed a performance boost on my site.

But the S3/CloudFront route felt like a hack. I relied on a Ruby gem to manually configure redirects (years ago I changed my permalink structure and have about a thousand redirects I need to keep around for old links) as well as invalidate CloudFront files each time I updated them. I also had to manage the site generation myself before deploying. Not a huge deal but it did mean I had to ensure I ran a few commands in the terminal in the correct sequence before deploying, and hanging around the terminal to ensure everything worked properly. I knew I needed to find a better solution when the gem I used for S3 deploys went into an unmaintained state.

Why didn't I migrate earlier?

I first heard of Netlify several years ago. It seemed interesting but honestly I was not comfortable relying on a product from a relatively new startup for something as critical as hosting my sites.

Netlify the product kept getting better and Netlify the company kept growing, both in terms of employees, funding and customers. It was time to take a closer look.

At first glance I didn't think Netlify was anything special. A hosting service that builds your site/app with a git hook before deploying was nothing new. And I did not even want that functionality at first. When I work on my long posts I commit to git dozens and dozens of times before publishing. I want my drafts backed up, but I also didn't want to have to bother with a separate git branch for example.

Fortunately, there are several ways to manage your deploys with Netlify. Though for now I do have the default deploy-on-git-push behavior.

Moving to Netlify

The first step was authenticating my GitHub repository for this blog and seeing if Netlify could build it and deploy to a temporary site (Netlify issues you a subdomain of your choice on their netlify.app domain). This part took some trial and error with my Jekyll setup.

First, I needed to ensure all the Ruby gems I relied on were in my Gemfile so Netlify could install from that. I had to change a few things there. Second, I needed to find out how to get my SCSS and Coffeescript/JS generated. I figured I could just add the commands I usually run inline with the regular jekyll build command.

This part took some more trial and error. I couldn't get it to run my Grunt (JavaScript task runner) commands. I used Grunt to compile, optimize and minify my SCSS, Coffeescript and JS files. For SCSS, I decided to move away from my Grunt setup and just use the jekyll-sass-converter Jekyll plugin to handle this for me easily. As for the Coffeescript, I used this as an opportunity to go back to JS files—something I had been wanting to do.

But that still left me with the task of how to concatenate and minify my JS files. Netlify has built-in asset optimization to do exactly this. It detects a series of script tags in your markup and bundles them all together to serve as one versioned file. This worked at first, then I realized that it wasn't actually minifying the JS. I'm guessing it's an issue on Netlify's side after seeing someone else with the same issue, so I'll have to revisit that later. For now I still have to compile the JS locally before deploying whenever I change it.

Netlify asset optimization settings

The last thing I had to figure out was how Netlify handled redirects. There are two ways to configure redirects on Netlify, either with the netlify.toml config file or a separate plaintext _redirects file. I went with the latter as the format was similar to what I was using to manage URL redirects with the s3_website gem. It only took a few find and replace all commands to the redirects file formatted properly.

I want to revisit the way I'm doing redirects (over 1,000+ for individual old articles when I changed my URL format years ago) and use a one-liner as described in the documentation. Something like this:

# I have lots of redirects like this for older posts
/2005/09/13/nero-7-announced    /nero-7-announced

# Will move to something like this
/:year/:month/:date/:slug  /:slug

At this point I had the site successfully building and deploying on Netlify. Here's what my netlify.toml file looks like below. My setup is fairly basic so everything here can be controlled on the Netlify website as well.

[build]
  publish = "_site/"
  command = "jekyll build"
[build.processing]
  skip_processing = false
[build.processing.css]
  bundle = true
  minify = true
[build.processing.js]
  bundle = true
  minify = true
[build.processing.html]
  pretty_urls = true
[build.processing.images]
  compress = false

The only thing left was updating the DNS to point to Netlify. To make this as smooth as possible I first logged into AWS and visited the Route53 dashboard to see all the DNS records attached to my domain. One by one I added the exact same DNS records to Netlify's DNS. There there were quite a few records as I use both G Suite and Amazon SES on this domain.

I waited a while for Netlify's DNS servers to add those rules and then I made the final change to update the DNS servers on the domain to point to Netlify's DNS servers. Moments later the domain started loading my site on Netlify and then after that the SSL certificate was running.

A note about media uploads

While I migrated my blog to Netlify, I still keep my media assets hosted separately with S3 with CloudFront as I have for many years. I originally did this for a few reasons I mention on my About this website page. The main one being that I don't want ~30GB of media in my main git repository.

Netlify has a solution for this with their Large Media feature that uses Git LFS to store media, but I haven't looked into that too much.

Netlify first impressions

I'm only a week into my new Netlify setup but I'm really loving it so far. Here are some of my initial thoughts:

• Worry-free deploys: I'm coming around to this build system way of pushing git and knowing it will do everything it needs to build, optimize and deploy. I had something similar for my staging site back when I was writing on my iPad Pro and before I had my code-server VPS—each push would trigger AWS CodeBuild to deploy to a CloudFront distribution. Though that wasn't a great experience for a staging server I was using for quick previews (having to wait a minute or more before the deploy is done), it's fine for regular production deploys. I should also mention that with Netlify I can easily rollback to a previous deploy if needed.

• Built-in form handling: This was a really, really nice perk. With a static site hosted on S3 there was no backend that could accept any contact form POSTs; I had to use a hosted contact form from another service. Netlify has some magic attributes like data-netlify="true" you can apply to forms to get form handling and spam prevention.

• Asset optimization: I would love nothing more than to ditch all of my local asset pipeline management like concatenating CSS/JS files and minifying. Netlify handles this all for you (though as I noted I couldn't get it to minify my JS for some reason but I'm sure that will be resolved soon).

• Server-side analytics: This is neat. Previously if I wanted server-side analytics I would have to enable S3/CloudFront logging and then use a tool (either paid or self-host) to read through the logs. While I don't think I'll use it forever as it's rather pricey add-on at $9/month with far less functionality than dedicated analytics tools, it's a great resource for seeing things like what bots are accessing or what 404s I may need to fix.

• Build times could be better: While it's not a huge inconvenience, I do think build and deploy times could be faster, especially with the Pro plan that I'm on. It feels like they can vary between 1-4 minutes. Sometimes it goes by quickly and sometimes it takes more time between each file (maybe not cached?). I also think it must be running on some not-so-great hardware if it takes about 7.5 seconds to generate my jekyll site; something that takes 3.5-4 seconds on my MacBook Pro.

What's next

I'm trying to make it easier for me to write and publish articles, especially with my new focus on writing more with these short-form posts. Netlify has been a huge help here by reducing some of the friction around deploys.

Other things top of mind for making it easier to publish include migrating away from Jekyll to a faster static site generator like Hugo and considering an admin front-end for my site generator like Netlify CMS.

For about a year now I have been using a Lightsail VPS running VSCode code-server that I can access from any computer to write and deploy to my site. However, I'm now trying out GitHub Codespaces, which is basically the same thing.

I'm also exploring simplifying my local writing process by trying Nova.app from Panic to have an all-in-one writing, dev and preview experience—no need to flip between 3 apps (Chrome, iTerm2, Atom). Though I wish I could figure out how to get a live browser preview with Live Reload working as mentioned on their homepage.