3 hours later, I've finally found one of the reasons I kept getting the error message ERROR Failed to contact the origin. from CloudFront while trying to set up a download bucket pointing to my newly-secured EC2 Apache instance.
I recently needed to set up a wildcard SSL certificate for a project. Because of this, I set up my VirtualHost in apache to list on port 443, and then set up another VirtualHost record on 80 that redirected to the secure protocol.
Seems simple enough, right?
I figured now all I needed to do was tell my CloudFront distribution bucket that when a request comes in for some static content on https://, make the request to my source server on the same protocol.
In order to do that, I went into my AWS Console, found the distribution, click on Distribution Settings, Origins tab, chose my origin and clicked Edit, and then finally, switched Origin Protocol Policy from HTTP Only to Match Viewer
That was my mistake :/
Doing so resulted in the ominous error message ERROR Failed to contact the origin. over and over again. While as of yet I still don't know the actual problem that's being encountered (eg. why does that result in my server being unreachable to AWS, but not to a browser?!), I did seem to narrow down the problem as being related to the redirect in my Apache config.
So, a way around this hiccup is to adjust my redirection strategy.
I changed my Apache config to also listen on port 80, and then do the redirect in the middle-ware (eg. the controller actions for my application).
This has the following effect: static content can still be accessed through port 80. It's also not a security hole since the only one accessing this content is CloudFront. The content itself is served out over https to the actual viewer.
In writing this, I realized I should have tried changing the order of the VirtualHost's in my config file. Additionally, I was using the following rules:
<VirtualHost *:80>
ServerName dev.domain.com
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>
Maybe I ought to have applied a redirect rather than a rewrite? Perhaps CloudFront see's that as an invalid redirect.
I can't be sure. And because changing a CloudFront distribution settings can take up to 15 minutes, it's likely not something I'm going to wait to find out.
But there's my solution to this strange, strange problem :)
Update
I tried both scenarios:
1) Changing the order of the VirtualHost records; no luck
2) Specifying the RewriteRule as a 301 redirect. I thought this would work, since it appears without it a 302 is sent by Apache
No luck :(