Edit
Due to popular demand, I have provided a git link to the pleb stack
https://github.com/rm2kdev/PLEB-Stack
Introduction to Site Loader
Creating a full application from scratch in two days is no trivial task and thus requires a great deal of planning and architecture to avoid a colossal waste of time. The idea itself stemmed from a problem I had at a previous job, we ran a number of websites that often generated the business customers and leads unfortunately these forms were tied into a bunch of businesses crap software CRM’s, Lead Tracking and a crappy third party analytic’s platform that some marketing person was up-sold as a better alternative to Google Analytic’s.
Due to all the installed crap ware and a poorly designed application the site often crashed and never came back up, it was at this time that I had the idea to come up with a simple domain checking platform while I knew about other alternatives that existed i found that they were often overpriced or just plain limited.
Enter Site Loader, the simple and free site monitoring tool that I designed and built over the weekend. With a core focus on being simple and usable while having some advanced features that the big boys have such as SMS Messaging and Email Alerts.
Cost Feasibility Study
Warning this section gets kind of math’y towards the end 🙂
Before I was able to build Site Loader I had to do some basic cost studies maybe this would enlighten me to why my competitors were so expensive and to decide if this idea was even possible to do at a free / freemium price point.
The first step was to identify the factors involved in site monitoring these would be elements such as:
- Server and hosting Costs?
- Domain costs?
- The size of the webpages that would be monitored?
- The frequency that the sites would be monitored at?
- Bandwidth Costs?
- SMS and Email Costs?
1. After evaluating a bunch of cloud server hosts I decided to go with an Amazon EC2 Tiny Instance running Ubuntu 12.04 my reasoning behind this was influenced by price and the technologies used in the development of the web application (which will be explained in a section below) namely NodeJS and MongoDB this comes at a cost of roughly $15/mo we’ll also be needing 1 elastic IP address and a few entries in route53 + $1
2. Domain costs $9.99 /year from go daddy so that’s $0.80/month
3. Now after doing some poaching around on a few websites I’ve come up with an average web page size of 30kb this is obviously going to change on a per client basis but we need a number to work with here.
4. The frequency we will monitor these websites at will be 24 Hours, 1 Hours and 5 minutes
5. Bandwidth costs for an EC2 Tiny instance are charged at $1.86 per gb per day so we need to work this figure back and find out what it costs per 30kb well that’s easy ((1.86 / 1024) / 1024) = $0.00000177383 per ping to an average 30kb web page
6. We are using SMSGlobal for sms handling at a cost of about 8c / message and SendGrid for emails at a cost of $9.99 / month for 40,000 emails. This brings us to a value of $0.00024975 per email
So with all of these factors now established we can begin to come up with some costing estimates to determine the feasibility of this application we are now looking at a base monthly cost of roughly $26.79 that is Item 1 + Item 2 + Item 6.
Now we need to know the cost of each ‘free’ client based on what we know already each ping costs $0.00000177383 and these clients will only receive one ping per day that means our daily cost per free client is $0.00000177383
Next our tier 1 clients who receive a ping every hour well that’s easy also $0.00000177383 * 24
Gives us a cost of : $0.00004257192 per day or $0.00131972952 per month
Next our tier 2 clients who receive a ping every 5 minutes this one is also simple we just need to find out how many ping’s occur every hour $0.00000177383 * ((60 / 5) * 24)
Gives us a cost of : $0.00051086304 per day or $0.01583675424 per month
Finally we can factor in some average sms and email costs, for these calculations i am using an average of 20 communications per month per client as it is better to overestimate than underestimate these kinds of figures.
Free Clients : $0.004995 / month + $0 for sms
Tier 1 : $0.004995 / month for emails + $1.60 for sms
Tier 2 : + $1.60 for sms
Giving us our final and TOTAL costs of :
Free Clients $0.00499677383
Tier 1: $1.60631472952
Tier 2: $1.62083175424
Per Client Per Month
Planning my time
Given that I only gave myself two days to write Site Loader I had to prioritize my time efficiently to get it completed by the end of Sunday night I accomplished this by first evaluating what the application needed in a version 1.0 release and writing it all down on a paper I start with the easiest features and work my way up into the hardest this gives me time to setup the simple things like login pages, or installing a bootstrap theme and focus on the technical side of things after.
I gave myself ample time to sleep (8 hours) eat (1 hour) shower (1 hour) etc as not to bother my family by being the hairy cave monster from lord of the rings the whole weekend. The best way I’ve found to work on these ’48 hour jam type’ projects is to just tackle them not being in a rush and maintain your regular life you get more done when your not feeling pressurized, that might be just me but never the less it has always worked for me
Choosing the technologies
I chose to build Site Loader in NodeJS due to the fact that the last 6 months at my new place of employment I’ve been using it predominantly and slowly began to gain more and more experience in it. This is my first solo project using NodeJS where as previously I was using ASP.net i find that writing my new applications in NodeJS to be cleaner and overall easier to maintain.
I am using MongoDB as the back end database mainly because NodeJS and Mongo basically go hand in hand together. I decided to assemble a framework called “PLEB Stack” that stands for Passport, Locomotive, Express, Bootstrap and it handles basic things like user account creation, MVC pattern, routing, login logout and all your typical basic application stackky stuff as well as implementing the basic twitter bootstrap theme.
The PLEB Stack was designed for Site Loader with the best practices of all the library’s used in mind also I was hoping that I’d be able to take it into work on Monday and use it in an effort to clean up some of the code me and my co-workers had assembled in the application we are developing there which had begun to spin out of control some house keeping was in order.
Some of the core packages I am using in the PLEB Stack and Why?:
- Locomotive
- This provides a MVC Architecture on top of expressjs
- Mongoose
- Supplies the modal and querying capabilities to mongodb
- Passport
- We use this to have a simple oauth compliant login process and authentication handling
- Bcrypt
- This provides the blowfish encryption that we use on password encryption
- Moment
- Javascripts inbuilt time methods suck quite badly so I use this to add additional features such as “addhours” etc
- Stripe
- These guys have a fantastic alternative to paypal for payment processing its very easy to use and works fantastically
- Mailer
- Compiles mustache templates and sends them via SMTP
Building the application
Actually building the application was the easiest part. I started by setting up my git repo’s and setting up the PLEB Framework building my basic register and login pages, handling authentication etc, once I had the basic website stuff done I made a fork of the PLEB Framework and began actually working on application specific features.
Creating simple forms like the dashboard overview and url submission pages were no more than your basic CRUD setup, now that i had a working front end for the application it was time to begin working on the back end.
I made a separate project with one difference, it was not a web server but an endless task using the basic JavaScript setInterval I was able to setup an endless loop that would soon call the functions that handle pinging and tracking of website uptime activity.
Now that both front end and back end were done it was time to add payments, this couldn’t have been easier thanks to the amazing stripe framework the way they handle credit card payments is ingenious upon registration to the website you are automatically created an account within the stripe payments system and a unique ID is returned from them to Site Loader and stored with your unique user, should this user ever decide to upgrade to a different package we collect their payment information and this is where the amazing stripe library takes front stage.
Rather than having to deal with storing sensitive credit card information the stripe team does this via a secure JavaScript the user entered their credit card details into the form which gets posted via SSL to stripe, stripe them returns a unique ‘card id’ which gets posted to the Site Loader system and stored with your user profile this means we never have to deal with your sensitive information and never have to worry about getting hacked and having credit card details stolen, when it comes time to make a charge we simply send the customer id along with the card id to stripe with a secure ‘secret’ key to authorize the request and stripe will handle the payments and send us a call back with the status.
Testing the application
Now that we’ve entered the last half of Sunday evening it was time to begin testing the application some of my main concerns were things like performance and ram usage on the tiny ec2 instance I had used , I began with an overall functionality test make sure everything was working as expected and then try to drill down further i had some small issues with url formatting requiring HTTP or HTTPS as part of the ‘request’ library in node.
The website its self was quite solid there were obvious validation issues but nothing that would stop the application from working, I fixed up a few of these server side and implemented some html5 validation tags on things like email fields, I will do a validation pass over the application in the neer future and add some kind of javascript implementation so that it works in older browsers but for the moment it seemed quite solid even if you can get away with having a stupid password.
Challenges faced
As i said above there were obvious technical challenges getting a service to run in node js takes a little tweaking and prouesse, There were a few issues with the technology its self my first implementation of the stripe payment system was completely wrong and i had to go back and re-do the lot.
Time restrictions obviously creating a full application in just two days is a challenge in its self not to mention taking care of life i.e wife, showering, sleeping and eating unfortunately I am a type 1 diabetic so I cant avoid these things without serious health impacts!
Cost management ideally this application would cost less than $30 a month to run excluding once off costs such as design.
Summary
With the conclusion of this article marks the official launch of Site Loader I have learnt a stack of things in building this application but most important by creating this tool I have designed and created my own application that people can use, a framework for writing applications in the future using node js and a framework for continuous processing all of which should allow me to create and maintain web apps easier and faster in the future.
I hope that you enjoyed reading about my process and the tools and tech that I used when building Site Loader it is still by no means finished, no application is ever truly finished but the core functionality I set out to accomplish is now completed I will be updating the application with newer features soon such as average ping for the day and uptime percentage etc
P.s. I am still working hard on my blog writing, I’ve never been too good at writing stories or blogs and I can barely read myself haha so if you do have any tips or comments about this blog post or just tips in general about writing good engaging posts please feel free to contact me on twitter @rm2kdev or leave a comment here on this blog
