# Army.ca's Server Infrastructure Changes



## Mike Bobbitt

*Summary*

Our Internet service provider is changing their hosting options, and the new choices do not work well for us. We need to come up with a plan to move our services within the next 60 days.

*Background*

Army.ca is currently on a hosted server at Slicehost. Slicehost was purchased by Rackspace and as part of that consolidation they are no longer offering the hosting plan we use. Their new 'cloud' approach will cost us more and give us less, so it's not an appealing option. What's more, they are migrating all existing customers (that's us) over to the new cloud server architecture.

This conversion process will be complete *NLT May 2012*.

Rather than be stuffed into a solution that doesn't fit our needs, I thought I would use this as an opportunity to assess our requirements and resources to come up with a plan. I have outlined a couple of high level options for consideration.

For comparison, our current setup includes:


CPU: Quad-Core AMD Opteron(tm) Processor 2374 HE
RAM: 3 Gb
Traffic limit: 1,800 Gb/month
No backups
Disk space: 113 Gb
Throughput: Unknown (slicehost doesn't post this) but at least 200k/s
We own but a 'slice' of a virtual server
Monthly cost: $200 USD
No contract

A quick analysis on this comes up with the following important points:


Initially the CPU was adequate, though in the last 6+ months, the server load has moved from "frequently idle" to "usually busy". Not a critical issue but as we continue to grow we may start to see performance degrade slightly.
3 Gb RAM is sometimes not enough - we could benefit from more and while it's not critical, we would see performance improvements with more.
1,800 Gb of traffic/month is far more than we need, we typically use around 100 Gb/mo.
I am conducting my own backups, but there are performance drawbacks to doing so with a remotely hosted server.
113 Gb of disk space is adequate. We currently use roughly 70% of that, with moderate growth over time.
Our throughput has rarely been an issue; usually some other part of the server tops out first.
Virtual servers are great for the hosting company as management and uptime are improved. However this often comes at the cost of performance, as your server is competing for physical resources with other virtual servers. This technology has improved drastically over the years, but sometimes problems on one 'slice' can spill to another.
Our availability (uptime) with Slicehost has been excellent. With a few exceptions, most of the outages have been at my hand.
Slicehost has no contract, if I cancelled today my bill would be prorated for the portion of the month I used the service.

*Option 1 - Find another hosted solution*

Pros:

No startup costs.
No hardware maintenance.

Cons:

Likely requires a contract, so we will be tied to a single host for 1-2 years. I tend to like to avoid this type of commitment as it locks us in to a host that may not end up suiting our needs.
Research is required to find an appropriate host may be extensive.
No console/physical access to our server. If something goes wrong it could be difficult to diagnose and fix.

*Option 2 - Run a dedicated server/connection*

Pros:

Easy management. In the past, managing a server that is in a datacenter has had it's issues. Mostly when I bring it down and can't bring it back up again!
Local backups are easy, fast and do not sap our performance.
We can get a significantly more powerful server to help handle load and growth.
Monthly costs are lower - I estimate half of a typical hosted service.

Cons:

Initial capital outlay required for new server. I estimate that a decent server will cost us $1,500, though more research is required on my part.
Hardware maintenance - if something goes wrong, we need to fix it.
A local fast connection is required, with good upload/download speeds. Bell FiberOP may fit the bill here, but is only now being rolled out to my area.
Uptime could be an issue; luckily I still have a 2200 VA UPS to help keep things going, at least for a short time.

These lists are far from exhaustive, and are meant more as preliminary notes at this stage.

*Request*

There are a lot of technically minded users here and as this change will have a big impact on how we operate, I wanted to open it up for discussion. Some folks may have been through a similar process recently, know where to find a good deal on hardware, have experience with a good hosting company, etc.

Let's get the conversation rolling!


Thanks
Mike


----------



## vonGarvin

I'm a Technoviking, but I'm no techno-geek, so though the music dances to me, I'm a slave to the computer.  So, my feedback is purely logic based on your original post:




			
				Mike Bobbitt said:
			
		

> *Option 1 - Find another hosted solution*
> 
> Pros:
> 
> No startup costs.
> No hardware maintenance.
> 
> Cons:
> 
> Likely requires a contract, so we will be tied to a single host for 1-2 years. I tend to like to avoid this type of commitment as
> Research is required to find an appropriate host may be extensive.
> No console/physical access to our server. If something goes wrong it could be difficult to diagnose and fix.
> 
> *Option 2 - Run a dedicated server/connection*
> 
> Pros:
> 
> Easy management. In the past, managing a server that is in a datacenter has had it's issues. Mostly when I bring it down and can't bring it back up again!
> Local backups are easy, fast and do not sap our performance.
> We can get a significantly more powerful server to help handle load and growth.
> Monthly costs are lower - I estimate half of a typical hosted service.
> 
> Cons:
> 
> Initial capital outlay required for new server. I estimate that a decent server will cost us $1,500, though more research is required on my part.
> Hardware maintenance - if something goes wrong, we need to fix it.
> A local fast connection is required, with good upload/download speeds. Bell FiberOP may fit the bill here, but is only now being rolled out to my area.
> Uptime could be an issue; luckily I still have a 2200 VA UPS to help keep things going, at least for a short time.
> 
> These lists are far from exhaustive, and are meant more as preliminary notes at this stage.


Of these two options, I would go with option 2.  The kicker for me was this:


> If something goes wrong it could be difficult to diagnose and fix.





> Hardware maintenance - if something goes wrong, we need to fix it.



I would change the second into a "pro" and restate is this:  if something goes wrong, we can fix it.


----------



## GAP

Not if he's away....server's are backed up using Raid, etc. to reduce data loss.


----------



## 2010newbie

I've used this company before and they were always reliable and cost-effective. 

http://www.bluegenesis.ca/


----------



## agc

I'm certainly no expert, but you might wish to investigate Amazon EC2. http://aws.amazon.com/ec2/pricing/


----------



## Gunner98

Mike,

Wouldn't an option be to close the shop and go hunting, fishing and spend time with your family?  Think of the free time you and the rest of us would have.  Just sayin'! >


----------



## Zoomie

I still fixated on the part where Mike mentions that we only currently use 100GB/month of the allowable 1800GB/month.  I don't think the Mods can use the whole bandwidth argument anymore - new rule for the server - don't use the SEARCH function.


----------



## armyvern

Zoomie said:
			
		

> I still fixated on the part where Mike mentions that we only currently use 100GB/month of the allowable 1800GB/month.  I don't think the Mods can use the whole bandwidth argument anymore - new rule for the server - don't use the SEARCH function.



Yeah, but that was way back when I was a mod too. Mike was running off his server vice the virtual and we all used to scream, "Mike, come back off vacation dammit!!" Uhhmmmm, good old days.  >


----------



## PuckChaser

I really think an offsite host is going to be the best option here. They will have the racks, the backups, and the connection. Your post was cut off, what is the reasoning behind not wanting a 1 year contract?


----------



## captloadie

The offsite option, if with a reliable company, should be the best option. Yes, having everything in house is nice, but in this case, it is literally in house (Mike's that is). If Mike is away, and things go pear shaped, what happens? If Mike finally realizes that there is more than just Milnet.ca to life, what happens then?

An offsite option should give Mike more flexibility, like assigning other trusted users access to do admin functions when he is away, and the ability to hand over the keys forever, if he ever wants to. It also means that if for some reason a legal issue arose that required the servers be taken into custody, he doesn't have his personal space invaded. (Alright, this last one is out there, but ehh, who knows what the man who's name shall not be mentioned on this site might do.)

My  :2c:


----------



## Mike Bobbitt

Thanks guys, I appreciate the feedback. All good points and will be considered. Keep the feedback coming!



			
				2010newbie said:
			
		

> I've used this company before and they were always reliable and cost-effective.
> 
> http://www.bluegenesis.ca/



Thanks, I hadn't head of Blue Genesis. They look like a decent hosting company, but I'm pretty fussy about our needs. Specifically, I don't want a shared account on a server, I prefer to have full control (insert maniacal laugh here). It looks like Blue Genesis doesn't give me my own server with root and console access to mould in my own image. Do you know what their throughput is like? (I.E. how much simultaneous traffic they can support; their actual speed provisioned to users?)

I ask because I have a DreamHost account that has unlimited monthly traffic, but the connection speed is so slow it is unusable for a 'big' task like Army.ca. (It's also a shared account - no root, no Mike. )



			
				agc said:
			
		

> I'm certainly no expert, but you might wish to investigate Amazon EC2. http://aws.amazon.com/ec2/pricing/



The last time I checked on this, it wasn't available in Canada, so I dismissed it this round. Looks like that has changed. I have signed up and will start investigating. Overall I have heard very good things about EC2 so I will be paying close attention here!



			
				Zoomie said:
			
		

> I still fixated on the part where Mike mentions that we only currently use 100GB/month of the allowable 1800GB/month.  I don't think the Mods can use the whole bandwidth argument anymore - new rule for the server - don't use the SEARCH function.



It's more about conservation of sanity than bandwidth.  All kidding aside, while this is mostly a leftover from the days of resource constraints, it is still a 'best practice.' Don't forget, while they allow us to use 1,800 Gb per month, our connection speed may not allow that in practice. See my note above about DreamHost's unlimited traffic; that account would crawl under Army.ca's load because the pipe is not big enough to serve our traffic needs. An analogy would be if someone gave you a lawn tractor and said "don't drive it over 50,000km per month." Probably can't hit that limit even if you drove it at 'top speed' 24/7. In short: you're still competing for traffic with every other user for limited traffic delivery, regardless of what our monthly cap is.



			
				PuckChaser said:
			
		

> I really think an offsite host is going to be the best option here. They will have the racks, the backups, and the connection. Your post was cut off, what is the reasoning behind not wanting a 1 year contract?



Sorry, fixed that line:



			
				Mike Bobbitt said:
			
		

> Likely requires a contract, so we will be tied to a single host for 1-2 years. I tend to like to avoid this type of commitment as it locks us in to a host that may not end up suiting our needs.


----------



## Prefect

You should definitely check out iweb.com. They operate 4 datacentres in Montreal.

I've had my server colocated with them for the last 5 years and have had very few issues. (Uptime on my system currently stands at 475days.) They have stopped offering colocation services for individual servers but their dedicated server prices are reasonable.

Here is an example of what they call a "Smart Server". (Virtualized for ease of remote management but the server itself is only used by one person.)

CPU: Core 2 Quad 2.66ghz
Mem: 4GB
HDD: 2x 1TB SATA2 (RAID 1)
Connection: 100mbit
Bandwidth: 10,000GB
IPs: 5 usable in a /29 block
OS: CentOS/Debian/Ubuntu

Price: $159 + taxes/month

As for actual throughput, from my experience as a colocation customer it is always a full 100mbit up/down with no exceptions.


----------



## Mike Bobbitt

Excellent. good to know. And it keeps it Canadian too, which is nice. I'll definitely check into them as well.


----------



## josh54243

Throwing my hat into iWeb.com as well. Used both their dedicated and shared services and was a pleasant experience.


----------



## blackemail

Mike,

I use 1and1.com for all my domain registration and hosting.  While I don't have the need for dedicated servers, I have never had an issue with their hosting in the 10+ years I have been with them.  I just did a quick check on their physical server hosting and it appears very competitive.  http://order.1and1.com/ServerPremiumL?__lf=Static&linkOrigin=ServerPremiumXL&linkId=hd.tab.packageselection.serverTypL


----------



## wildman0101

Mike. Give netzero a call. 1800-560-7429. They got some good package deal's 
both personal/web. Might be just what your lookin for. Cheer's mate and best
regard's. Scoty B


----------



## Mike Bobbitt

A quick update: I have settled on Option 1 (new host) as it looks like there are some good options available. The cost and extra work of a local setup is probably too much.

I have not yet settled on a host, but thanks to the suggestions here, I have a short list to work from.

As part of this transition, we will be moving from Fedora to Ubuntu, which is going to be a big change for me. The next major revision of Ubuntu comes out in about 3 weeks, so I will wait and use that as a starting point.

More to follow.


----------



## Mike Bobbitt

Another update, I have settled on Amazon's EC2 service as it has the most flexibility and the best price, depending on the package we end up with.

The current package I am looking at is $600 up front with a small monthly fee for 3 years. This gets us a server comparable to the one we use today.

I'm trying to find a way to get a much improved server; it's costs are $1,200 USD up front but about the same monthly costs as the above package. This gets us on a server that is significantly better than the one we are using now. I'm crunching some numbers to see if we can make that option work. In the long run, it is a much better deal for us and allows future growth.

I'll provide an update shortly.


Cheers
Mike


----------



## GAP

Many here would not be adverse to "kicking in" per se if it substantially upgrades the server.....just saying..... ;D


----------



## Old Sweat

GAP said:
			
		

> Many here would not be adverse to "kicking in" per se if it substantially upgrades the server.....just saying..... ;D



What he said.


----------



## Bruce Monkhouse

Yea,...I could go without hookers and blow for one weekend.


----------



## tango22a

Me Too!!

tango22a


----------



## Edward Campbell

GAP said:
			
		

> Many here would not be adverse to "kicking in" per se if it substantially upgrades the server.....just saying..... ;D



 :ditto:


----------



## Mike Bobbitt

OK, I've looked over the books and accounting for May's expenses (Internet service via the current provider) we have about $450 cash on hand. I was kind of hoping we were a little closer to that "big" server but if we save up I think we can get there. Part of the problem is that I spent our small surplus on inventory however those sales have been going well so I think that works out OK in the end.

Some details I didn't have time to fill in previously:

$600 initial cost server: 3.75 GB of memory, 1 virtual core, 410 GB of local instance storage, 32-bit or 64-bit platform
$1,200 initial cost server: 7.5 GB of memory, 2 virtual cores, 850 GB of local instance storage, 64-bit platform

What's more, once we make the initial cash layout, both of these options become significantly cheaper than our current host. They essentially become $50/month, vs. our current $200/month hosting fee. As you can see, we recoup that $600/$1,200 pretty fast. (4 months or 8 months depending on the plan.)

The downside is that in order to get this good monthly rate, we have to lock in for 3 years, so I want to be really sure we select the right server size now, as it has to last us until mid 2015. Given that, I would rather wait and save for the 'big' server than lock us in to a 'medium' server that will bottom out before the contract is up.

Those are my thoughts on it, I welcome any questions or discussion. I'll keep everyone updated on our financial progress as we move forward.


Thanks
Mike


----------



## PuckChaser

I think you've got a great plan and have covered yourself for future growth, especially with the Afghan Ops game eating into some of that server.


----------



## Fishbone Jones

Mike,

Give me an email that I can send a PayPal donation to.


----------



## GAP

While saving up, you are still paying $200/month.....I think as a group we can all do better than that and get the plan that does not bottom out before 2015.

It can't hurt to ask it of the membership


----------



## Mike Bobbitt

Agreed, there is some logic to that approach, so I'll encourage anyone who might be on the fence about a subscription to take a closer look. Not only do you get a cool shirt (which can be worn under your uniform!) and an Army.ca battle badge, but you would also be directly pushing the site towards a more sustainable future. And a much faster one too! 

Another option that works well is advertising. We have a lot of unused ad space here - over 4 million unused impressions per month! If you know of anyone who might benefit from advertising to this focused audience, please send them my way. Although nobody 'likes' ads, it is a great way to financially support the site without tapping the membership.

P.S. Thanks recceguy for the generous donation, that pushes us ever closer to our goal!


----------



## larry Strong

Pm me to we're I can send a donation


----------



## PMedMoe

Larry Strong said:
			
		

> Pm me to we're I can send a donation



Me too.   :nod:


----------



## Fishbone Jones

You can send a PayPal donation to:

Milnet@Milnet.ca


----------

