Author Topic: Major Site Upgrade  (Read 132121 times)

0 Members and 1 Guest are viewing this topic.

Offline Mike Bobbitt

  • Army.ca Owner
  • Directing Staff
  • Army.ca Legend
  • *
  • 126,920
  • Rate Post
  • Posts: 12,280
    • Army.ca
Re: Major Site Upgrade
« Reply #225 on: June 04, 2017, 05:39:34 »
Unfortunately a known issue. We simply hit too many connections for the database server to handle, and new connections are deferred until things ease up. Been looking at way to optimize database access for as long as the screen has been coming up, but clearly haven't made significant progress yet.

The issue sort of stems (ironically) from the recent server upgrade. We have two pieces, the front end (web server) and back end (database). We upgraded the front end a couple of months ago, giving us room to stretch our legs a bit more. Doing so is now overloading the back end, which has not been upgraded in 2+ years.  :-\


Cheers
Mike
« Last Edit: June 04, 2017, 05:43:43 by Mike Bobbitt »

Offline SeaKingTacco

  • Army.ca Fixture
  • *****
  • 177,560
  • Rate Post
  • Posts: 6,043
  • Door Gunnery- The Sport of Kings!
Re: Major Site Upgrade
« Reply #226 on: June 04, 2017, 06:53:21 »
Thanks Mike! Keep up the good work.

Offline Chris Pook

  • Army.ca Subscriber
  • Army.ca Legend
  • *
  • 218,760
  • Rate Post
  • Posts: 13,066
  • Wha daur say Mass in ma lug!
Re: Major Site Upgrade
« Reply #227 on: June 04, 2017, 10:47:42 »
Ditto to SKT, Mike.

On your own time!   :nod:
"Wyrd bið ful aræd"

"If change isn’t allowed to be a process, it becomes an event." - Penny Mordaunt 10/10/2019

“Life can only be understood backwards, but it must be lived forwards” ~ Soren Kierkegaard

ignoramus et ignorabimus

Offline kev994

  • Army.ca Subscriber
  • Full Member
  • *
  • 4,215
  • Rate Post
  • Posts: 255
Re: Major Site Upgrade
« Reply #228 on: June 04, 2017, 14:06:55 »
For the past ~3 weeks I've been unable to connect with Tapatalk during peak periods but usually have no issues using Safari on the same phone. Fair assumption it's the same capacity issue?

Offline Mike Bobbitt

  • Army.ca Owner
  • Directing Staff
  • Army.ca Legend
  • *
  • 126,920
  • Rate Post
  • Posts: 12,280
    • Army.ca
Re: Major Site Upgrade
« Reply #229 on: June 04, 2017, 22:11:09 »
Yes, we are hitting a high number of concurrent database connections, so the "last in" time out or are dropped outright depending on how far over the limit we go. Tapatalk adds more connections. As a test I may temporarily disable it to see if it is a key contributor to the issue. I suspect it is not though.

Offline Target Up

    ........pull, patch, and score.

  • Army.ca Subscriber
  • Army.ca Fixture
  • *
  • 259,175
  • Rate Post
  • Posts: 6,247
  • that's how we roll in redneck land
Re: Major Site Upgrade
« Reply #230 on: June 04, 2017, 23:40:34 »
Yes, we are hitting a high number of concurrent database connections, so the "last in" time out or are dropped outright depending on how far over the limit we go. Tapatalk adds more connections. As a test I may temporarily disable it to see if it is a key contributor to the issue. I suspect it is not though.

No skin off my rump if TapaTalk never comes back.  How it`s supposed to make forums easier beats me, I hate it.  8)
Apparently, a "USUAL SUSPECT"

“In peace there's nothing so becomes a man as modest stillness and humility; but when the blast of war blows in our ears, then imitate the action of the tiger; stiffen the sinews, summon up the blood, disguise fair nature with hard-favor'd rage.”

 Every normal man must be tempted at times to spit on his hands, hoist the black flag, and start slitting throats

Offline Fishbone Jones

    MSC -7995.

  • Army.ca Subscriber
  • Army.ca Myth
  • *
  • 283,582
  • Rate Post
  • Posts: 18,683
    • Army.ca
Re: Major Site Upgrade
« Reply #231 on: June 05, 2017, 00:32:15 »
I installed it and took it out within the hour. It wants too many permissions and wants to filter all your internet activity through the program. I wish there was a way to kill the banner request to 'enjoy tapatalk'. I hate turning it off everytime I visit.
Corruption in politics doesn't scare me.
What scares me is how comfortable people are doing nothing about it.

Offline Colin P

  • Army.ca Legend
  • *****
  • 178,910
  • Rate Post
  • Posts: 10,596
  • Civilian
    • http://www.pacific.ccg-gcc.gc.ca
Re: Major Site Upgrade
« Reply #232 on: June 09, 2017, 16:13:36 »
Getting lot of these on Government supported (ha!) Firefox

"Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later."

Offline Mike Bobbitt

  • Army.ca Owner
  • Directing Staff
  • Army.ca Legend
  • *
  • 126,920
  • Rate Post
  • Posts: 12,280
    • Army.ca
Re: Major Site Upgrade
« Reply #233 on: June 11, 2017, 06:33:00 »
It's our server, not the browser. Still tweaking the configuration to try to get the maximum out of it, but we are hitting some hard limits.

Offline Mike Bobbitt

  • Army.ca Owner
  • Directing Staff
  • Army.ca Legend
  • *
  • 126,920
  • Rate Post
  • Posts: 12,280
    • Army.ca
Re: Major Site Upgrade
« Reply #234 on: June 22, 2017, 14:13:25 »
Folks,

I am converting our database from (slow) magnetic to (fast) SSD. This will have an adverse effect on performance during the conversion but "should" help long term. I'll post again when the conversion is complete.


Cheers
Mike

Offline Mike Bobbitt

  • Army.ca Owner
  • Directing Staff
  • Army.ca Legend
  • *
  • 126,920
  • Rate Post
  • Posts: 12,280
    • Army.ca
Re: Major Site Upgrade
« Reply #235 on: June 22, 2017, 14:43:52 »
...and it's complete. Fingers crossed this will be a "good" change. :)

Offline Infant_Tree

    Training very hard.

  • New Member
  • **
  • 1,440
  • Rate Post
  • Posts: 42
Re: Major Site Upgrade
« Reply #236 on: June 22, 2017, 14:53:15 »
Folks,

I am converting our database from (slow) magnetic to (fast) SSD. This will have an adverse effect on performance during the conversion but "should" help long term. I'll post again when the conversion is complete.


Cheers
Mike

...and it's complete. Fingers crossed this will be a "good" change. :)

Yeah just noticed the site's no longer down and it seems to be running smoothly :P I appreciate the hard work you did upgrading the site for us all.
Recruiting center: Toronto
Regular/Reserve: Regular
Officer/NCM: NCM
Trade Choice 1: Infantry
Application Date: January 12, 2017
First contact: January 12, 2017
CFAT: February 8, 2017
Medical: March 2, 2017
Interview: March 7, 2017
Merit listed: Sometime between April 11 and April 27, 2017
Position Offer: May 30, 2017
Enrollment: August 23, 2017
BMQ: September 4, 2017


Release date: October 19, 2017 (I VR'd from BMQ)

Offline mariomike

  • Directing Staff
  • Army.ca Legend
  • *
  • 570,960
  • Rate Post
  • Posts: 10,966
  • Keep 'em rolling.
    • The job.
Re: Major Site Upgrade
« Reply #237 on: June 22, 2017, 14:56:35 »
I appreciate the hard work you did upgrading the site for us all.

A way to show your appreciation,

Milnet.ca Subscriptions
http://milnet.ca/subscribe/index.php?action=profile;u=16
In any war, there are two tremendous tasks. That of the combat troops is to fight the enemy. That of the supply troops is to furnish all the material to insure victory. The faster and farther the combat troops advance against the foe, the greater becomes the battle of supply. EISENHOWER

Offline Loachman

  • Former Army Pilot in Drag
  • Directing Staff
  • Army.ca Fixture
  • *
  • 219,842
  • Rate Post
  • Posts: 7,502
Re: Major Site Upgrade
« Reply #238 on: June 22, 2017, 17:12:21 »
And it only took thirty minutes between the two posts.

You make it look so easy, Mike. You should work on your Scottyness a bit more - "I canna do it in less than three days, Captain".

It may be (I hope) the DWAN connection, but I got a "Server too busy, please try again in a few minutes..." message when trying to compose this response.

Offline kev994

  • Army.ca Subscriber
  • Full Member
  • *
  • 4,215
  • Rate Post
  • Posts: 255
Re: Major Site Upgrade
« Reply #239 on: June 22, 2017, 19:37:48 »
It's working much better for me, everything has loaded instantly, I normally have to wait a while at this time of night.

Offline Mike Bobbitt

  • Army.ca Owner
  • Directing Staff
  • Army.ca Legend
  • *
  • 126,920
  • Rate Post
  • Posts: 12,280
    • Army.ca
Re: Major Site Upgrade
« Reply #240 on: June 23, 2017, 06:23:29 »
In the recent past, the "big" problem is overwhelming the database with connections. Basically the database starts to fall behind when things get busy, and connections stack up, eventually hitting our max (700*) at which point new connections are flat out refused. That's wen you see the "could not connect to database" error, which I am trying to resolve.

Yesterday's conversion will hopefully allow the database to respond faster, preventing a large "lineup" of connections from forming. Watching the performance, I did see it approach 700 yesterday, then immediately drop to ~400 as the requests were processed. I won't be so daring as to say the problem is fixed, but I believe things are improved. We will find out how much as we move forward.

Note that this does not in any way solve the "high server load" issue... in fact it may make it worse. Before, when processes were languishing about waiting to be served by the database, they weren't causing load. If we improve the database response time, the server has to do more work, faster to keep up. See the attached technical diagram for a more detailed explanation.


Cheers
Mike

* 700 does not mean only 700 users can be visiting the site. It's significantly more complicated, but essentially the web server forks processes to handle requests (each process handles many simultaneous and sequential requests), and each process then generally creates one connection.

Offline kev994

  • Army.ca Subscriber
  • Full Member
  • *
  • 4,215
  • Rate Post
  • Posts: 255
Re: Major Site Upgrade
« Reply #241 on: June 23, 2017, 08:10:37 »
I wasn't following until I saw the diagram :)

Offline dapaterson

    Halfway to being an idiot-savant.

  • Army.ca Subscriber
  • Army.ca Myth
  • *
  • 543,775
  • Rate Post
  • Posts: 18,495
Re: Major Site Upgrade
« Reply #242 on: June 23, 2017, 09:29:59 »
Having big issues connecting with TapaTalk; the mobile site on Chrome works fine.  Mind you, I stopped updating TapaTalk several versions ago; it wants too much access and I do not have a good sandbox in system for Android.
Putting the *** in acerbic.

Offline kev994

  • Army.ca Subscriber
  • Full Member
  • *
  • 4,215
  • Rate Post
  • Posts: 255
Re: Major Site Upgrade
« Reply #243 on: June 23, 2017, 09:52:42 »
Having big issues connecting with TapaTalk; the mobile site on Chrome works fine.  Mind you, I stopped updating TapaTalk several versions ago; it wants too much access and I do not have a good sandbox in system for Android.
Updated iPhone version is working for me.

Offline Mike Bobbitt

  • Army.ca Owner
  • Directing Staff
  • Army.ca Legend
  • *
  • 126,920
  • Rate Post
  • Posts: 12,280
    • Army.ca
Re: Major Site Upgrade
« Reply #244 on: June 23, 2017, 11:28:39 »
Currently at 72 connections so "all is good" in that regard.

Offline Mike Bobbitt

  • Army.ca Owner
  • Directing Staff
  • Army.ca Legend
  • *
  • 126,920
  • Rate Post
  • Posts: 12,280
    • Army.ca
Re: Major Site Upgrade
« Reply #245 on: June 23, 2017, 16:08:15 »
A quick update with good news... Immediately after the conversion, there was not much difference in the performance and number of connections (we even peaked at our max of 700 at least once). However we are now looking much better.

For those who care, the database uses I/O credits to allow bursts of activity, but limit prolonged high-utilization. (I.E. when you use all your credits, things slow to a crawl until the credits regenerate.) Yesterday's conversion consumed all available credits, as it was a pretty hefty operation. The credits regenerate over time, but the backlog took some time to burn off. Today, we are finally hitting our normal operating state, with I/O credits regenerated and the server running normally.

With over 700 active users, we hit a max of 60 connections, but right now are sitting at zero. As in no pending connections at all. When I first checked, I thought the site was down (the only other cause for seeing zero connections in the past.)

So this is great news, it means that yesterday's change was very effective.

Now we just need to put wheels on the donkey so it can keep up. ;)


Cheers
Mike

Offline Mike Bobbitt

  • Army.ca Owner
  • Directing Staff
  • Army.ca Legend
  • *
  • 126,920
  • Rate Post
  • Posts: 12,280
    • Army.ca
Re: Major Site Upgrade
« Reply #246 on: July 07, 2017, 14:21:30 »
A couple of weeks with only the regular amount of issues... I think we can put this one to bed and move on to the next one. :)

Offline BeyondTheNow

  • Directing Staff
  • Army.ca Veteran
  • *
  • 113,870
  • Rate Post
  • Posts: 1,789
Re: Major Site Upgrade
« Reply #247 on: July 19, 2017, 18:54:06 »
A couple of weeks with only the regular amount of issues... I think we can put this one to bed and move on to the next one. :)

Thanks for the update Mike!
”You don’t have a right to the cards you believe you should have been dealt. You have an obligation to play the hell out of the ones you’re holding. ”
~Cheryl Strayed

“The aim of argument, or of discussion should not be victory, but progress.”~Joseph Joubert