Results 1 to 7 of 7

Thread: A debrief of the outage we had over the weekend.

  1. #1
    Child of the One True King Raphael's Avatar
    Join Date
    Jan 2014
    Location
    Middle Earth, New Zealand
    Faith
    Christian
    Gender
    Male
    Posts
    4,710
    Amen (Given)
    2317
    Amen (Received)
    2675

    A debrief of the outage we had over the weekend.

    So here is what happened with the outage.

    There were complaints regarding some users randomly being redirected to a site that tried to offload malware on them (myfilestore). From reading up on it this was related to a vulnerability in the vBulletin software that allowed there to be data injected into some SEO templates.

    I logged in to check the settings, and decided to finally allow the Ubuntu 16.04 LTS update to happen. I foolishly didn't take a snapshot before hand, but DigitalOcean wants you to have the Virtual Machine shutdown when you do a snapshot, so I skipped it. dumb mistake.

    And that's where the problems started. The site just wasn't coming back up, more frantic investigations and I found that PHPv5 had been uninstalled as it isn't supported by Ubunti 16.04 (took me a while because the error message was complaining about something else, not php)
    I installed PHP7, and got the site back up, as I knew there were some issues relating to our version of vBulletin and PHP7, I did some small changes to allow the site to run, but in testing there were some issues with posting, so I turned the forum off (so the site was at least displaying an outage message). It was now 4:30am, and I had started at 8:30pm. So I went off for a few hours sleep.

    I was up again at 9, with my beloved topping me up with copious amounts of coffee, and got stuck back into trying to fix things so that I could make posts (when the board is turned off, moderators and the like can still post). Sparko in the mean time had tried a few things, but they had been about as successful as my attempts.

    As I was working through the day, with the odd break to make sure the kids were listening to their mother ...school summer holidays, we're ready for them to be over... I wasn't making much headway, I would fix something to make it compatible with php7, and then the next item would start giving errors. So when it got towards Saturday evening I decided that the best option was going to be to try and backup the database, restore to the previous VM backup, and then try and restore the database.
    With that in mind I backed up the db and started a download of it to my pc. It's a 9GB file so I knew it would take a while and left it overnight. However we had a BIG storm on Saturday night, and my internet connection dropped out and the download didn't complete.
    I didn't restart it on Sunday morning as I was on Sound at church and had to be there early.

    Come Sunday afternoon and I got the download restarted, and it took 3 hours to complete(the internet connection was still not overly happy following the storm so was running slower), but I had the DB backed up off the VM and was then able to restore the backup from 2 days before the outage.

    I then spoke to Sparko and said that while I could restore the DB, it was going to take up to 6 hours to upload to the server before I could restore it, and I really wanted the site back up and running. So we made an executive decision to not proceed with restoring the missing two days, and just to turn the forum back on, I then applied the suggested advice regarding redirects that started this whole thing.


    Where do we go from here with it?
    We're investigating our options around upgrading to vBulletin 5 as it supports both php5 and php7, we'll give more information on that later in the year as there is a licence cost involved, and people have been not overly happy with vBulletin5 although it seems to have improved in the last couple of years, so we will want to test it out in a Dev environment before we make any final decisions.
    vBulletin is working on a version of v4 that does support php7 but it's still in it's early beta stages with no release date in sight.
    Once we've done that we will look at running the update to Ubuntu 16.04.
    Last edited by Sparko; 01-24-2017 at 12:07 PM.
    TheologyWeb Needs YOU! - Please Help Us Upgrade Tweb! Click here for more info


    "If you can ever make any major religion look absolutely ludicrous, chances are you haven't understood it"
    -Ravi Zacharias, The New Age: A foreign bird with a local walk

    Be watchful, stand firm in the faith, act like men, be strong.
    1 Corinthians 16:13

    "...he [Doherty] is no historian and he is not even conversant with the historical discussions of the very matters he wants to pontificate on."
    -Ben Witherington III

  2. Amen TheWall amen'd this post.
  3. #2
    tWebber
    Join Date
    Aug 2016
    Faith
    Christian
    Gender
    Male
    Posts
    2,083
    Amen (Given)
    964
    Amen (Received)
    1053
    So me and sparko noticing this helped save tweb. Wow. Thank you Raphael.

  4. #3
    Child of the One True King Raphael's Avatar
    Join Date
    Jan 2014
    Location
    Middle Earth, New Zealand
    Faith
    Christian
    Gender
    Male
    Posts
    4,710
    Amen (Given)
    2317
    Amen (Received)
    2675
    Quote Originally Posted by TheWall View Post
    So me and sparko noticing this helped save tweb. Wow. Thank you Raphael.
    More like it caused the outage. I would have run the update if I hadn't logged in to check the issue you guys found
    TheologyWeb Needs YOU! - Please Help Us Upgrade Tweb! Click here for more info


    "If you can ever make any major religion look absolutely ludicrous, chances are you haven't understood it"
    -Ravi Zacharias, The New Age: A foreign bird with a local walk

    Be watchful, stand firm in the faith, act like men, be strong.
    1 Corinthians 16:13

    "...he [Doherty] is no historian and he is not even conversant with the historical discussions of the very matters he wants to pontificate on."
    -Ben Witherington III

  5. #4
    tWebber Bisto's Avatar
    Join Date
    Feb 2016
    Location
    Chile
    Faith
    Christian
    Gender
    Male
    Posts
    678
    Amen (Given)
    301
    Amen (Received)
    205
    Quote Originally Posted by Raphael View Post
    I was up again at 9, with my beloved topping me up with copious amounts of coffee, and got stuck back into trying to fix things so that I could make posts (when the board is turned off, moderators and the like can still post). Sparko in the mean time had tried a few things, but they had been about as successful as my attempts.
    Now Tweb was without form and void, and errors covered the screen of the users, and the posts of Raphael moved over the face of the forums. And Sparko said, "Let us go back two days", and behold, we did...


    Glad to know you were able to handle it, bro!
    Last edited by Sparko; 01-24-2017 at 12:08 PM.
    We are therefore Christ's ambassadors, as though God were making his appeal through us. We implore on Christ's behalf: 'Be reconciled to God!!'
    - 2 Corinthians 5:20.
    In deviantArt: ll-bisto-ll.deviantart.com
    Christian art and more: Christians.deviantart.com

  6. Amen DesertBerean amen'd this post.
  7. #5
    Troll Magnet Sparko's Avatar
    Join Date
    Jan 2014
    Faith
    Christian
    Gender
    Male
    Posts
    30,498
    Amen (Given)
    2566
    Amen (Received)
    15109
    Just a note. In order to update the software ($359) and run a separate server for development ($25/month for as long as it takes, probably a few months) we are going to need some financial help from you guys. I will probably be starting a donation drive thread soon, but in the meantime if anyone wants to help out with a donation, please head over to our donation page. There you can sign up for a monthly donation of as little as $5/month, or make a one time donation of any amount.

    http://www.theologyweb.com/campus/fa...ologywebdonate

  8. #6
    Oops....... mossrose's Avatar
    Join Date
    Jan 2014
    Faith
    slave & child of Christ
    Gender
    Female
    Posts
    15,039
    Amen (Given)
    10898
    Amen (Received)
    6237
    Raphael is amazing. Thank you, dear knight!


    Securely anchored to the Rock amid every storm of trial, testing or tribulation.

  9. Amen The Melody Maker amen'd this post.
  10. #7
    Theologyweb's Official Grandfather Jedidiah's Avatar
    Join Date
    Jan 2014
    Location
    Peter's Creek, Alaska
    Faith
    Christian
    Gender
    Male
    Posts
    10,293
    Amen (Given)
    15494
    Amen (Received)
    5448
    Quote Originally Posted by mossrose View Post
    Raphael is amazing. Thank you, dear knight!
    Yeah, Raph, your efforts are much appreciated.
    Micah 6:8 He has told you, O man, what is good; and what does the LORD require of you but to do justice, and to love kindness, and to walk humbly with your God?

  11. Amen mossrose, Christianbookworm, Bisto amen'd this post.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •