Announcement

Collapse

Newsdesk Guidelines

The NEWSDESK is the area for TheologyWeb news and announcements. This is not a debate area. There will be times when you won't agree with certain official announcements and commentary. If so, do not argue or dispute in this area but take it up in an appropriate area of the forum or by Private Message. Threads may only be started by TheologyWeb leadership, but responses may be posted by the entire community.

General TheologyWeb forum rules: here.
See more
See less

A debrief of the outage we had over the weekend.

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • A debrief of the outage we had over the weekend.

    So here is what happened with the outage.

    There were complaints regarding some users randomly being redirected to a site that tried to offload malware on them (myfilestore). From reading up on it this was related to a vulnerability in the vBulletin software that allowed there to be data injected into some SEO templates.

    I logged in to check the settings, and decided to finally allow the Ubuntu 16.04 LTS update to happen. I foolishly didn't take a snapshot before hand, but DigitalOcean wants you to have the Virtual Machine shutdown when you do a snapshot, so I skipped it. dumb mistake.

    And that's where the problems started. The site just wasn't coming back up, more frantic investigations and I found that PHPv5 had been uninstalled as it isn't supported by Ubunti 16.04 (took me a while because the error message was complaining about something else, not php)
    I installed PHP7, and got the site back up, as I knew there were some issues relating to our version of vBulletin and PHP7, I did some small changes to allow the site to run, but in testing there were some issues with posting, so I turned the forum off (so the site was at least displaying an outage message). It was now 4:30am, and I had started at 8:30pm. So I went off for a few hours sleep.

    I was up again at 9, with my beloved topping me up with copious amounts of coffee, and got stuck back into trying to fix things so that I could make posts (when the board is turned off, moderators and the like can still post). Sparko in the mean time had tried a few things, but they had been about as successful as my attempts.

    As I was working through the day, with the odd break to make sure the kids were listening to their mother ...school summer holidays, we're ready for them to be over... I wasn't making much headway, I would fix something to make it compatible with php7, and then the next item would start giving errors. So when it got towards Saturday evening I decided that the best option was going to be to try and backup the database, restore to the previous VM backup, and then try and restore the database.
    With that in mind I backed up the db and started a download of it to my pc. It's a 9GB file so I knew it would take a while and left it overnight. However we had a BIG storm on Saturday night, and my internet connection dropped out and the download didn't complete.
    I didn't restart it on Sunday morning as I was on Sound at church and had to be there early.

    Come Sunday afternoon and I got the download restarted, and it took 3 hours to complete(the internet connection was still not overly happy following the storm so was running slower), but I had the DB backed up off the VM and was then able to restore the backup from 2 days before the outage.

    I then spoke to Sparko and said that while I could restore the DB, it was going to take up to 6 hours to upload to the server before I could restore it, and I really wanted the site back up and running. So we made an executive decision to not proceed with restoring the missing two days, and just to turn the forum back on, I then applied the suggested advice regarding redirects that started this whole thing.


    Where do we go from here with it?
    We're investigating our options around upgrading to vBulletin 5 as it supports both php5 and php7, we'll give more information on that later in the year as there is a licence cost involved, and people have been not overly happy with vBulletin5 although it seems to have improved in the last couple of years, so we will want to test it out in a Dev environment before we make any final decisions.
    vBulletin is working on a version of v4 that does support php7 but it's still in it's early beta stages with no release date in sight.
    Once we've done that we will look at running the update to Ubuntu 16.04.
    Last edited by Sparko; 01-24-2017, 07:07 AM.
    Be watchful, stand firm in the faith, act like men, be strong.
    1 Corinthians 16:13

    "...he [Doherty] is no historian and he is not even conversant with the historical discussions of the very matters he wants to pontificate on."
    -Ben Witherington III

  • #2
    So me and sparko noticing this helped save tweb. Wow. Thank you Raphael.
    sigpic

    Comment


    • #3
      Originally posted by TheWall View Post
      So me and sparko noticing this helped save tweb. Wow. Thank you Raphael.
      More like it caused the outage. I would have run the update if I hadn't logged in to check the issue you guys found
      Be watchful, stand firm in the faith, act like men, be strong.
      1 Corinthians 16:13

      "...he [Doherty] is no historian and he is not even conversant with the historical discussions of the very matters he wants to pontificate on."
      -Ben Witherington III

      Comment


      • #4
        Originally posted by Raphael View Post
        I was up again at 9, with my beloved topping me up with copious amounts of coffee, and got stuck back into trying to fix things so that I could make posts (when the board is turned off, moderators and the like can still post). Sparko in the mean time had tried a few things, but they had been about as successful as my attempts.
        Now Tweb was without form and void, and errors covered the screen of the users, and the posts of Raphael moved over the face of the forums. And Sparko said, "Let us go back two days", and behold, we did...


        Glad to know you were able to handle it, bro!
        Last edited by Sparko; 01-24-2017, 07:08 AM.
        We are therefore Christ's ambassadors, as though God were making his appeal through us. We implore on Christ's behalf: 'Be reconciled to God!!'
        - 2 Corinthians 5:20.
        In deviantArt: ll-bisto-ll.deviantart.com
        Christian art and more: Christians.deviantart.com

        Comment


        • #5
          Just a note. In order to update the software ($359) and run a separate server for development ($25/month for as long as it takes, probably a few months) we are going to need some financial help from you guys. I will probably be starting a donation drive thread soon, but in the meantime if anyone wants to help out with a donation, please head over to our donation page. There you can sign up for a monthly donation of as little as $5/month, or make a one time donation of any amount.

          http://www.theologyweb.com/campus/fa...ologywebdonate

          Comment


          • #6
            Raphael is amazing. Thank you, dear knight!


            Securely anchored to the Rock amid every storm of trial, testing or tribulation.

            Comment


            • #7
              Originally posted by mossrose View Post
              Raphael is amazing. Thank you, dear knight!
              Yeah, Raph, your efforts are much appreciated.
              Micah 6:8 He has told you, O man, what is good; and what does the LORD require of you but to do justice, and to love kindness, and to walk humbly with your God?

              Comment

              widgetinstance 221 (Related Threads) skipped due to lack of content & hide_module_if_empty option.
              Working...
              X