Frankenstein Blog

My Host Died?

I finally set aside some time to do a little site maintenance the other day. I wanted to do a few quick things. I wanted to backup the site, increase the volume size, and then backup the site again once finished. Note that I had no backups of the site EEEK I know I know shame on me. Anyways my task seemed simple, until I discovered that I couldn’t SSH into my web server. What gives? I took a look in EC2 and found the machine in a running state. Ok. I couldn’t hit the site via http. Ok. So I force restarted it via the EC2 console. Ok Machine came back up as per Amazon but I still couldn’t get into it. Le sigh. What to do? No backups dammit! I need to access the data on this virtual machine. Well I figured since I was going to resize the volume maybe I just extract the data I needed and restore it to a new instance, upgrading the operating system in the process, use a larger volume, and be done with it.

AWS CLI (Command Line Interface)

Let me introduce you to the Amazon Web Services Command Line Interface. This open source tool was developed in order to allow for programmatic access and administration to AWS services via command line. It’s available for various shells like bash, zsh, and tcsh as well as PowerShell. It’s super useful, easy to setup, and easily repeatable. Plus there are things you can only do via command line that you can’t do in the AWS GUI (so get comfortable with a terminal). My goal was to export a copy of my danielpaluszek.com virtual machine to an S3 bucket and then download it so I could extract the data I needed from it. I started by installing the CLI tool onto my MacOS laptop:
Installing the AWS CLI version 2 on macOS
Installing the pkg payload into the default location (/usr/local/aws-cli) puts the cli binary into your $PATH.
Once done I had to configure the tool to look at my AWS account:
Configuring the AWS CLI
Running the following command puts the cli tool into configuration mode where you can enter your account attributes. This allows the CLI tool to speak to your account:

dpaluszek@Dans-MacBook-Pro ~ % aws configure
AWS Access Key ID: 
AWS Secret Access Key:
Default region name: 
Default output format: 

After each of these prompts I pasted the keys from my account. I created a new user in IAM (Identity and Access Management) and used the newly generated keys. The region name is whatever region the machine(s) you wish to manipulate are in. The output format is the format in which results of commands are displayed. JSON is the default and is easily readable so I left it blank

So let’s get to it. I started by shutting down the VM in the EC2 console. I then created a new S3 bucket in my account and named it “danblog”. I configured S3 access to be public as required for me to download the contents of the bucket once I get the export in there. I ran the following command to initiate an export of the virtual machine:

aws ec2 create-instance-export-task --description "dan_blog" --instance-id i-0893e588d5fdf55e2 --target-environment vmware --export-to-s3-task DiskImageFormat=vmdk,ContainerFormat=ova,S3Bucket=danblog

Now this should return something along these lines:

{
    "ExportTask": {
       "Description": "dan_blog",
       "ExportTaskId": "export-i-050174cb06f17ecbe",
       "ExportToS3Task": {
         "ContainerFormat": "ova",
         "DiskImageFormat": "vmdk",
         "S3Bucket": "danblog",
         "S3Key": "export-i-050174cb06f17ecbe.ova"
       },
       "InstanceExportDetails": {
         "InstanceId": "i-0893e588d5fdf55e2",
         "TargetEnvironment": "vmware"
       },
       "State": "active"
     }
}

There is a command you can use to check the status of an operation (the export-task-id can be found in the previous output):

aws ec2 describe-export-tasks --export-task-ids export-i-050174cb06f17ecbe

This returns the same output as the create-instance-export-task command. Take note of the “State” entry. It will change from “active” once the process completes.

Down the Rabbit Hole, a Dead End?

So after the export completed I hopped into my S3 console and downloaded the ova file. I used the “Import Appliance” function in VirtualBox to create a VM from the ova file. I zipped through the prompts and booted it. I was met with a black screen. Le sigh. I had a rogue Ubuntu VM setup with MySQL I had setup a while back for a class I was taking that I could potentially use. I could mount the OVA file’s vmdk on this box and poke around. So I added the volume to the VM and booted Ubuntu. I checked the location of the disk and mounted it to a folder I created in my home folder. The file system seemed to be intact. I was able to access the web directory with no issue. The real question was whether or not I could get into the database. WordPress stores just about all content in a MySQL database, including the content of posts. That’s essentially, well, the site so without that I would lose everything. It’s not a ton of content but still.

I wasn’t too worried though the database is in my possession and I know the credentials so it was a matter of just getting in there. So I did the logical thing of changing the datadir directive in the /etc/mysql/mysql.conf.d/mysql.cnf file to point to the mysql directory on the “external” drive, stopped mysql, then started it. Now from here on things get a bit hazy. I saw a myriad of issues including but not limited to: mysql not successfully restarting due to permissions issues., attempting to authenticate in mysql but creds that I know work (taken from WordPress config file) do not work due to an authorization issue with the wordpressuser user account, root account reset attempts failing, amongst other things. I effectively Swiss Cheesed that drive up trying to get it to work. On a whim I decided to start over with a fresh copy of the ova downloaded from S3 and imported it into VirtualBox. While doing so I noticed that the Guest OS Type setting was set to Windows Server 2003 32 bit. I clicked through the options and set it to Ubuntu 64bit. Why would this make a difference? I thought selecting this was just a means to marking the vm graphic on theVirtualBox interface? Does it change anything boot related? There’s no such choice required in any other hypervisor I’ve played with. I did some research and I couldn’t find anything relating to that setting. The setting appears changeable in the “General” section in the “Basic” tab under a configured vm’s settings. When set to Ubuntu 64bit the machine booted, although it took about 15 minutes as it appeared hung on performing a random number generator function. I’ve read that processes such as these require entropy input into the system but I am unsure if me banging on the keyboard did anything to speed that up. Regardless the machine was booted and at the login prompt.

Now that I have the machine booted let’s see if I can log into it. This AWS machine didn’t have any creds that I knew about (perhaps I failed to record them?). By default the user AWS uses is “ubuntu” but my attempts at logging in failed. I am pretty sure I didn’t setup a password for any account on here. I did setup an ssh key via AWS for it, which I of course had as that was what I used to access the machine normally. So I grabbed the MAC address from VirtualBox and used it to find the machine’s IP via its DHCP lease in my firewall (I know I was surprised too). I was then able to SSH into the box using that IP. From here I was able to access everything normally. I zipped up the site’s root directory using tar. I then exported the MySQL WordPress database as a .sql file. I scp’d them to my local machine where I promptly backed hem up. Saved!

It’s Alive!

I switched to Digital Ocean for this machine. I wanted to try out their service based on the recommendation of a good tech friend by the name of Matthew Fox (you can peep his blog here: http://www.100781.org/). So I spun up an Ubuntu 20 machine, installed all the dependencies WordPress requires, and scp’d the tarball and sql export files to the machine. I unpacked the web directory back into place and modified permissions by chowning the www-user account and group. Next I setup a MySQL database for WordPress and imported the sql file. I had to create the WordPress user account configured in the WordPress config file in MySQL and assign it permissions. I had to do some apache virtualhost configuring in order to get apache serving the pages out of the right directory. I installed a letsencrypt cert using certbot, pointed apache to the correct certificate files, and ensured apache was serving over port 443 (and that a http–>https redirect was configured). The last issue I had to fix was a database connection issue when loading the site. I tracked this down to a difference in the database name from the database referenced in the WordPress php config file. It’s case sensitive so I changed it and the site loaded. Back in business.

Site is up. Backups secured.