How to automate your Lego Universe Server Backups and Updates
In the previous article, we learned how to set up a Lego Universe Server using the Darkflame Universe project. In this guide we will explore how we can automate our server’s backups and updates to run the server as autonomously as possible.
Repository Setup
First, and perhaps most importantly, we are going to handle server backups. This includes your users’ data, as well as your server configurations for a quick reboot after a server update.
An easy way of achieving this is using a Github repository (note: we will not be storing any of the server’s build files on our repository) — here the Github repository is called LegoUniverseServer
. Therefor, go onto Github and create a new blank repository and then clone it onto your Lego Universe server’s machine.
Navigate into the cloned repository, and create the following directories (my repo is called LegoUniverseServer
):
LegoUniverseServer/
├── build/
├── darkflame_database/
├── fix_database/
├── ini_files/
├── scripts/
└── server_status/
The scripts folder is unused in this guide, but can house the account management tools found on my Gist.
SQL Import/Export
To back up and restore our users’ data on the fly, were going to automate these operations first. Navigate into the fit_database/
directory and create a file export_database
with the following contents:
rm ../darkflame_database/darkflame.sql
mysqldump -u dflame -pdflame darkflame > ../darkflame_database/darkflame.sql
cd ../darkflame_database/ && ./backup_to_discord
(*) The last line is optional, and allows uploading the exported database to discord, such that your users can anytime leave the server without having their data lost (user passwords are excluded from this, more on that later).
Then create another file called import_database
with the following contents:
sudo service mysql start
mysql -u dflame -pdflame darkflame < ../darkflame_database/darkflame.sql
Then set both scripts to be marked as executable
parent_dir/LegoUniverseServer/fix_database ~> chmod +x ./export_database
parent_dir/LegoUniverseServer/fix_database ~> chmod +x ./import_database
With both of these scripts configured, you can now export the database using ./export_database
and re-import it using ./import_database
Automating Server Updates
This part is a little bit more tricky, as the server automatically overwrites any files during the update process. ini files sometimes get updated and settings may not persist through updates. Therefore, were going to work around that.
Head back to the root of your new repository and create a file called updateServer
with the following contents:
pkill MasterServer
cd ../Github/DarkflameServer
git pull
git submodule update --recursive
cd ../../LegoUniverseServer/build
sed -i -e 's/add_subdirectory(tests)/#add_subdirectory(tests)/g' ../CMakeLists.txt
cmake .. && cmake --build . --config Release -j4
cd ..
git checkout CMakeLists.txt
./server_status/discord_status_notify updated > /dev/null 2>&1
As you may see, this script assumes that the DarkflameServer repository is present within the parent directory of your current repository. To adjust for this, navigate out of your current repository, then create a child directory called Github
and move the DarkflameServer
directory into the Github
directory. At the end, you should have the following folder structure:
parent_dir/
├── Github/
| └── DarkflameServer/
| └── ...
└── LegoUniverseServer/
Navigate back into your new repository and create a second file (at the root level) called fixlnks
with the following contents:
rm CMakeLists.txt
rm CMakeVariables.txt
rm dAuthServer
rm dChatFilter
rm dChatServer
rm dCommon
rm dDatabase
rm dGame
rm dMasterServer
rm dNet
rm dPhysics
rm dScripts
rm dWorldServer
rm dZoneManager
rm migrations
rm resources
rm thirdparty
rm vanity
rm versions.txt
ln -s ../Github/DarkflameServer/CMakeLists.txt CMakeLists.txt
ln -s ../Github/DarkflameServer/CMakeVariables.txt CMakeVariables.txt
ln -s ../Github/DarkflameServer/dAuthServer dAuthServer
ln -s ../Github/DarkflameServer/dChatFilter dChatFilter
ln -s ../Github/DarkflameServer/dChatServer dChatServer
ln -s ../Github/DarkflameServer/dCommon dCommon
ln -s ../Github/DarkflameServer/dDatabase dDatabase
ln -s ../Github/DarkflameServer/dGame dGame
ln -s ../Github/DarkflameServer/dMasterServer dMasterServer
ln -s ../Github/DarkflameServer/dNet dNet
ln -s ../Github/DarkflameServer/dPhysics dPhysics
ln -s ../Github/DarkflameServer/dScripts dScripts
ln -s ../Github/DarkflameServer/dWorldServer dWorldServer
ln -s ../Github/DarkflameServer/dZoneManager dZoneManager
ln -s ../Github/DarkflameServer/migrations migrations
ln -s ../Github/DarkflameServer/resources resources
ln -s ../Github/DarkflameServer/thirdparty thirdparty
ln -s ../Github/DarkflameServer/vanity vanity
ln -s ../Github/DarkflameServer/versions.txt versions.txt
Then, mark both of your newly created files as executable
parent_dir/LegoUniverseServer ~> chmod +x ./updateServer
parent_dir/LegoUniverseServer ~> chmod +x ./fixlnks
The fixlnks
script sets up symbolic links to the files within the DarkflameServer repository, allowing you to update it independently of your server files. The updateServer
script automates the update process, first pulling the new files to the DarkflameServer
repository, then re-executing the build process for the server.
Automating Server Configs
This section focuses mainly on maintaining the server settings when migrating from one (physical) server to another, allowing quick re-setup of the Lego Universe Server.
Navigate into the ini_files
directory and create a file called fix_ini_files
with the following contents:
cd ../build
ln -s ../ini_files/authconfig.ini authconfig.ini
ln -s ../ini_files/chatconfig.ini chatconfig.ini
ln -s ../ini_files/masterconfig.ini masterconfig.ini
ln -s ../ini_files/worldconfig.ini worldconfig.ini
ln -s ../ini_files/sharedconfig.ini sharedconfig.ini
Then, copy over the ini files you permanently use into this ini_files
directory. The file we just created will allow the server to look for the configs in this directory in the future, rather than in the original build
directory.
Finally, mark the file as executable
parent_dir/LegoUniverseServer/ini_files ~> chmod +x fix_ini_files
Automating Discord Backups
Finally, we add the discord backup. This allows your users to download their data and use it on their own if they ever decide to leave your community.
Navigate into the darkflame_database
folder in your new repo, and create a file called backup_to_discord
with the following contents
#!/usr/bin/env python3
import discord
import datetime
TOKEN = 'YOUR_API_KEY'
filename = 'darkflame.sql'
data = []
with open(filename, 'rb') as f:
for data in f.readlines():
if b'INSERT INTO `accounts` VALUES' in data:
break
if data == []:
print('Password hashes not found! Exiting!')
exit(-1)
data = data.replace(b'INSERT INTO `accounts` VALUES (', b'').replace(b');\n', b'').split(b'),(')
hashes = []
i = 0
for entry in data:
entry = entry.decode().split(',')
entry[2] = "''"
entry = ','.join(entry)
data[i] = entry
i += 1
statement = 'INSERT INTO `accounts` VALUES (' + '),('.join(data) + ');\n'
new_filename = 'darkflame_backup.sql'
with open(new_filename, 'wb+') as w:
with open(filename, 'rb') as f:
for data in f.readlines():
if b'INSERT INTO `accounts` VALUES' in data:
w.write(bytes(statement, 'utf-8'))
else:
w.write(data)
client = discord.Client(intents=discord.Intents.default())
@client.event
async def on_ready():
channel = client.get_channel(YOUR_CHANNEL_ID)
await channel.send(f'Latest SQL data dump ({datetime.datetime.now()})', file=discord.File(new_filename))
exit(0)
client.run(TOKEN)
(*) Make sure to replace YOUR_API_KEY and YOUR_CHANNEL_ID with your Discord API key and Channel IDs.
This python script allows backing up the data to Discord, while removing user passwords for privacy reasons. To run this module, you need the discord
python library which you can install using pip3 install discord
Next, just mark the file as executable chmod +x ./backup_to_discord
Putting It All Together
With all the above scripts, we just need to automate the entire process. We are going to leave server updates as a manual task, as it only needs to be carried out once in a while. Data backups however, are going to be scheduled as a daily task.
To automate the data backups (both to Github and Discord), we are going to create a new cron job that runs at 1 AM ever day. To create this job, execute the command crontab -e
to edit the cron tab, then paste the following job at the bottom of the crontab (be sure to replace the path to your repository with the correct one)
0 1 * * * cd /path/to/LegoUniverseServer/fix_database/ && ./export_database && cd .. && git add . && git commit -m "database backup `date`" && git push
This will execute the export_database
script we created earlier, which will automatically export the current darkflame
database to an SQL file, store it within the darkflame_database
subdirectory, push an updated copy without passwords into a Discord community channel and then upload the database backup to Github, tagging the commit with the current date and time. Viola, automation!
(*) It should go without saying, but pushing to git requires you to have the permission to access the repository. I would not recommend leaving a trusted ssh key on the server just for the sake of backups, but rather using an access token for pushing the backups (Repo settings > Deploy keys > Add the ssh key there and enable write access).
To update the server at any given time, terminate the current instance, then navigate into the root of your new repository and just execute the command ./updateServer
and then re-run the server after the update is done.
Restoring/Migrating Server to New Server
With all these tools, it is quite straightforward to migrate your server to new infrastructure.
First, start off by shutting down your server and executing the ./export_database
script. Then, push all the changes to Github.
On the new server, create a parent directory and clone your repository into it. Within the root of this parent directory, create a Github
directory and clone into it the DarkflameServer
repository (with all its submodules). After doing this, you can ignore the Github
directory and proceed with your repository.
Navigate into the root of your repository and execute the fixlnks
script. This will automatically set up the symbolic links to the DarkflameServer repository. Then, navigate into the ini_files
directory and execute the fix_ini_files
script.
To build the server, simply execute the updateServer
script in the root of your repository.
Once done, navigate into the fix_database
directory and execute the import_database
script to re-import all the data from the old infrastructure.
With these steps done, all your previous infrastructure should be replicated on the new machine, and you can simply navigate into the build
directory and execute the ./runserver
script provided below (or however you choose to run the server).
Make sure to execute the script from their predefined directories. Since the scripts use relative paths, make sure to adhere to my above documentation as otherwise errors will occur during the execution.
Add-On: Server Run Script
To notify your Discord users of server restarts, you can add the following runserver
file to your new repository’s build
directory
/path/to/LegoUniverseServer/server_status/discord_status_notify start > /dev/null 2>&1
cd /path/to/LegoUniverseServer/build && \
sudo setcap 'cap_net_bind_service=+ep' AuthServer && \
./MasterServer
/path/to/LegoUniverseServer/server_status/discord_status_notify stop > /dev/null 2>&1
Under the directory server_status
create the script discord_status_notify
with the following contents
#!/usr/bin/env python3
import discord
import sys
import datetime
TOKEN = 'YOUR_API_KEY'
if len(sys.argv) < 2:
print("Usage: './discord_status_notify start' or './discord_status_notify stop'")
exit(-1)
mode = sys.argv[1]
client = discord.Client(intents=discord.Intents.default())
@client.event
async def on_ready():
channel = client.get_channel(YOUR_CHAT_ID)
if mode == 'start':
await channel.send(f'Server started up! Time: ({datetime.datetime.now()})')
elif mode == 'stop':
await channel.send(f'Server shut down! Time: ({datetime.datetime.now()})')
elif mode == 'updated':
await channel.send(f'Server was updated! Time: ({datetime.datetime.now()})')
exit(0)
client.run(TOKEN)
(*) Make sure to replace YOUR_API_KEY and YOUR_CHANNEL_ID with your Discord API key and Channel IDs.
Make sure to mark the script as executable using chmod +x ./discord_status_notify
The best way of running the server using this script is starting a screen
instance, then executing ./runserver
and then using the shortcut ctrl+a D
to detach from the instance.
As a final add-on, here is the .gitignore
file for the abovementioned automations
temp/
cmake-build-debug/
RelWithDebInfo/
build/CMakeFiles/
build/*
!build/res/
!build/locale/
!build/runserver
darkflame_database/*.ibd
darkflame_database/*.frm
darkflame_database/darkflame_backup.sql
# Third party libraries
thirdparty/mysql/
thirdparty/mysql_linux/
#CMakeVariables.txt
build/_deps/
# Build folders
linux_build/
# Codeblocks
*.layout
# User-specific files
*.rsuser
*.suo
*.user
*.userosscache
*.sln.docstates
# User-specific files (MonoDevelop/Xamarin Studio)
*.userprefs
# Build results
[Dd]ebug/
[Dd]ebugPublic/
[Rr]elease/
[Rr]eleases/
x64/
x86/
bld/
[Bb]in/
[Oo]bj/
[Ll]og/
# Visual Studio 2015/2017 cache/options directory
.vs/
# Files built by Visual Studio
*_i.c
*_p.c
*_h.h
*.ilk
*.meta
*.obj
*.iobj
*.pch
*.pdb
*.ipdb
*.idb
*.pgc
*.pgd
*.rsp
*.sbr
*.tlb
*.tli
*.tlh
*.tmp
*.tmp_proj
*_wpftmp.csproj
*.log
*.vspscc
*.vssscc
.builds
*.pidb
*.svclog
*.scc
*.lastbuildstate
# Visual C++ cache files
ipch/
*.aps
*.ncb
*.opendb
*.opensdf
*.sdf
*.cachefile
*.VC.db
*.VC.VC.opendb
# Visual Studio Log Files
*.tlog
*.log
# VS Code
.vscode
# Clion (IntelliJ)
.idea
.DS_Store
# Exceptions:
CMakeSettings.json
*.vcxproj
*.filters
*.cmake
CMakeCache.txt
#*.bin
CMakeFiles/3.17.3/CompilerIdC/CMakeCCompilerId.c
CMakeFiles/3.17.3/CompilerIdCXX/CMakeCXXCompilerId.cpp
CMakeFiles/3.17.3/VCTargetsPath.txt
CMakeFiles/3e6b105f924142e3e299a1a30e09566e/generate.stamp.rule
CMakeFiles/cmake.check_cache
CMakeFiles/generate.stamp
CMakeFiles/generate.stamp.depend
CMakeFiles/generate.stamp.list
CMakeFiles/TargetDirectories.txt
*.sln
*.recipe
# clangd
.cache
thirdparty/zlib-1.2.11/
Closing Words
I hope these tools help you in more efficiently handling your Lego Universe server. Should you have any questions, do not hesitate to leave them below this article or on my Gist.
If you like more cool tools, check out my Website which has a whole variety of useful tools and automations.