selfhosted-apps-docker/kopia_backup

Kopia

guide-by-example

WORK IN PROGRESS
WORK IN PROGRESS
WORK IN PROGRESS

Purpose & Overview

Backups.

• Official site
• Github

Kopia is a new open source backup utility with basicly all modern features.
Cross-platform, deduplication, encryption, compression, multithreaded speed, native cloud storage support, repository replication, snapshots mounting, GUI versions, server version,...

Written in golang.

Some aspects of Kopia

There are 3 ways to go about running kopia

• cli - Command line.
  You call the binary passing some commands, it executes stuff, done.
  Requires extra work - scripts with configs, scheduling.
• KopiaUI - GUI version of kopia.
  Easier managment, takes uppon itself scheduling.
  Drawback is that it runs under one user and only when logged in.
• Kopia Server - running kopia and its webserver in the background
  Managment through web browser at an url, can run as a docker container.

Official Getting Started Guide
Official Features

• Kopia is a single ~35MB binary file.
• Backups are stored in a repository that needs to be created first, and is always encrypted - requires password.
  • Before any action, Kopia needs to connecto to a repo.
• Snapshot, appart from typical meaning, kopia also uses it to for targets(paths) to be backed up.
• Policy is a term used to define behaviour of the backup/repo, like backups retention, what to ignore, logging, scheduling(server/UI), actions before and after backup,...
• Policies are stored inside a repo and can apply at various levels and can inherit from each other
  • global policy, the default that comes predefined during repo creation
  • per user policy and per machine policy
  • snapshot level policy, only applying for that one path
• Maintence is automatic
• ..

Kopia on a linux machine

cli version of kopia will be used, with a script and systemd-timers for scheduling.

install kopia

for arch linux, kopia is on AUR yay kopia-bin

the initial steps

General use of sudo so that kopia has access everywhere.

• repo creation

mkdir -p /mnt/mirror/KOPIA/docker_host_kopia
sudo kopia repo create filesystem --path /mnt/mirror/KOPIA/docker_host_kopia
sudo kopia repo connect filesystem --path /mnt/mirror/KOPIA/docker_host_kopia
sudo kopia repo status

• the policy info and change

sudo kopia policy get --global
sudo kopia policy list
sudo kopia policy set --global --ignore-cache-dirs true --keep-annual 1 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 14

• manual run

sudo kopia snapshot create /home/spravca/docker /etc
sudo kopia snapshot list

• mounting a backup

sudo kopia mount all /mnt/tmp & - mounts all snapshots
sudo kopia snapshot list
sudo kopia mount k7e2b0a503edd7604ff61c68655cd5ad7 /mnt/tmp &
sudo umount /mnt/tmp

The backup script

/opt/kopia-backup-home-etc.sh

#!/bin/bash

#sudo kopia policy set --global --keep-annual 1 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 14

REPOSITORY_PATH='/mnt/mirror/KOPIA/docker_host_kopia'
BACKUP_THIS='/home /etc'
KOPIA_PASSWORD='aaa'

kopia repository connect filesystem --path $REPOSITORY_PATH --password $KOPIA_PASSWORD
kopia snapshot create $BACKUP_THIS
kopia repository disconnect

make the script executable
sudo chmod +x /opt/kopia-backup-home-etc.sh

Scheduled backups using systemd

Usually cron is used, but systemd provides better logging and control, so better get used to using it.
Heres some discussion on unit files.
ntfy can be used for notifications, more info here

• sudo micro /etc/systemd/system/kopia-home-etc.service

kopia-home-etc.service

[Unit]
Description=kopia backup
Wants=network-online.target
After=network-online.target
ConditionACPower=true
# OnFailure=ntfy@failure-%p.service
# OnSuccess=ntfy@success-%p.service

[Service]
Type=oneshot

# Lower CPU and I/O priority.
Nice=19
CPUSchedulingPolicy=batch
IOSchedulingPriority=7

IPAccounting=true
PrivateTmp=true
Environment="HOME=/root"

ExecStart=/opt/kopia-backup-home-etc.sh

• sudo micro /etc/systemd/system/kopia-home-etc.timer

kopia-home-etc.timer

[Unit]
Description=Run kopia backup

[Timer]
OnCalendar=*-*-* 02:00:00
RandomizedDelaySec=10min
Persistent=true

[Install]
WantedBy=timers.target

• sudo systemctl enable --now kopia-home-etc.timer
• systemctl status kopia-home-etc.timer
• journalctl -u kopia-home-etc.timer - see history

Mounting network storage using systemd

• files are placed in /etc/systemd/system
• the name of mount and automount files MUST correspond with the path
  replacing / with a -, but otherwise it must be the mounting path in the name
• for mounting that does not fail on boot if there are network issues, and mounts the target only on request - enable automount file, not mount file, so:
  sudo systemctl enable mnt-mirror.automount

mnt-mirror.mount

[Unit]
Description=3TB truenas mirror mount

[Mount]
What=//10.0.19.11/Mirror
Where=/mnt/mirror
Type=cifs
Options=rw,username=kopia,password=aaa,file_mode=0644,dir_mode=0755,uid=1000,gid=1000

[Install]
WantedBy=multi-user.target

mnt-mirror.automount

[Unit]
Description=3TB truenas mirror mount

[Automount]
Where=/mnt/mirror

[Install]
WantedBy=multi-user.target

Remote backup

... some day ...

Kopia in Windows

Kopia Server on Windows

• download this repo, delete shit, keep kopia_server_deploy folder
• run DEPLOY.cmd, it will
  • Removes powershell scripts restriction.
  • Creates folder C:\Kopia and kopies files there
  • imports a task schedule that will start Kopia Server at boot
• visit in browser localhost:51515
• setup repo
• setup what to backup and schedule
• edit the kopia_backup_scipt.ps1, set what to backup and where
• for the same repo location execute
  sudo kopia repository create filesystem --path C:\Backup
• run the script

Kopia cli on Windows

While GUI version seems like a way to go.. IMO its not there yet. The schedule seems to be dependant on user logging in... and general weird feeling of poor GUI quality.

So here goes cli version. As after some hands-on experience with cli version the GUI version might click in better too, later.

• download this repo, delete shit, keep kopia-deploy folder
• run DEPLOY.cmd, it will
  • Removes powershell scripts restriction.
  • Install scoop, sudo, kopia.
  • Creates folder C:\Kopia and kopies there
    kopia_backup_scipt.ps1 and the VSS ps1 before and after files.
  • imports a task schedule
• edit the kopia_backup_scipt.ps1, set what to backup and where
• for the same repo location execute
  sudo kopia repository create filesystem --path C:\Backup
• run the script

To do the above things manualy:

• install kopia using scoop

  • open terminal as admin
  • Set-ExecutionPolicy RemoteSigned
  • iex "& {$(irm get.scoop.sh)} -RunAsAdmin"
  • scoop install sudo --global
  • sudo scoop install kopia --global

• download this repo, extract to c:\kopia

• edit kopia-backup-home-etc.sh as see fit

• powershell as as administrator

• --enable-actions

• in tray, right click on the icon - Launch At Startup
  this creates registry entry - HKCU\Software\Microsoft\Windows\CurrentVersion\Run\KopiaUI

kopia policy set <target_dir> --before-folder-action "powershell -WindowStyle Hidden <path_to_script>\before.ps1" kopia policy set <target_dir> --after-folder-action "powershell -WindowStyle Hidden <path_to_script>\after.ps1"