diff --git a/kopia_backup/kopia_server_deploy_win/DEPLOY.cmd b/kopia_backup/kopia_server_deploy_win/DEPLOY.cmd
index e66a7ac..a32d801 100644
--- a/kopia_backup/kopia_server_deploy_win/DEPLOY.cmd
+++ b/kopia_backup/kopia_server_deploy_win/DEPLOY.cmd
@@ -41,10 +41,14 @@ if exist C:\Windows\System32\Tasks\kopia_server_backup_start (
echo - starting Kopia Server
schtasks /run /tn kopia_server_backup_start
+echo - copying link to Desktop
+robocopy "%~dp0\" "%USERPROFILE%\Desktop" "Kopia.url" /NDL /NJH /NJS
+
echo.
echo --------------------------------------------------------------
echo.
echo DEPLOYMENT DONE
echo KOPIA SERVER CAN NOW BE FIND AT WEB PAGE: localhost:51515
+echo A LINK SHOULD BE ON YOUR DESKTOP
echo.
pause
diff --git a/kopia_backup/kopia_server_deploy_win/Kopia.url b/kopia_backup/kopia_server_deploy_win/Kopia.url
new file mode 100644
index 0000000..fbd84cb
--- /dev/null
+++ b/kopia_backup/kopia_server_deploy_win/Kopia.url
@@ -0,0 +1,6 @@
+[InternetShortcut]
+URL=http://localhost:51515/
+IDList=
+HotKey=0
+[{000214A0-0000-0000-C000-000000000046}]
+Prop3=19,2
diff --git a/kopia_backup/readme.md b/kopia_backup/readme.md
index bc19fb7..8931940 100644
--- a/kopia_backup/readme.md
+++ b/kopia_backup/readme.md
@@ -7,65 +7,93 @@
WORK IN PROGRESS
WORK IN PROGRESS
WORK IN PROGRESS
+#Kopia-on-a-linux-machine
+* [Kopia on a linux machine](#Kopia-on-a-linux-machine)
+* [Kopia on a Windows machine](#Kopia-on-a-windows-machine)
+* [Kopia in Docker](#Kopia-in-Docker)
# Purpose & Overview
Backups.
* [Official site](https://kopia.io/)
+* [Official Forum](https://kopia.discourse.group/)
* [Github](https://github.com/kopia/kopia)
-Kopia is a new open source backup utility with basicly **all** modern features.
+Kopia is a new open source backup utility with basically **all** modern features.
Cross-platform, deduplication, encryption, compression, multithreaded speed,
native cloud storage support, repository replication, snapshots mounting,
GUI versions, server version,...
-Written in golang.
+Written in golang.
+Embedded webGUI for server mode is done in React. KopiaUI comes with electron.
+
+### Ways to use Kopia
+
+* **cli** - Command line.
+ You call the kopia binary passing some commands, it executes stuff, done.
+ Deployment requires extra work - scripts with configs, scheduling.
+* **Kopia Server** - kopia binary runs in the server mode.
+ Runs in the background, with its web server answering at url: `localhost:51515`
+ Through the web GUI the management is easier than using cli.
+ As an addition to normal backup functions, the server mode can also act
+ as a centralized repository for other machines running kopia instances.
+ Deployment requires extra work similar to cli, but actual use is through web GUI.
+* **KopiaUI** - GUI version.
+ KopiaUI comes packaged with electron to provide the feel of a standalone desktop app.
+ Good for simple deployment where average user just wants to backup stuff.
+ Benefits over cli or server is easier setup and management.
+ Drawback is that it runs under one user and only when that user is logged in.
+* **Kopia in Docker** - Kopia Server running in docker
+ Can fulfill two needs:
+ * Backup docker-host stuff to a cloud or a mounted network storage.
+ Managed through webgui instead of cli.
+ * A centralized kopia repository where other machines on the network,
+ that also use kopia, backup their data.
+
+
# Some aspects of Kopia
-There are 3 ways to go about running kopia
-
-* **cli** - Command line.
- You call the binary passing some commands, it executes stuff, done.
- Requires extra work - scripts with configs, scheduling.
-* **KopiaUI** - GUI version of kopia.
- Easier managment, takes uppon itself scheduling.
- Drawback is that it runs under one user and only when logged in.
-* **Kopia Server** - running kopia and its webserver in the background
- Managment through web browser at an url, can run as a docker container.
-
[Official Getting Started Guide](https://kopia.io/docs/getting-started/)
+[Kopia Build Architecture](https://github.com/kopia/kopia/blob/master/BUILD.md)
[Official Features](https://kopia.io/docs/features/)
* Kopia is a single ~35MB binary file.
* Backups are stored in a **repository** that needs to be created first,
- and is always encrypted - requires password.
- * Before any action, Kopia needs to connecto to a repo.
-* **Snapshot**, appart from typical meaning, kopia also uses it to for
- targets(paths) to be backed up.
-* **Policy** is a term used to define behaviour of the backup/repo,
+ and is always encrypted.
+ * Before any action, Kopia needs to connect to a repo.
+* **Snapshots**, apart from typical meaning, kopia also uses the term for
+ targets(paths) that are being backed up.
+* **Policy** is a term used to define behavior of the backup/repo,
like backups retention, what to ignore, logging, scheduling(server/UI),
actions before and after backup,...
* **Policies** are stored inside a repo and can apply at various levels and
- can inherit from each other
- - global policy, the default that comes predefined during repo creation
- - per user policy and per machine policy
- - snapshot level policy, only applying for that one path
-* Maintence is automatic
+ can **inherit** from each other
+ - **Global** policy, the default that comes predefined during repo creation,
+ can be edited like any other.
+ - Per user policy, per machine policy.
+ - Snapshot level policy, only applying for that one path.
+* **Maintenance** is automatic
+* During snapshots Kopia uses local **cache**, location varies depending on the OS.
+ Default max size is 5GB, but it gets swept periodically every few minutes.
+ Useful commands are `kopia cache info` and `kopia cache clear`
+* [Here's](https://kopia.discourse.group/t/trying-to-understand-retention-policies/164/4)
+ how **retention** works under the hood.
* ..
# Kopia on a linux machine
-cli version of kopia will be used, with a script and systemd-timers for scheduling.
+
-### install kopia
+cli version of kopia will be used to periodically backup to a mounted network storage.
+The backup script will be executed using systemd-timers for scheduling.
-for arch linux, kopia is on AUR `yay kopia-bin`
+### Install Kopia
-### the initial steps
+For arch linux, kopia is on AUR `yay kopia-bin`
-General use of sudo so that kopia has access everywhere.
+### The initial steps and general use commands
* **repo creation**
@@ -78,7 +106,7 @@ General use of sudo so that kopia has access everywhere.
`sudo kopia policy get --global`
`sudo kopia policy list`
-`sudo kopia policy set --global --ignore-cache-dirs true --keep-annual 1 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 14`
+`sudo kopia policy set --global --keep-annual 2 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 3`
* **manual run**
@@ -94,11 +122,17 @@ General use of sudo so that kopia has access everywhere.
### The backup script
+In linux, passing multiple paths separated by space seems to work fine.
+So both `/home` and `/etc` are set to be backed up.
+
`/opt/kopia-backup-home-etc.sh`
```bash
#!/bin/bash
-#sudo kopia policy set --global --keep-annual 1 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 14
+# initialize repository
+# sudo kopia repo create filesystem --path /mnt/mirror/KOPIA/docker_host_kopia
+# adjust global policy
+# sudo kopia policy set --global --keep-annual 2 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 3
REPOSITORY_PATH='/mnt/mirror/KOPIA/docker_host_kopia'
BACKUP_THIS='/home /etc'
@@ -108,6 +142,7 @@ kopia repository connect filesystem --path $REPOSITORY_PATH --password $KOPIA_PA
kopia snapshot create $BACKUP_THIS
kopia repository disconnect
```
+
make the script executable
`sudo chmod +x /opt/kopia-backup-home-etc.sh`
@@ -167,7 +202,9 @@ WantedBy=timers.target
* `systemctl status kopia-home-etc.timer`
* `journalctl -u kopia-home-etc.timer` - see history
-# Mounting network storage using systemd
+
+
+Mounting network storage using systemd
* files are placed in `/etc/systemd/system`
* the name of mount and automount files MUST correspond with the path
@@ -205,69 +242,246 @@ Where=/mnt/mirror
WantedBy=multi-user.target
```
-# Remote backup
+
-... some day ...
+---
+---
-# Kopia in Windows
+# Kopia on a Windows machine
+
+
## Kopia Server on Windows
-* download this repo, delete shit, keep `kopia_server_deploy` folder
-* run `DEPLOY.cmd`, it will
+* Download this repo, keep `kopia_server_deploy_win` folder, delete rest.
+* Run `DEPLOY.cmd`, it will:
* Removes powershell scripts restriction.
* Creates folder `C:\Kopia` and kopies files there
- * imports a task schedule that will start Kopia Server at boot
+ * Imports a task schedule that will start `C:\Kopia\kopia_server_start.cmd`
+ Can edit flags used for the server.
+ * Places kopia.url on the current user desktop
* visit in browser `localhost:51515`
* setup repo
* setup what to backup and schedule
-* edit the `kopia_backup_scipt.ps1`, set what to backup and where
-* for the same repo location execute
- `sudo kopia repository create filesystem --path C:\Backup`
-* run the script
-## Kopia cli on Windows
+Kopia should now run on boot and be easy to manage through web GUI.
+Be it creating backup jobs, mounting old snapshots, or just looking around
+if all works as it should.
+
+It is also popular to use nssm to start up and manage kopia as a service.
+
+While Kopia runs now in server mode, the fact that we pass `--insecure` flag
+means it cant serve as a repository for other kopia instances.
+For that look at docker deployment section or in to making changes too.
-While GUI version seems like a way to go.. IMO its not there yet.
-The schedule seems to be dependant on user logging in...
-and general weird feeling of poor GUI quality.
+
+Kopia cli on Windows
-So here goes cli version. As after some hands-on experience with cli version
-the GUI version might click in better too, later.
+
-* download this repo, delete shit, keep `kopia-deploy` folder
-* run `DEPLOY.cmd`, it will
+*Written before I realiezed I could be using kopia server.*
+*Also, at the moment its the only way I know how to make actions work for VSS snapshots.*
+
+While KopiaUI version seems like the way to go I really dislike the idea
+of not having trust in kopia if using it on servers that run with users
+logged out. Since KopiaUI works only if user is logged in.
+
+* Download this repo, keep `kopia_cli_deploy_win` folder, delete rest.
+* Run `DEPLOY.cmd`, it will:
* Removes powershell scripts restriction.
- * Install scoop, sudo, kopia.
+ * kopies kopia.exe in to `C:\Windows\System32`
* Creates folder `C:\Kopia` and kopies there
`kopia_backup_scipt.ps1` and the VSS ps1 before and after files.
* imports a task schedule
-* edit the `kopia_backup_scipt.ps1`, set what to backup and where
-* for the same repo location execute
- `sudo kopia repository create filesystem --path C:\Backup`
-* run the script
+* Open the `kopia_backup_scipt.ps1` and follow the instructions there.
+ Which is just to create repo before running the script.
+ `kopia repo create filesystem --path C:\kopia_repo --password aaa`
+* edit the scheduled task to the prefered time, default is daily at 21:19
+* run scheduled task manually
+* check if it worked
+ * `kopia repo status`
+ * `kopia snap list --all`
-To do the above things manualy:
+### VSS snapshots
-* install kopia using scoop
- * open terminal as admin
- * `Set-ExecutionPolicy RemoteSigned`
- * `iex "& {$(irm get.scoop.sh)} -RunAsAdmin"`
- * `scoop install sudo --global`
- * `sudo scoop install kopia --global`
-* download this repo, extract to `c:\kopia`
-* edit kopia-backup-home-etc.sh as see fit
+To make use of VSS snapshots before running backup execute these two commands,
+replacing `C:\Test` with the snapshot target path.
-* powershell as as administrator
-* --enable-actions
-* in tray, right click on the icon - `Launch At Startup`
- this creates registry entry - *HKCU\Software\Microsoft\Windows\CurrentVersion\Run\KopiaUI*
-*
+* `kopia policy set C:\Test --before-folder-action "powershell -WindowStyle Hidden C:\Kopia\win_vss_before.ps1"`
+* `kopia policy set C:\Test --after-folder-action "powershell -WindowStyle Hidden C:\Kopia\win_vss_after.ps1"`
-kopia policy set --before-folder-action "powershell -WindowStyle Hidden \before.ps1"
-kopia policy set --after-folder-action "powershell -WindowStyle Hidden \after.ps1"
+Note the use of `--enable-actions` in the backup script `kopia_backup_scipt.ps1`,
+which is required for actions to work.
+
+### Kopia install using scoop, machine-wide
+
+Just something to have note of, if decided to switch to heavy scoop use.
+
+* open terminal as admin
+* `Set-ExecutionPolicy Bypass`
+* `iex "& {$(irm get.scoop.sh)} -RunAsAdmin"`
+* `scoop install sudo --global`
+* `sudo scoop install kopia --global`
+
+
+
+---
+---
+
+# Kopia in Docker
+
+
+
+### Files and directory structure
+
+```
+/mnt/
+└── mirror/
+
+/home/
+└── ~/
+ └── docker/
+ └── kopia/
+ ├── kopia_config/
+ ├── kopia_cache/
+ ├── kopia_logs/
+ ├── some_data/
+ ├── kopia_repository/
+ ├── kopia_tmp/
+ ├── .env
+ └── docker-compose.yml
+```
+
+* `/mnt/mirror/...` - a mounted network storage share
+* `kopia_config/` - repository.config and ui-preferences.json
+* `kopia_cache/` - cache
+* `kopia_logs/` - logs
+* `some_data/` - some data to be backed up
+* `kopia_repository/` - repository location
+* `kopia_tmp/` - temp used for snapshots
+* `.env` - a file containing environment variables for docker compose
+* `docker-compose.yml` - a docker compose file, telling docker how to run the containers
+
+### docker-compose
+
+The data to be backed up are mounted in read only mode.
+To be able to mount snapshots, extra privilages are required and fuse access.
+
+```
+services:
+
+ kopia:
+ image: kopia/kopia:latest
+ container_name: kopia
+ hostname: kopia
+ restart: unless-stopped
+ env_file: .env
+ privileged: true
+ cap_add:
+ - SYS_ADMIN
+ security_opt:
+ - apparmor:unconfined
+ devices:
+ - /dev/fuse:/dev/fuse:rwm
+ ports:
+ - "51515:51515"
+ command:
+ - server
+ - start
+ - --tls-generate-cert
+ - --disable-csrf-token-checks
+ - --address=0.0.0.0:51515
+ - --server-username=$USERNAME
+ - --server-password=$KOPIA_PASSWORD
+ volumes:
+ # Mount local folders needed by kopia
+ - ./kopia_config:/app/config
+ - ./kopia_cache:/app/cache
+ - ./kopia_logs:/app/logs
+ # Mount local folders to snapshot
+ - ./some_data:/data:ro
+ # Mount repository location
+ - ./kopia_repository:/repository
+ # Mount path for browsing mounted snaphots
+ - ./kopia_tmp:/tmp:shared
+
+networks:
+ default:
+ name: $DOCKER_MY_NETWORK
+ external: true
+```
+`.env`
+```bash
+DOCKER_MY_NETWORK=caddy_net
+TZ=Europe/Bratislava
+# KOPIA
+USERNAME=admin
+KOPIA_PASSWORD=aaa
+```
+
+**All containers must be on the same network**.
+Which is named in the `.env` file.
+If one does not exist yet: `docker network create caddy_net`
+
+### Reverse proxy
+
+Caddy is used, details
+[here](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/caddy_v2).
+
+To function as a repository server, Kopia must be setup with https,
+which is achieved by using `--tls-generate-cert` flag and removal
+of `--insecure` flag.
+
+So now Kopia sits behind Caddy, but caddy needs to be told the traffic is
+https and to ignore that the certificate is not valid.
+
+`Caddyfile`
+```
+kopia.{$MY_DOMAIN} {
+ reverse_proxy kopia:51515 {
+ transport http {
+ tls
+ tls_insecure_skip_verify
+ }
+ }
+}
+```
+
+### First run
+
+
+
+* visit kopia.example.com
+* create new repository as `Local Directory or NAS`, set path to `/repository`,
+ set password
+
+Now this container can do backups of mounted stuff in to other mounted places
+while managed through webgui.
+To also make it function as a repository server a user account needs to be added.
+The users are stored in the repo.
+
+* exec in to the container
+ `docker container exec -it kopia /bin/bash`
+* add user@machine and set the password
+ `kopia server user add user1@machine1`
+* on another machine on network install kopia server or koppiaUI
+ on first run:
+ * Pick `Kopia Repository Server`
+ * Server address: `https://kopia.example.com:443`
+ * *Trusted server certificate fingerprint (SHA256)*
+ can be left empty, or if you put something there, it gives you error
+ where it tells you fingerprints of the server to pick from.
+ * In advanced option one can override user@machine with the one set
+ when exec-ed in to the docker container.
+ Or exec again there and add another one.
+
+### Troubleshooting
+
+* check kopia docker contaienr logs, I like using [ctop](https://github.com/bcicen/ctop)
+* `nslookup kopia.example.com` check if you are gettin to you server from client
+* Make sure you use port 443 in server address.