You are not logged in.

#1 2020-01-15 14:45:12

clusterF
Member
Registered: 2019-05-07
Posts: 539

Backing Up With Borg

Thought there should be a dedicated thread on how to use borg for a full system backups and other methods of creating backups with borg...

https://borgbackup.readthedocs.io/en/stable/#
https://packages.debian.org/buster/borgbackup


I was using rsync for a long while but decided to get educated on borg, it was well worth it today as i have managed to back up 200GB of files into a 140GB borg repo that i can search with borgfs, but more on that later.

1. make sure you have a large amount of space somewhere, i use backups to an external drive via usb 3.0 cable, there are ways to backup over a network but i dont know how yet.

#### For a Full System Backup ####

This is how i do it below, i post this up for informational purposes only, it pays to RTFM https://borgbackup.readthedocs.io/en/stable/


Note that this backs up everything on /
Note that BACKUPNAME means you have to create that directory (naming scheme of your choice, eg; "bunsenlabs_backups") in the first instance of mkdir -p

#### Borg Backup ####

## Create an encrypted repository

# creates directory for backed up files

mkdir -p /mnt/BACKUPNAME/borg/repos/$HOSTNAME/

# changes to this directory

cd /mnt/BACKUPNAME/borg/repos/$HOSTNAME/  

 
# initializes a borg repository     

borg --verbose init --encryption=repokey

# verify the creation was executed successfully                   
cat README



## Backup of the whole system:

sudo borg --progress --verbose create --comment "Use as root. full system backup." -e /dev -e /run -e /tmp -e /sys -e /proc -e /mnt -e /media -s /mnt/BACKUPNAME/borg/repos/$HOSTNAME/::fullsystembackup /

# This creates a backup of the whole system, excluding locations
# that are re-generated at every boot up, as well as, the `/mnt` location
# as we don't want to back up our backups, especially when you
# have a backup drive containing several hundred GB of data.


## Bashrc function - usage = borg_full "This is my comment."

borg_full(){
borg_comment=$1;
repo=/BACKUPNAME/borg/repos/laptop::fullsystembackup;
sudo borg --progress --verbose \
create --comment "Use as root. $borg_comment" \
-e /dev -e /run -e /tmp -e /sys -e /proc -e /mnt -e /media -e /var/log -s \
/mnt/BACKUPNAME/$repo::\{hostname\}+\{user\}:\{now:%Y-%m-%dT%H:%M\} /;
}


## usefull borg commands

sudo borg list /mnt/BACKUPNAME/borg/repos/$HOSTNAME/
sudo borg info /mnt/BACKUPNAME/borg/repos/$HOSTNAME/
sudo borg delete /mnt/BACKUPNAME/borg/repos/$HOSTNAME/::fullsystembackup

Much of this technique relies on the backup destination being mounted to /mnt so make sure this is so.

Please let me know if ive missed anything or this tutorial should/could be improved.

Last edited by clusterF (2020-01-15 15:05:47)

Offline

#2 2020-01-15 16:53:39

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2015-09-29
Posts: 767

Re: Backing Up With Borg

^Interesting, thanks for sharing.

This 'borg' actually looks like a mix of rsync and git: it will backup ('copy') data as good as rsync, while simultaneously keep old files (as git). And all this with reasonable syntax.

Definitely worth a try, especially looks convenient for incremental(!) backup of /home/$HOME directory ...


Postpone all your duties; if you die, you won't have to do them ..

Offline

#3 2020-01-16 08:18:34

ohnonot
...again
Registered: 2015-09-29
Posts: 4,877
Website

Re: Backing Up With Borg

I started using borg when I was looking for a backup solution that does deduplication - at that time I wasn't aware that rsync can do that too, just calls it differently.
But I am satisfied and see no reason to switch now. I guess borg has a higher convenience factor. I do weekly backups (almost full system) that add almost nothing to my backup disk's usage, and have restored one or two files succesfully (I don't use a trash bin). Never done a full restore though.
https://dt.iki.fi/borgbackup


BL quote proposals to this thread please.
how to ask smart questions | my repos / my repos | my blog
---
Thank you for posting direct image links!

Offline

#4 2020-01-16 11:31:41

clusterF
Member
Registered: 2019-05-07
Posts: 539

Re: Backing Up With Borg

@ imbecil, thanks im quite happy with it, been playing around with it for a few days now.

@ ohonot, was not aware that rsync can do deduplication, ill have to investigate, usually all i do is mount my encrypted usb drive and rsync as per arch wiki https://wiki.archlinux.org/index.php/Rs … tem_backup

Figured out mounting the full system backup using borg mount so i can search through the backup and copy files if need be.Borgfs is a bit complex as you can have multiple borgfs option like exclusion and archive filters , sort of like a reverse backup but just mounting using fuse, but if you just want to mount the whole backup i did the following. I did need python-llfuse for this.

create a mountpoint mine was <borg-mount>, has to be read write.

# borg mount /mnt/BACKUPNAME/borg/repos/$HOSTNAME/::fullsystembackup::laptop+root:2020-01-16T20:49 <borg-mount>

Last edited by clusterF (2020-01-16 11:34:23)

Offline

#5 2020-01-16 13:42:31

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2015-09-29
Posts: 767

Re: Backing Up With Borg

Thanks clusterF and ohnononono.

clusterF wrote:

@ ohonot, was not aware that rsync can do deduplication

Deduplication ... if it means NOT syncing identical files, then rsync does this ... i.e. it will not waste time to copy src-->dest for files which are identical. This really reduces backup time.

What I don't know: can rsync not only sync new files, but also keep previous version of those new files (a reminscence of git)? I think not, but I would like to know if there is such an option ...


Postpone all your duties; if you die, you won't have to do them ..

Offline

#6 2020-01-16 13:59:07

clusterF
Member
Registered: 2019-05-07
Posts: 539

Re: Backing Up With Borg

Not sure imbecil, i cant find any documentation that corresponds to what borg is capable of.

Ive made 5 incremental backups so far, i could automate this as long as my usb is connected and mounted but i like to manually do it.

sudo borg list /mnt/usb/backups/borg/repos/laptop/

firstbackup                          Wed, 2020-01-15 10:17:25 [xxx]
fullsystembackup::laptop+root:2020-01-15T11:57 Wed, 2020-01-15 11:57:13 [xxx]
fullsystembackup::laptop+root:2020-01-15T12:03 Wed, 2020-01-15 12:03:09 [xxx]
fullsystembackup::laptop+root:2020-01-16T01:20 Thu, 2020-01-16 01:20:49 [xxx]
fullsystembackup::laptop+root:2020-01-16T20:49 Thu, 2020-01-16 20:49:33 [xxx]

sudo borg info /mnt/usb/backups/borg/repos/laptop/

------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:               37.46 GB             12.78 GB              3.85 GB

                       Unique chunks         Total chunks
Chunk index:                  161668               628120

Last edited by clusterF (2020-01-16 14:00:28)

Offline

#7 2020-01-16 16:02:27

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2015-09-29
Posts: 767

Re: Backing Up With Borg

clusterF wrote:

Not sure imbecil, i cant find any documentation that corresponds to what borg is capable of.

Oh ... it was actually question for ohnononono

I'm also not aware of rsync keeping old file version, and I have read documentation, too.


Postpone all your duties; if you die, you won't have to do them ..

Offline

#8 2020-01-16 20:34:42

ohnonot
...again
Registered: 2015-09-29
Posts: 4,877
Website

Re: Backing Up With Borg

iMBeCil wrote:

What I don't know: can rsync not only sync new files, but also keep previous version of those new files (a reminscence of git)? I think not, but I would like to know if there is such an option ...

Borg definitely does that.
rsync itself - I don't know, sorry.
Some of the apps based on rsync probably can - but then, borg is also based on (lib)rsync.


BL quote proposals to this thread please.
how to ask smart questions | my repos / my repos | my blog
---
Thank you for posting direct image links!

Offline

#9 2020-01-16 20:36:51

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2015-09-29
Posts: 767

Re: Backing Up With Borg

^Confirms what I thought. Thanks.


Postpone all your duties; if you die, you won't have to do them ..

Offline

#10 2020-01-17 05:53:46

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 7,299
Website

Re: Backing Up With Borg

iMBeCil wrote:

What I don't know: can rsync not only sync new files, but also keep previous version of those new files (a reminscence of git)?

Would the --backup option not help a bit?

        -b, --backup            make backups (see --suffix & --backup-dir)
        --backup-dir=DIR        make backups into hierarchy based in DIR
        --suffix=SUFFIX         backup suffix (default ~ w/o --backup-dir)

Make SUFFIX date-dependent and every backed-up file would be different. Perhaps nowhere near as efficient as git, though.


...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Introduction to the Bunsenlabs Lithium Desktop

Offline

#11 2020-01-17 08:56:56

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2015-09-29
Posts: 767

Re: Backing Up With Borg

^Indeed, after reading manual, this can be used for incremental backup, with old files still present. But it would need a bit scripting to automate this ... borg seems to do it kind of directly/cleanly, user doesn't have to think about the options too much.

Thanks for the 'RTFM' johnraff smile (I should have done it myself ...)


Postpone all your duties; if you die, you won't have to do them ..

Offline

#12 2020-01-17 13:38:03

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,934

Re: Backing Up With Borg

ohnonot wrote:
iMBeCil wrote:

What I don't know: can rsync not only sync new files, but also keep previous version of those new files (a reminscence of git)? I think not, but I would like to know if there is such an option ...

Borg definitely does that.
rsync itself - I don't know, sorry.
Some of the apps based on rsync probably can - but then, borg is also based on (lib)rsync.

https://github.com/rdiff-backup/rdiff-backup : rdiff-backup uses the rsync algorithms for transfer but also has an incremental backup future - you can roll back increments and restore an old version of the file.

That being said, I stick to rsync for my core data backup; I somehow trust it more to do the right thing than borg or rdiff-backup.

I use rmlint  for deduplication of data in my file collections. It is smart enough to distinguish hardlinks (USEFUL when you want to have the really same file both in your 'taxes' and 'bank statements' folders, for organization) from actual duplicates (different inodes, same data, it is SIMPLE and PREDICTABLE and has REVIEW functionality built in, so I like using it more than 100% integrated tools that do everything.

I think it's important to keep your tools in important areas such as backups of personal data as simple as possible and the FORMATS easily reversable. rdiff-clone and borg use special metadata; rsync + rmlint just use the file system (in a simple way).


Per aspera ad astra.

Offline

#13 2020-01-18 04:13:42

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 7,299
Website

Re: Backing Up With Borg

twoion wrote:

I think it's important to keep your tools in important areas such as backups of personal data as simple as possible

+1


...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Introduction to the Bunsenlabs Lithium Desktop

Offline

#14 2020-01-18 13:27:56

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2015-09-29
Posts: 767

Re: Backing Up With Borg

^+1 fully agree.

Hadn't stumbled on rmlint before, thanks @twoion ... seems interesting, now I'm reading its documentation instead of doing my paid work ops


Postpone all your duties; if you die, you won't have to do them ..

Offline

#15 2020-01-18 16:39:40

clusterF
Member
Registered: 2019-05-07
Posts: 539

Re: Backing Up With Borg

twoion wrote:
ohnonot wrote:
iMBeCil wrote:

What I don't know: can rsync not only sync new files, but also keep previous version of those new files (a reminscence of git)? I think not, but I would like to know if there is such an option ...

Borg definitely does that.
rsync itself - I don't know, sorry.
Some of the apps based on rsync probably can - but then, borg is also based on (lib)rsync.

https://github.com/rdiff-backup/rdiff-backup : rdiff-backup uses the rsync algorithms for transfer but also has an incremental backup future - you can roll back increments and restore an old version of the file.

That being said, I stick to rsync for my core data backup; I somehow trust it more to do the right thing than borg or rdiff-backup.

I use rmlint  for deduplication of data in my file collections. It is smart enough to distinguish hardlinks (USEFUL when you want to have the really same file both in your 'taxes' and 'bank statements' folders, for organization) from actual duplicates (different inodes, same data, it is SIMPLE and PREDICTABLE and has REVIEW functionality built in, so I like using it more than 100% integrated tools that do everything.

I think it's important to keep your tools in important areas such as backups of personal data as simple as possible and the FORMATS easily reversable. rdiff-clone and borg use special metadata; rsync + rmlint just use the file system (in a simple way).

Could you perhaps share some part of your backup routine using rmlint and rsync ?

Offline

#16 2020-01-19 13:14:49

Zeno
Member
Registered: 2020-01-16
Posts: 27

Re: Backing Up With Borg

I only want to take snap shot of only Installed BL & all its settings and Configurations. Thats it. Like Timeshift does. Can I use Timeshift ???

Thanks

Offline

#17 2020-01-19 13:20:01

clusterF
Member
Registered: 2019-05-07
Posts: 539

Re: Backing Up With Borg

Zeno wrote:

I only want to take snap shot of only Installed BL & all its settings and Configurations. Thats it. Like Timeshift does. Can I use Timeshift ???

Thanks

https://github.com/teejee2008/timeshift

Timeshift is similar to applications like rsnapshot, BackInTime and TimeVault but with different goals. It is designed to protect only system files and settings. User files such as documents, pictures and music are excluded. This ensures that your files remains unchanged when you restore your system to an earlier date. If you need a tool to backup your documents and files please take a look at the excellent BackInTime application which is more configurable and provides options for saving user files.

Offline

#18 2020-01-19 13:28:17

Zeno
Member
Registered: 2020-01-16
Posts: 27

Re: Backing Up With Borg

Thanks. I just want System & Settings etc. So i would prefer Timeshift and for other Data backup i use Grsync.

Is my choice ok?? Thanks

Offline

#19 2020-01-19 13:36:23

clusterF
Member
Registered: 2019-05-07
Posts: 539

Re: Backing Up With Borg

Zeno wrote:

Thanks. I just want System & Settings etc. So i would prefer Timeshift and for other Data backup i use Grsync.

Is my choice ok?? Thanks

You do what works for you Zeno smile

Offline

#20 2020-01-19 13:38:32

Zeno
Member
Registered: 2020-01-16
Posts: 27

Re: Backing Up With Borg

Ok Thanks

Offline

#21 2020-01-20 07:29:35

ohnonot
...again
Registered: 2015-09-29
Posts: 4,877
Website

Re: Backing Up With Borg

Zeno wrote:

I only want to take snap shot of only Installed BL & all its settings and Configurations. Thats it. Like Timeshift does. Can I use Timeshift ???

That's "Only" the whole system, no? Your complete install? Not just some config files under your $HOME?
You have to be clear what you want.
https://github.com/teejee2008/timeshift
Read the Readme.


BL quote proposals to this thread please.
how to ask smart questions | my repos / my repos | my blog
---
Thank you for posting direct image links!

Offline

#22 2020-05-29 16:04:23

yoda
Member
From: Montreal
Registered: 2018-12-28
Posts: 150
Website

Re: Backing Up With Borg

what is the easiest Backup software for BL,  with a GUI LOL ?

Is there is no easy answer to that, i'll read about borg and rsync and decide wich I want to use

I simply want to backup my HOME FOLDER Daily.  If I do it using a file Manager ( thunar, nemo etc) too many errors get in the way and I can't finalise it.

I don't care about backuping my whole system, but my application config ( .config ) is usefull..

So my Home Folder, without headaches, is what I want to backup.


According to Buddhism, we all met several times... This time, it's on this forum :-) May we meet again

Offline

#23 2020-05-29 16:28:26

yoda
Member
From: Montreal
Registered: 2018-12-28
Posts: 150
Website

Re: Backing Up With Borg

for now I will be testing this

https://askubuntu.com/questions/545655/ … ss-folders

The exclude list may only contain filenames, foldernames and lines starting with #. A comment behind the foldername is not allowed. I created a Git repository with all known files and folders that are superfluous:

Download this ignorelist to /var/tmp/ignorelist

wget https://raw.githubusercontent.com/rubo7 … cludes.txt -O /var/tmp/ignorelist
Then start the rsync with

rsync -aP --exclude-from=/var/tmp/ignorelist /home/$USER/ /media/$USER/linuxbackup/home/
Note:
In the ignorelist there is a section at the start with folders, that are directories, probably not worth a backup.




TIMESHIFT seems to be very simple and efficient for noobies ?

Last edited by yoda (2020-05-29 18:50:13)


According to Buddhism, we all met several times... This time, it's on this forum :-) May we meet again

Offline

#24 2020-05-30 06:06:48

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 7,299
Website

Re: Backing Up With Borg

^Hi Yoda, thanks for publishing that exclude list! It looks very complete, and I might use it to add to my own excludes. Though, instead of just downloading it to a temporary folder you might want to have a look and maybe edit it a bit before using it. There are some commented-out lines you might consider using (ie un-commenting).

I've been using rsync for backups some time now, and like it. I usually back up /etc too (it's not too huge and occasionally there's something useful there).


...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Introduction to the Bunsenlabs Lithium Desktop

Offline

#25 2020-05-30 11:37:02

yoda
Member
From: Montreal
Registered: 2018-12-28
Posts: 150
Website

Re: Backing Up With Borg

Glad it can be of any use.

I just installed my new Samsung EVO 860 SSD drive (1 TB), OMG it's very fast ( a lot faster than my Crucial 500 GB drive with similar specs).

I will now I backup 5 days / week + 2 times / week ..Then I will be able to go back 2 week before. 

I have been lazy yesterday and used TimeShift.. (using an external western digital 4 TB USB Drive)...

Considering may be using syncthing.net to backup on my son's PC (OffSite)... I am going away from the cloud :-) (installed /e/ on my 1st android phone, I was IOS for always).

What advantages do you see of using rsync through the CLI beside the fun of tweaking it LOL.    (TimeShift make it easy for me to use rsync)

Last edited by yoda (2020-05-30 12:33:07)


According to Buddhism, we all met several times... This time, it's on this forum :-) May we meet again

Offline

Board footer

Powered by FluxBB