TLUG Talk: TLE-BU, 2004
Note: This program is no longer being developed as the parent company, TLE, is no longer in busines. This is here mainly for historical reasons. This was the first of two talks I gave on TLE-BU. This one first introduced TLE-BU as an open source program.
Warning: TLE-BU is still on Source Forge for download, but it it not recommended for use. There was a critical bug with non-ASCII file names that caused unreliable backups. TLE folded before I got to fix it, unfortunately.
Future: The AN::Tools module suite directly evolved from the initial failings of TLE-BU. I decided to write it as a portable module suite that would provide all of the core functions TLE-BU would have needed but in a much more robust package. Once AN::Tools is finished, my first personal project will be to write AN!Backup, a completely new "under the hood" version of TLE-BU. I plan for it to have the same look and feel of TLE-BU when the time eventually comes.
Open Source, Web-Based Backup Software
Lead Technician, TLE
Oct. 12, 2004
I decided to write a new backup application from the ground up when I was unable to find an existing backup application that provided certain features a client of ours had requested. Do not get me wrong, there are several very mature and very well written open source backup programs! The main feature that we found lacking though was a friendly front end. All of the viable existing backup software I found was console-based which was not helpful for our needs. The existing software for most part also did not provide the features I wanted like intelligent spanning of destinations, an archival component, etc.
There is a lot of information I hope to cover in the next hour. I ask you all to refrain from making comments or asking questions until I am finished. If you think I have made a mistake please feel free to correct me if it is significant. If we can get through the presentation within the one hour time period we will have at least half an hour for a question and answer (and corrections).
Finally before we begin; TLE-BU is in alpha-stage at this moment. It is functioning and can be installed and played with however it is certainly not ready to be used on any production machine nor should it be used yet on any machine you value. There is still a lot to be done and that is partly why I asked to present this evening. This is my first open-source program and, though I am biased, I think it has wonderful potential. With any luck one or more of you will agree by the end of today and would like to join the development efforts and help refine TLE-BU.
Perhaps one of the most important tasks that a computer user of any level needs to do is backup their important data. If you run, or work in an IT role for, a company this is infinitely more important. None the less, almost everyone here in this room has either themselves neglected to be religious about their backup and ended up losing critical data or they know someone who has. I personally have both lost and seen others lose important data so I can attest to the horrible feeling that goes along with data loss. It was because of this that I took a very strong interest in storage technology and now why I decided to write a new backup application.
Regular backups almost never happen if a person tries to manually copy over important data to removable media like a CD/DVD or external drive. Human nature almost always gets in the way. People get busy, people put it off "to tomorrow" and people forget. Eventually, and usually pretty quickly, the regular backups simply stop happening. I know this is true for me and I have seen it be true with others as well.
I wanted a set of features and when I couldn't find any existing software that would work for me I, like any true geek, decided to write my own. This meant I had a few decisions to make:
What did I want to backup to?
- CD/DVDs; inexpensive but limited storage space and are a pain if what I want to backup needs to span two or more disks it means I have to watch the backup job.
- Tapes; drives are expensive and only the highest-priced Ultrium drives approached the size of modern system's hard drive. Tapes are relatively unreliable and in the case of the drive being damaged recovering the data would require finding another compatible drive to read the data.
- Hard drives; Inexpensive, fast and reliable. The drives can be connected to almost any system.
For a long time hard drives had a distinct disadvantage. If the server was destroyed by an electrical spike or fire the backup drive would usually also fail or be destroyed. Hard drive carriers could be installed that would allow you to easily unplug and remove the drive and insert another however unless you could afford high-end SCA-SCSI drives this meant first powering off your computer.
USB and Firewire created a way to connect IDE drives externally and allowed for "hot-swapping" the drives. The restriction then was that USB1.1 was slow (1.5MB/sec. max) and Firewire (50MB/sec. max) was, at first, quite expensive. Today though we have access to USB2 drives that are much faster (60MB/sec. max), Firewire is much more affordable and firewire2 (100MB/sec. max) is available for those willing to spend the extra money. These advancements have made externally connected, inexpensive and hot-swappable IDE drives viable for use as backup media.
Once I settled on USB2-connected drives I then had to find a backup application that supported this and ran under Linux. I wanted to find an application that met certain standards. I knew that I wanted a backup program for myself but I also wanted a backup program that I could install on the machines of less-technical people who I have helped migrate to Linux.
My requirements were
- Had to support external hard drives via USB2.
- Had to support scheduled backups which run automatically.
- Had to let me define in detail what I wanted and did not want to backup.
- Had to be easy to use for people who were not technically inclined once installed.
There a number of wonderful open-source programs out there that meet most of these requirements but none that I could find that provided the last item.
Finding a backup program that is easy to use was critical to me. I have been actively moving home users and small businesses over to Linux for some time now. The greater Linux community has done amazing work in the last few years making Linux easier to use and thus letting me be successful in these migrations. Most people here will agree that distros like Fedora Core and Mandrake have made using Linux "easy" and realiatic for John and Jane Q. Public and that is what has been scarring Microsoft badly (and "That's a Good Thing").
I ran into a particular client where not having an easy to use backup program became the deal-breaker for a Linux migration. This client ran a small office with two servers (Netware 4.11 and MS Windows 2000 Server) with about 6 employee workstations. The only thing that kept him on MS and Netware instead of Linux/Samba was the easy selection of files to backup and recover that he had available to him using ArcServe. This person was not a technician so he declined to use several of the Linux command-line based backup programs I presented to him.
This lack of an application that was easy to use was the driving force behind TLE-BU. The goal of TLE-BU is to both lighten the load of sys-admins and to provide one more option to those of us working to migrate less-technical people and businesses.
With the great help of others I wrote TLE-BU to address the requirements I just mentioned. TLE-BU has reached a point now where most of the key components are fully functional. You can get the latest copy of the program from Sourceforge.net under to project name 'tle-bu'. The program can be used to backup software already but I have intentionally not implemented to actual scheduling component and I won't until just before releasing v0.2. I have done this because I do not want anyone yet relying on TLE-BU until the last functionality is implemented and the initial testing is completed. You can manually run the backup jobs you create and I will show you how to do that a little later on.
Overview, What TLE-BU is
- It is first and foremost a backup program designed to use external USB2/Firewire hard drives.
- It is an archiving program making even large catalogs of external drives easily manageable.
- It supports automated backing up of multiple source partitions to multiple destination partitions.
- It is easy to use via a web page interface that can be accessed from any workstation.
- It requires only a web server and a database (ie: Apache/PostgresSQL) plus a few systems apps.
- It is easy to administer with easy to understand errors, warnings and logs entries.
- It has a built-in log viewer.
- It uses a simple plain-text configuration file with ample documentation.
- All aspects of the program are well documented and easy to follow.
- It is easy to secure using the tried and tested security of the web server and, if you wish, iptables.
- It is fast using 'rsync' to only copy files that have been modified when an past backup exists.
- Data is easy to recover; TLE-BU is NOT required to recover data. Simply connect and mount the partition on any compatible system and copy the files you wish to recover.
How to use TLE-BU
Now that I have covered the why and what, I will cover the how.
I will not cover in the presentation how to install TLE-BU but I did want to include an overview here in the hand-out to help you if you decide to try out TLE-BU for yourself.
At the moment there is no installer package for TLE-BU. To install it you simply need to extract the source tarball which will show two directories; 'cgi-bin' and 'non-cgi'. I will assume for now that you know how to configure Apache and PostgresSQL. If you do not, I will explain that step after the presentation is over. There will also be detailed instruction in the attached 'readme.txt' file in the 'cgi-bin' directory as v0.2 gets closer.
- First you need to create an account on your system that TLE-BU will run as. For me, I simply decided to use my own personal account because I will be the one managing it. You will need to tell the web server you decide to use to run as this user. If you have another web server already on your system that is used by another project you may want to install a parallel copy or use a separate dedicated web server like 'thttpd'. There are plans to integrate a dedicated web server (like 'CUPS' does) before v0.4 but that is a little ways off.
- Copy the files from the 'cgi-bin' directory either into the directory (or a subdirectory) you specified as the 'cgi-bin alias'. If you will use the web server for anything else you will find it helpful to create a subdirectory just for TLE-BU. Next, copy the files under the 'non-cgi' into the directory (or subdirectory) that you set as the 'document root' in your web server. Like the 'cgi-bin alias', if you will use the server for other projects you may want to place these files in a sub-directory.
- Edited the web server's configuration to have it run as the user you specified in the first step. You may need to adjust other settings depending on the web server you use but that is all you need to do so far as TLE-BU is concerned.
- All you need to do with Postgres is to create the user you chose in step one as a 'psql' user and then create a database under this user. Remember the name that you use (ie: I call my database 'tle-bu'). You will need to tell TLE-BU in step 6b what database name it is to use.
- As the root user, edit your '/etc/sudoers' file and add the line to the end:
<username> ALL = NOPASSWD: ALLBe sure to use the same user name that you chose above.
- Edit the 'tle-bu.conf' file that is in the 'cgi-bin' directory. The options in the configuration file are documented in detail so I will only talk about the key options now.
- Set the 'user' and 'group' to the user and group you told your web server and SQL DB to use above.
- Set 'db_name' to the name of the database you created in the fourth step.
- Set 'installdir' to the directory where you copied the 'cgi-bin' files. Remember to use the full directory if you used a subdirectory under your 'cgi-bin alias'!
- Set 'url_cgi' to the URL where the 'cgi-bin' files can be found. For example, if you placed the 'cgi-bin' files in the directory 'tle-bu' under the 'cgi-bin alias' directory you would enter 'http://localhost/cgi-bin/tle-bu/'.
- Set 'url_non_cgi' to the URL where the 'non-cgi' files can be found (similar to above).
- The 'log_append' setting is set to default '0' which tells TLE-BU to re-start the log file each time you enter the 'Part. Conf.' menu which keeps it small. If you want to keep the logs change this to '1' which will only ever append to the log file. There will be a more logical cycling of the log file (similar to how the system cycles the 'message' log file) by v0.4.
- Set the 'mnt_dir' to the directory where you want partitions that are handled by TLE-BU mounted at. Do not use the same directory where you place other mounts! It is a good idea to create a directory specially for TLE-BU. For example, I normally mount partitions manually under '/mnt/' so I told TLE-BU to use '/mnt/tle-bu/'.
- If TLE-BU complains that it can't find a certain application (like 'rsync' or 'fdisk') you may want to tell TLE-BU to look for executables in other directories. By default it searches within the directories in you '$PATH' variable (type '$ echo $PATH' from your shell to see what these are). If you specify extra directories in 'scanbin' they will be checked as well.
- Set the 'exclude_list' to include directories that you never what backed up. Typically directories that you will want to exclude are virtual file systems like '/dev' and '/proc' and temporary directories like '/tmp'.
- Lastly you may want to change 'parallel_streams' depending on the speed of your machine. Normally (when set to '1') TLE-BU will create a "stream" for each set of files going from a given source to a given destination partition and copy each stream at the same time. For example, if I have two source partitions and 1 destination partition two streams will be created; one for the files coming from the first source to the only destination and the other for the files going from the second source to the only destination. If your system gets too bogged down when this happens you can change this to '0' which will tell TLE-BU to only run one stream at a time. This will make the backup job take longer but it will be far kinder to your system load.
- You may find it helpful to change other settings but these are the most important ones.
- The last step is to load the help file. The contents of help are stored in the database and are exported to a file. There is a simple interface for loading the help file that is shipped with TLE-BU (this will be automated by v0.2). Go to the "[ Help ]" menu, then click on "[ Edit ]", then click on "[ Restore ]" and then lastly click on "[ Reload ]" beside the help file. Voila! Help is now reloaded.
At this point you should be able to simply browse to the URL you specified above as the 'url_non_cgi' path (ie: 'http://localhost/tle-bu/' and see TLE-BU running!
Preparing Partitions, Directories and Files for Backup
The first menu you will use is '[ Part. Conf. ]' (partition configuration). This is the menu where you will see each partition on your system and where you will manage those partitions. If you have not yet connected your USB2/Firewire drive(s) you can do so at any time and simply reload the '[ Part. Conf. ]' menu to see and use them.
fig. 1.0, The 'Part. Conf.' Menu:
To start you need to select which partitions you want to save data from and assign them as a "source" partitions. Next you need to connect each drive/partitions that you want to save data to and assign them as a "destination" partition.
It is very helpful if you add a comment (or catalog number) of your choice to each source and destination partition. You can do this from here as well by clicking on "[ Add/Edit Part. Comment ]" and entering any plain-text string. This will come in very handy when you search for files because the location of each search result is indicated by this comment.
At this point you have already done enough to backup your system. If you do no more each partition you selected will be fully backup when you run a backup job that calls a given source. If you want to refine which directories and files you wish to backup simply click on the "[ Add/Edit Backup List ]" button which is visible for each source. This will start a scan of what is on the source you chose. This scan can take a while depending on the speed of your computer and the number of files (not the size) on the source partition. This is because TLE-BU records all of the file and directory names and information in it's database. This is how TLE-BU can offer it's archival functions and allow you to search within off-line partitions.
fig. 1.1, The Partition Update Screen:
As you can see above the program displays a status block after every 1,000 files and directories are processed. This acts as a type of progress bad and lets you know that the program is still working. When the update is finished TLE-BU will fully vacuum the database because the update can create a lot of database activity. If you have done a lot of other work in TLE-BU before you updated the partition the vacuum could take a bit of time. Please be patient during this step. When the vacuum is finished a final performance display block will show you how many files and directories where updated, how long it took and how long the full vacuum took. This can be useful when you first setup TLE-BU if you are playing with your database configuration and want a benchmark of how the changes effected your performance.
When the update is finished the next step is to click on the "[ Select Files For Backup ]" button. This will take you to a file browser that looks and functions like many other file browsers and should be very familiar to you.
fig. 1.2, The File Browser:
From a user interface point of view the File Browser is probably the most complex part of TLE-BU. Along the top is a status bar that shows you if the partition you are viewing is a source ("Backup") or a destination ("Recover"). The next item is the partition's current mount point. This may be helpful to you if you want to look at the contents of a file before you decide to back it up or not. The third item is the partition's block path which may help in some cases where you are using a separate disk management software. The last item is the partition's "UUID" number which is what TLE-BU uses internally to identify and track partitions.
The Directory Tree starts with a summary showing how many files are currently visible and how many directories there are total on this partition. Below that each directory will be displayed with a check box that you will use to tell TLE-BU if you want to backup or not the directory, it's sub-directories and files. The second item is an icon that shows if there are subdirectories below the directory or not and if so whether it has been expanded yet or not. A hyphen ('-') indicates that there are no subdirectories, a right-pointing arrow indicates that there are hidden subdirectories and a downward-pointing icon which indicates that there are subdirectories and that they are currently visible. The last item is the name of the directory itself. When you click on the directory it's contents will be displayed in the file list to the right.
The File List starts with a header showing you how many files are in the current directory, which directory is being shown and column headers indicating each file's size, modification date and the "More Info." icon. To the left of each file name is a check box which you can use to indicate whether you want each individual file backed up or not. The rest of each line shows the file name, the file's human-readable size, the date and time the file was last copied or edited and finally an icon that you can place you mouse over which will show extended information on the file. That is the gray bar you can see in the screen capture above. This extra information includes the file size in bytes, the user and group that owns the file and the file's permissions.
You may have noticed that unlike many file browsers of this type, the TLE-BU file browser does not list directories in the file list. This may be added later but at this time I could not see a good reason to add the complexity that would be needed to have the directories in the file list and have them behave like other similar file browsers. I would love to hear feedback from users on how they feel about this. If there is enough demand for this feature I will add it.
A note on selecting directories and files; When you select or unselect a directory for backup the setting you changed the directory to will propagate to all of it's subdirectories and files. Because of this it is most effecting to make you selection starting at top level directories and refining down until you reach the files in each directory. This will save you the possible headache and making all of you selections only to have them lost when you change the backup state of a parent directory.
You now know how to refine what is backed up from each partition down to the individual file. You will notice the each destination partition has the button labeled "[ View/Recover Data ]". If you select this you will go through the exact same steps as you did when you selected which files to backup. For now making a selection on a destination will not accomplish anything. Later (by v0.4) this will allow you to select files to be recovered back to your system (either to their original location or to an alternate location) easily. For now to recover data simply bring the destination partition on-line, mount it (which you can do from the "Part. Conf." menu) and use your system's file browser to copy any files you wish to recover.
Creating and Scheduling Backup Jobs
After you have assigned your source and destination partitions and refined what you want to backup the next step is to create a "Backup Job". You can create multiple backup jobs and each can use some or all of the source and destination partitions. Each job can also be scheduled to run independently of one another. Once a job is created you will also be able to manually run a backup job by simply clicking "[ Run Now ]" from the main scheduling menu.
fig. 2.0, The Main Scheduling Menu:
When you first select "[ Schedule ]" there are no backup jobs yet and you will need to create one. Click on the "[ New ]" button and the first step is to give your backup job a name. This can be pretty much anything you want but by default the name is "TLE-BU Backup Job Created on <date>, <time>". You can leave the name, make a small edit or change it entirely. This field is there to let you easily identify each backup job.
After you give the job a name you will be asked to select which source partitions you want to backup when the backup job is run. There are two other options at this point as well. If you select "Automatically add any new source partitions to this backup job." any source partitions that you add to you system later will automatically be added to this backup job. This option will NOT add existing sources. Be sure to select the existing sources that you want to use manually. The second is "Use strict." which, when chosen, will cause the backup job to act in an "all or nothing" manner. If you select "Use Strict." and one of the source partitions that are part of this backup job are not on-line when the job is run, the backup job will abort. This will also cause the backup job to fail and not be updated if one of the sources you selected is removed from the system or reassigned.
The next step after selecting sources for each job is to select which destinations you want to use. This menu looks identical to the previous source partition selection menu and has the same two extra options.
The last step is to tell TLE-BU when you want each backup job to run. Anyone familiar with 'crontab' files will recognize how this works. You enter the hour that you want the job to run (in 24h format), the minute of that hour, the days of the week, the days of the months and the months of the year. For example, if you enter [ "23":"30", "1-3,5", "*", "*" ] the backup job will run at 11:30pm on Monday through Wednesday and Friday any of the month, any month of the year. That is it, you have a backup job now!
If you already have created one of more jobs when you go into the "[ Schedule ]" menu you will see the option "[ Edit ]" is available and the existing backup jobs will be shown below. When you click on the "[ Edit ]" button instead of being asked to enter a name for a backup job you are instead asked to "[ Select ]" from one of the pre-existing jobs to edit. After this step editing a backup job is the same as creating a new job. You will also have the option to delete a given backup job by clicking on the "[ Delete ]" button the the right of each backup job entry.
The Search Engine
One of the unique and powerful features of TLE-BU is it's built-in search engine.
fig. 3.0, The Search Engine Menu:
Every time a partition is used in a backup job or every time you click on "[ Add/Edit Backup List ]" or "[ View/Recover Data ]" the files and directories on each partition is scanned and recorded in TLE-BU's database. Because of this step you can search for a directory or file on a partition that isn't even connected to your machine. In this way you can locate the file you want to recover and identify which physical drive contains the file. This way you can have even a very large number of destination partitions and still easily be able to manage them. If you're main system is lost for some reason you simply need to bring each drive back on-line once, assign it, create a comment for each, scan it and you're catalog is rebuilt.
The search engine does not need any setup to use. You can simply enter the "[ Search ]" menu and enter a search query and the string you entered will be searched for on all destination partitions (not sources) and the results will be sorted in ascending order by the result's name. If you wish to refine the search though you merely need to click on the "[ Set Preferences ]" button on the upper-right and you will be able to refine in detail how your search is performed and how results are displayed.
You can also create multiple search profiles so that you can quickly perform a search using entirely different criteria by simply selecting an option from a drop-down menu. To help keep them separate you can even assign a short name to each profile.
fig. 3.1, The "Set Preferences" Menu:
In the "Set Preferences" menu you will see four sections; "Set variables for this search profile", "Where do you want to search for files", "Restrict your search to" and "Set the order in which results are sorted by".
In the "Set variables for this search profile" section you can set how many results are displayed per page, you can name the search profile, set the current profile as the default if it isn't already, switch to edit the preferences of another existing search profile and enter the "[ Manage Profiles ]" menu where you can add, copy and delete search profiles.
In the "Where do you want to search for files" section you can decide if you want a search to look in source partitions, destination partitions or both. In the "Restrict your search to" section you can restrict the search to files and directories backed up by TLE-BU only, to directories only, to files only, to some combination of these or leave them all blank and let TLE-BU search without restrictions.
Before you use the fourth section save any changes you have made. When you make a change to this section the changes take effect immediately and the values above are not passed along. The last section, "Set the order in which results are sorted by" is where you can specify how you want your search results sorted. You can add as many items as you would like and each can be sorted in ascending or descending order separate from the rest. To add a new item to sort by simply click on the "[ Add New Sort Item ]" button and then select an item not already used.
When you click on the "[ Manage Profiles ]" button at the top of the preferences menu you will see a new menu where you are presented with a few additional options. You can click on "[ New Profile ]" which will create a new search profile based on the TLE-BU defaults. You can change which profile is used by default (when you simply click on "[ Search ]" and enter a query). You can click on "[ Copy ]" to make a new search profile based on an existing one (useful if you have a heavily modified search profile that you want to make a slightly modified version of). Lastly you can click on "[ Delete ]" which will let you delete a search profile you no longer need or use.
View Logs and Help Menus
The two final items in TLE-BU are the built-in log viewer and the Help menu. If you click on '[ View Logs ]" you will be able to quickly and easily see TLE-BU's log file. If you find yourself using this option you may find it helpful to set 'log_append' to '1' in the 'tle-bu.conf' file so that the log file isn't restarted each time you use the "[ Part. Conf. ]" menu.
The help menu is still under development at this point so the presentation isn't very attractive just yet. The help menu is already very thorough and should prove quite helpful to those who wish to try out TLE-BU. There is already a mechanism that will let people translate the TLE-BU help menu into other languages. The contents of the help are stored in the database so that later I will be able to tie easily parts of the help file into parts of the program. This also makes it easy to export and import help files. The Help menu also has it's own built-in form for entering and editing categories.
If you would like to help internationalize TLE-BU click on "[ Help ]" then click on "[ Edit ]" at the top. To use the currently loaded help file as the template for your new only click on "[ Backup ]" and give you're new copy a name. Typically I use '**-<identifier>' where "**" is the typical two-letter short form of the language (ie: "en" for english, "jp" for Japanese, "fr" for French and so on). The "<identifier>" is what I used to indicate who is working on the file or, in the case of the default help file which is "canada", to indicate that I use Canadian/British spelling. This can ultimately be whatever you want and the default new help file name is "<date>-XX" where "-XX" is a serial number.
Remember to always backup your help file after you have made any changes! If you don't and the database is dumped or you load a different help file your changes will be lost! This warning is displayed prominently to help you remember.
Thank you for your time and I hope you are able to take something useful away from this presentation. Of course I would love nothing more than help from the community in continuing development of TLE-BU. This help could simply be emailing me your experiences using it, suggestions on how you think it could be made more intuitive or easier to use, problems you find and so on.
If you have any questions I can be reached on the TLUG mailing list or off-list at 'email@example.com' (Ed. That email is no longer used, please contact me at firstname.lastname@example.org).
About "The Linux Experience"
TLE is a migration, support and open-source software development company in the greater Toronto area. We specialize in helping clients migrate away from proprietary solutions in favor of Open Source Software. TLE contributes to the collection of Open Source Software through the development of it's "TLE-BU" backup software. TLE offers end-to-end support, training and services during all stages of migration including extended support post-migration. TLE serves as a single point of contact for all of their client's technology needs.
- TLE-BU on the web:
- http://tle-bu.thelinuxexperience.com/ (Ed.: no longer in business)
- TLE-BU on SourceForge:
- Note: TLE-BU is under active development and there are updates made available through SourceForge on a nearly daily basis. Please check in often for updates!
- Ed. TLE-BU is no longer maintained and is not recommended for use.
- The Linux Experience
Note: TLE was a migration, support and software development company. Their objective was migrating clients from proprietary to open-source and supporting them during and after.
-= The Linux Experience =-
|Any questions, feedback, advice, complaints or meanderings are welcome.|
|Us: Alteeve's Niche!||Support: Mailing List||IRC: #clusterlabs on Freenode||© Alteeve's Niche! Inc. 1997-2018|
|legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.|