A better way of synchronising files between several linux clients and a linux backup server

We currently run a backup service that uses a cron-job scheduled rsync-based script to synchronise files between several small file servers and a backup server. However, the limited broadband bandwidth of the file servers means that jobs often get stuck on large files and never get around to backing up the small files. The current backup log is also very hard to use and does not detail the number of files that remain to be backed up and how out of date each file has become. We require a new linux backup script (or other) that will backup files in a better order to ensure the maximum number of files are backed up given the constraints. We also require a better log so that we know exactly how out-of-date? the backups of each of? the clients are. ## Deliverables We require modifications to the backup script and/or an application that can run on the Linux system that will iterate the files that remain to be copied and order them by a score based on both size (small files should backup first) and how out of date they are (files that are more out of date should copy first). The files will then be copied and a log detailing all the differences be saved on the system (using the date as a filename) and emailed to us. The exact calculation used to create a score for each file must be agreed with us during development. The system should take a different approach for large files (> 50MB for now but size must be defined in the script). Often the content of such files change daily (such as Outlook pst files) and as such they are never backed up. When the script or application encounters a file of this size or larger it should take a copy and then ensure that the copy is fully backed up before starting again on the latest copy. In this way large files will always be successfully backed up although a backup copy will only be obtained every few days or more. It would also seem a good idea to break these copies of larger files into smaller pieces before sending them so that they are more likely to be sent. We’re happy for the files to remain in pieces server-side as long as they could be reassembled easily to form the original file at a later point. The progress of all files should also be included in the log. An example log would read as follows: Client name: ClientX Backup date: 26/02/08 Total files: 4,567 Files different: 5 Different Files: File Name? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? File Size? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Client Date? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Server Date BIGFILE_CPY_080222_PT#1? ? ? ? 50MB? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 22/02/08? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 12/02/08 BIGFILE_CPY_080222_PT#2? ? ? ? 32MB? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 22/02/08? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 12/02/08 [login to view URL] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1.2MB? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 25/02/08? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 19/02/08 [login to view URL] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 5MB? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 24/02/08? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 20/02/08 [login to view URL] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 3MB? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 24/02/08? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 20/02/08 [login to view URL] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 17MB? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 23/02/08? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 17/02/08 Notice how the file “BIGFILE?? which was originally 82MB has been broken up into two parts before sending. The original file “BIGFILE?? is not included in the backup, rather its copy made on the 22nd (“BIGFILE_CPY??) is being transferred. It has also been broken up into two parts (one 50MB and one 32MB). The successful applicant will be given SSH access to our server and one of the backup clients to develop the system. We will then test the system for all our clients to ensure it works before the project is complete.