CLKF: Splitting files in linux

Command Line Kung Fu: Splitting files in linux

So I have a flash drive thats pretty irrevocably damaged, and for some reason or another it wont format into NTFS, but happily formats into FAT32 (exFAT is an option that I chose to ignore). And I have a 4.4GB file to transfer.

We all know that storing a file larger than 4GB is not possible on a partition formatted in FAT32 (which is a pretty old Microsoft file system type, when no-one actually had files larger than a few megabytes, now superseded by NTFS in an age when video files, Operating System images and games come at several Gigabytes in size, and a Terabyte is becoming the norm for any PC sold); In any case the solution to this problem was to split the large file into several smaller chunks that would fit on the drive or several drives.


There is a whole heap of tools available to this end, both freeware and commercial. A quick search on google reveals that the choices available are numerous. Any compression program (7Zip ftw!) does file splitting too. What I want is a command line tool, that will do the task as minimalistically as possible.

One of the original linux tools created for this purpose was 'tar', which stands for Tape ARchive, which is able to bundle a file or a collection of files into one (or several) file (essentially a precursor for ZIP without compression) ready to be stored on what was the best form of backup medium in those days, magnetic tape.

So using tar on the command line, heres how to split a file up:

$ tar -cvM --tape-length=1024000 --file=OUTPUT_FILE1.tar ORIGINAL_FILE

The --tape-length parameter accepts data in kilobytes, so 1024000 = 1GB. Just keep entering sequential filenames when it asks for another filename for each part it creates.
And to join files back up:


$ tar -xvM --file=INPUT_FILE1.tar ORIGINAL_FILE


So that works, but it isn't the most simplistic solution; there is also the much more simpler, split and cat commands:


$ split -b 1024m debian.iso debiansplit


and then when I had to join the files back:


$ cat debiansplit* > debian.iso



split command (click to enlarge)

Some more awesome Command Line Kung Fu to the rescue!

Command Line Kung Fu

Recently I had to migrate hard drives, due to a disappointing Samsung Spinfoint F3 hard drive failure (this is the second Samsung F3 1TB hard drive that I've had, thats gone on to meet its maker). This drive wasn't fully dead, and I figured I'd take what I could whilst it was on its last leg. Thanks in part due to Amazons speedy delivery of  a 2TB Seagate Barracuda Eco-Green Sata 3 (6Gb/s) drive.

Now in my opinion, windows copy and paste, just doesn't cut it for copying large files. It may say it has copied the files but I have had MD5 hashes fail before, and given that this hard drive wasn't performing optimally I just didnt want to risk copying and then pasting corrupt data. I usually prefer copying from the command line as I find it a bit more effective.

There are win32 versions of md5 hash checking programs available, but doing singular hash checks sucks when you have multiple files in multiple folders, hidden in multiple depths. I needed a recursive program to check md5 sums for files with certain extensions only... hmmmn.

Batch file with some sort of for loop? Nah, since every windows box I have has a copy of the excellent cygwin program installed, why not go all out and do it the linux way? I mean I have the Bash shell, with a whole load of unix utilies installed on the computer, including md5sum; So ladies and gentlemen, I now produce below, for your perusal, the extravagant single liner I used to alleviate my problems (not all of them, mind you, I still have to claim on my Samsung's warranty, but this command solved one of my many problems):

#find ./ -maxdepth 5 -type f -name "*.iso" -exec md5sum {} \;

This should ideally be piped into some sort of text file for comparision to the source, to ensure that the files were copied exactly bit-for-bit as opposed to a logical copy. Cygwins 'cp' command outdid windows default file copy routines, since all of the md5 hashes checked out. Attached is a screenshot of the sort of output this command produces. (Click to see enlarged version)

Once again, some awesome Command Line Kung Fu saves the day. Happy new year!

I should point out a few things here:

Cygwin is a great tool, built upon greater tools with some excellent  (and ancient) Unix heritage.

Windows 'just works' most of the time, perhaps the file copy errors were hardware faults (i.e. temperature, seek speeds, time of day) - so this is by no means a comprehensive or fair experiment.

I'm aware there is a utility (with a cygwin port) called md5deep, but it wouldn't search recursively for .iso files. Their 'man' page states 'Please note that recursive mode cannot be used to examine all files of a given file extension'.

And finally Samsungs spinpoint drives have recieved some excellent reviews, I was perhaps unlucky with 2 faulty drives (bought at different times), dont let this put you off Samsungs hard drives, they are an awesome company coming up with some awesome products (foreshame on Apple for trying to stall competition in court by some very underhanded techniques against Samsung).