It is currently Tue, 30 Nov 2021 17:09:53 GMT



 
Author Message
 Split input file based on size of file
Hello,

Can some one help in creating an output files based on the file size of
the individual files:

I have an input file with complete path of the existing files in a file
system as below:

Input.txt
--------------
/data/volume/admin1/file1.txt
/data/volume/admin1/file2.pdf
/data/volume/admin2/file3.cfg
/data/volume/admin3/file4.cfg

I would like to split the input files based on the file size of the
individual files.

/data/volume/admin1/file1.txt (file size 50 MB)
/data/volume/admin1/file2.pdf (file size 35 MB)
/data/volume/admin2/file3.cfg (file size 75 MB)
/data/volume/admin3/file4.cfg  file size 20 MB)

Say (if it exceeds 100 MB size, create a new output file)

Output1.txt
-----------------
/data/volume/admin1/file1.txt
/data/volume/admin1/file2.pdf

Output2.txt
/data/volume/admin2/file3.cfg
/data/volume/admin3/file4.cfg

Thanks in advance.



 Sat, 26 Apr 2008 02:27:33 GMT   
 Split input file based on size of file

 > Hello,
 >
 >
 > Can some one help in creating an output files based on the file size of
 > the individual files:
 >
 >
 > I have an input file with complete path of the existing files in a file
 > system as below:
 >
 >
 > Input.txt
 > --------------
 > /data/volume/admin1/file1.txt
 > /data/volume/admin1/file2.pdf
 > /data/volume/admin2/file3.cfg
 > /data/volume/admin3/file4.cfg
 >
 >
 > I would like to split the input files based on the file size of the
 > individual files.
 >
 >
 > /data/volume/admin1/file1.txt (file size 50 MB)
 > /data/volume/admin1/file2.pdf (file size 35 MB)
 > /data/volume/admin2/file3.cfg (file size 75 MB)
 > /data/volume/admin3/file4.cfg  file size 20 MB)
 >
 >
 > Say (if it exceeds 100 MB size, create a new output file)
 >
 >
 > Output1.txt
 > -----------------
 > /data/volume/admin1/file1.txt
 > /data/volume/admin1/file2.pdf
 >
 >
 > Output2.txt
 > /data/volume/admin2/file3.cfg
 > /data/volume/admin3/file4.cfg
 >
 >
 > Thanks in advance.
 >

It's really not clear what you're trying to do. Your words seem to
contradict your example. In case it helps, with GNU "ls", you can do
something like this:

ls -1 -S -s < Input.txt | awk 'BEGIN{sfx=1}$1<10{sfx=2}{$1="";print >
"Output"sfx".txt"}'

In that case, ls is sorting by block size and awk is checking for 10 or
more. Check the "ls" man page for more options and let us know if you
can't figure it out.

Regards,

        Ed.



 Sat, 26 Apr 2008 02:40:16 GMT   
 Split input file based on size of file

#!/usr/bin/perl

$hundred_meg = 100 * 1024 * 1024;

open IN, "<Input.txt" or die "open Input.txt: $!";

while ( $filename = <IN> ) {
    chomp $filename;

    if ( !$total || ( $total + -s $filename ) > $hundred_meg ) {
        open OUT, ">Output" . ++$count . ".txt"
            or die "open Output$count.txt: $!";
        $total = 0;
        }

    $total += -s $filename;
    print OUT "$filename\n";
    }

__END__

John
--
use Perl;
program
fulfillment



 Sat, 26 Apr 2008 03:18:10 GMT   
 Split input file based on size of file

Since you posted also in comp.lang.awk I suppose you might be interested in
an awk solution. But beware; I just hacked it without any test or review...

ls -l $(< Input.txt) |
   awk 'BEGIN { nr=1; limit=100000000 }
        { if( (sum += $5) > limit) { close("Output"nr++".txt"); sum = $5 }
          print $1 >"Output"nr".txt"
        }'

What it should do... Accumulate file size until limit exceeds, if so close
the previous file, add 1 to the file name counter and reset sum to current
value, finally print filename to the output file with current number.

Janis



 Sat, 26 Apr 2008 03:41:58 GMT   
 Split input file based on size of file
Hi Janis and Others,

Thanks you all for the reply and sorry for the initial confusion.

Janis, the script works just the way I wanted.

Regards, Karthik



 Sun, 27 Apr 2008 00:01:32 GMT   
 
   [ 5 post ] 

Similar Threads

1. Splitting an output file based on it's size (UNIX)

2. Split binary files into 2 files of different size (or eliminate some bytes from the start)

3. Stopping offsite file linking based on file size?

4. Size max files based on memory size

5. "Split" input into two files

6. Need help splitting a file based on key fields

7. split a file based on pattern

8. splitting a very large file based on characters in a record (performance)

9. Large Text File Split based on a field value

10. Help W/Unix Shell Script - File Splitting Based on Text String


 
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by ST Software