It is currently Fri, 01 Aug 2014 23:31:26 GMT



 
Author Message
 Tar Limitations
We are having a few problems using tar with large volumes of data. For one,
we can't create an archive with more than 2147483648 bytes in it
(suspicously looking like a signed int problem). For two, when trying to
read large archives (7 Gig +) from a third party we get erratic behavior,
with tar skipping headers and throwing garbage on the screen (suspicously
looking like overflow or pointer issues).

We are using Red Hat 6.0. Are there known limitations to tar? Can it only
handle 2 gig archives? How many items can be in an archive?

This is NOT a disk space problem (unless perhaps in /tmp or /var) as the
filesystems typically have 10 to 15 GB of free space.

Any help would be appreciated.

Thanks.



 Sun, 27 Jan 2002 03:00:00 GMT   
 Tar Limitations

Never tested it but Linux should currently have a 2Gb file size limit.

Have you tried named pipes to get around the problem?

e.g.

mkfifo /tmp/pipe ( is this the command to make a named pipe in Linux
or is it mknod? )

split -b 1024m /tmp/pipe split.aaa &

tar cvf /tmp/pipe *

The above was typed in without testing but the idea is to create a
named pipe, start a background process to read information from the
named pipe and create lots of smaller files from the input from the
pipe. A finally send the output of tar to the named pipe.

This is a very useful technique from getting around the 2Gb limit.

HTH

Robin



 Sun, 27 Jan 2002 03:00:00 GMT   
 Tar Limitations

What architecture are you running on? AFAIK, i386 Linux will not handle
files with sizes > 2Gb, as it runs out of address space for memory mapped
I/O. I believe there is *some* way of doing it, but you'll need to
recompile things (including tar) - and I'm afraid I don't have any
references for it. If you're on Alpha (or another 64 bit architecture)
then you may have found a bug in tar.

Have you considered using tar -M for multiple volume support? I'm not
sure exactly how it works, but you may want to have a play with it.

--
Jon Skeet - sk...@pobox.com
http://www.pobox.com/~skeet/



 Sun, 27 Jan 2002 03:00:00 GMT   
 Tar Limitations
On Wed, 11 Aug 1999 15:26:16 +0100,
 Jon Skeet <sk...@pobox.com> wrote:

Actually, you would have to do a lot of work to do it.

Think of 'lseek' or 'fseek' and how they would handle >32bit values for
offsets.

The only real solution is to leave the 32 bit world: anything else is
really ugly.

--
Brian Moore                       | Of course vi is God's editor.
      Sysadmin, C/Perl Hacker     | If He used Emacs, He'd still be waiting
      Usenet Vandal               |  for it to load on the seventh day.
      Netscum, Bane of Elves.



 Sun, 27 Jan 2002 03:00:00 GMT   
 Tar Limitations
In article <14fs3.2485$5E3.92...@ndnws01.ne.mediaone.net>,

The limit is in the disk file holding the archive.  You can go a
bit bigger by using the -z flag which arranges a pipe to
compress.  I'm not sure, but I would expect it to have similar
problems reading archives from other systems if any of the
content files were >2gigs.  You shouldn't have other problems
if you can use a tape device directly or use pipes to avoid
needing a single file >2gigs.

  Les Mikesell
   l...@mcs.com



 Sun, 27 Jan 2002 03:00:00 GMT   
 Tar Limitations
Thanks for the replies to date. As a follow up. We are using i386 / 32 bit
systems.
I did try various ways to get around any filesystem issues, and have become
convinced
that our core issue in this case is a tar/OS/tapedrive incompatibility
issue, not a 2GB file size issue. (Although it has become painfully evident
that Linux cannot handle files > 2GB!).

I can dd the tape device directly just fine. I can start to tar from the
tape device, only tar quickly
gets confused and starts skipping files in the archive. I can dd the tape to
a raw partition and
tar from the raw partition - only to get the same result.

The vendor supplying the tapes for us is using IRIX. As a short term attempt
to resolve the issue
they are switching to GNU tar, rather than the stock IRIX tar. If that
doesn't work they will try to
switch to a Linux variant for tape creation.

Thanks again to those that replied.

<14fs3.2485$5E3.92...@ndnws01.ne.mediaone.net>...



 Mon, 28 Jan 2002 03:00:00 GMT   
 
   [ 6 post ] 

Similar Threads

1. TAR command 2GB limitation

2. TAR command 2GB limitation

3. tar -xvf mozilla.tar = tar: directory checksum error

4. Install TAR and TAR.GZ

5. tar -cvfX save.tar foo ./dirtosave/..

6. Scripting Help: tar a dir with time and date as file tar file name..................TIA

7. tar/gnu tar question

8. rsh, rcp, GNU tar cvf user@host:./tarfile.tar file FAIL


 
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by ST Software