It is currently Wed, 20 Jan 2021 16:16:03 GMT



 
Author Message
 copying a lot of files
Hi,
next week, I will have to copy a lot of files ( 6500000 files, 30 GB)
on a Sol 2.8 machine with ufs FS from one volume to another volume.
Syncing with VxVM is not an option, so find/cpio , ufsdump/ufsrestore
and tar came to my mind. Could someone tell me which of those tools
should be the fastest ?


 Mon, 25 Sep 2006 01:05:53 GMT   
 copying a lot of files

In all likelihood, you'll be limited by the speed of creation of the
files on the remote filesystem rather than the speed of the tool.  You
might want to use fastfs or at least logging on the target filesystem.

'star' is often claimed to be faster than any you've mentioned.

If you have the memory, you might want to use 'rsync'.  You could do an
initial sync prior to the cutover.  Then you do another one for the
actual cutover which only copies a few changes.  With 6.5M files, it'll
take a while to build the file tree, but it should work.

--
Darren Dunham                                           ddun...@taos.com
Senior Technical Consultant         TAOS            http://www.taos.com/
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >



 Mon, 25 Sep 2006 01:43:19 GMT   
 copying a lot of files
In article <2b7e55f1.0404070905.12b44...@posting.google.com>, jrm_66
@hotmail.com says...

Are the volumes on the same system, or on two networked systems?

1. SAME SYSTEM.  Just use "cp" (in a shell script, if you're copying
files from a lot of different unrelated directories).  Launch it shortly
before you go home for the day and let it finish over night.

2. DIFFERENT SYSTEMS.  I'd just NFS mount the directory from one machine
to the other, and use "cp" as mentioned above.  That's going to take a
while (especially if your network is slow), but like I said just kick it
off at the end of the day and it will be done in the morning.

COMPLICATION #1 -- IF YOU CAN'T LET THE PROCESS RUN OVER NIGHT.  Then
try to time the operation with another project that will keep you busy
for a few hours, and then you can take a break and check on the copy
progress every now and then.

COMPLICATION #2 -- IF YOU HAVE TWO DIFFERENT SYSTEMS, WITH A FIREWALL
BETWEEN THEM THAT BLOCKS NFS.  Damn those network administrators.  If
you're in this situation, and you can't bribe the network guys into
letting NFS traffic through for a few hours, then I'd recommend
archiving to tape.  That will really blow, because it's even slower than
copying over a network and you'll have to sit there and swap tapes...  
For 30GB you might be looking at 16 hours of waiting!  {*filter*}!  Or if the
firewall permits FTP, and if you have 30GB of spare room on both
systems, you could archive to a file and then FTP that, which would
still be slower than using NFS but it would be a hell of a lot faster
than using tapes.  All I can say is, try to have some other project to
work on so you won't go stircrazy waiting for tar to finish (and if you
have to use tapes then take the next day off).

COMPLICATION #3 -- SOME OF THE FILES YOU NEED TO COPY ARE SPECIAL FILES.  
Okay, "cp" won't work for this, because it will access the device that a
special file represents instead of actually copying the special file
itself.  That could be frustrating, because you'd launch your script and
then go home, only to return in the morning and find that "cp" has only
copied 3 files because it was sitting there all night trying to copy
/dev/null for you.  Heh.  For special files, you'll need to use tar,
because it will correctly archive/unarchive special files.  I think cpio
works for that, as well.



 Mon, 25 Sep 2006 02:18:00 GMT   
 copying a lot of files

That complication is so huge, I'd never recommend 'cp' for moving data.
Besides the fact that it won't move special files, it will also try to
traverse symlinks (imagine recursive ones).

Multiply linked files and sparse files are also a problem for some
programs.

--
Darren Dunham                                           ddun...@taos.com
Senior Technical Consultant         TAOS            http://www.taos.com/
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >



 Mon, 25 Sep 2006 02:44:09 GMT   
 copying a lot of files

Well, also you will need to remember to do the NFS export so that
the root user is not mapped to "nobody".  Otherwise, you will not
be able to copy any files readable only by root.

   - Logan



 Mon, 25 Sep 2006 02:54:28 GMT   
 copying a lot of files

If you are able to do an extra copy ahead of time to do most of
the work in advance, then you can do this:

        ufsdump 0ufN - mylabel /path/to/filesystem |
                ( cd /new/filesystem && ufsrestore rf - )

This will put a record of the level 0 dump into /etc/dumpdates,
but the record will be under the name "mylabel" instead of the
normal device name.

Then when it comes time to do the big switchover, you do a level
1 dump, referencing that label in /etc/dumpdates:

        ufsdump 1ufN - mylabel /path/to/filesystem |
                ( cd /new/filesystem && ufsrestore rf - )

Because you are using the "r" and not the "x" argument to ufsretore,
you are doing a restore, not an extraction, and this should in theory
take care of all the issues related to incremental dumps, like
renaming files, removing files (on the source) between the two dumps,
etc.

This method only makes sense if you have a regular amount of activity
on the original filesystem.  If you have a huge amount of activity,
the incremental dump will take basically just as long as a full dump
would.

Of course, you can do this across machines with "rsh" or whatever
as well, although it sounds like you have both filesystems on
the same machine.

   - Logan



 Mon, 25 Sep 2006 03:00:39 GMT   
 copying a lot of files
In article <2b7e55f1.0404070905.12b44...@posting.google.com>,

star -copy is the fastest

ftp://ftp.berlios.de/pub/star/alpha/

http://www.blastwave.org/packages.php

--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
      j...@cs.tu-berlin.de                (uni)  If you don't have iso-8859-1
      schill...@fokus.fraunhofer.de       (work) chars I am J"org Schilling
URL:  http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily



 Mon, 25 Sep 2006 20:38:32 GMT   
 copying a lot of files
In article <rdYcc.37611$Tx6.7...@fe2.texas.rr.com>,
Logan Shaw  <lshaw-use...@austin.rr.com> wrote:

The best is to use:

        star -copy -p -sparse fs=128m -xdot -no-fsync -C fromdir . todir

You may like to add '-acl'.

http://www.blastwave.org/packages.php

--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
      j...@cs.tu-berlin.de                (uni)  If you don't have iso-8859-1
      schill...@fokus.fraunhofer.de       (work) chars I am J"org Schilling
URL:  http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily



 Mon, 25 Sep 2006 20:41:23 GMT   
 copying a lot of files
[Joern ]

|  Hi,
|  next week, I will have to copy a lot of files ( 6500000 files, 30 GB)
|  on a Sol 2.8 machine with ufs FS from one volume to another volume.
|  Syncing with VxVM is not an option, so find/cpio , ufsdump/ufsrestore
|  and tar came to my mind. Could someone tell me which of those tools
|  should be the fastest ?

Have you considered rsync?  (http://rsync.samba.org/) I usually set up
rsync and run it on beforehand (on the live system) and then close all
access to the system while I do the last synchronisation - saves a lot
of time with rater static files and I want minimal down-time.

-Sigurd



 Mon, 25 Sep 2006 20:50:02 GMT   
 copying a lot of files

Special files are still a problem, but symbolic links are handled well
with GNU cp if you use the option -d. But it's not the fastest tool,
no.

--
                - M?rten

mail: m...@kth.se *** ICQ: 4356928 *** mobile: +46 (0)707390385



 Mon, 25 Sep 2006 23:38:02 GMT   
 copying a lot of files

Yesterday I had to copy the /home filesystem (7 GB) on my Sol8 (117000-03)
E3500 system from a Disksuite mirrored internal FC-disk to an external
Disksuite mirrored Ultra2-SCSI-disk.
I did that "ufsdump 0f - <fs> | ( cd ...; ufsrestore rf - )" copy in
single user mode with no other disk activity occuring with all four
disks attached to their own controller, respectively.
It took me more than two hours, with ufsdump reporting a mere 900 KB/s
transfer rate!
Any ideas why it took so long? Is it due to the many small files?
Would Sol9 perform better?

Uwe



 Tue, 26 Sep 2006 17:22:41 GMT   
 copying a lot of files
In article <20040409112241.46110f4a.uwe.wolf...@dummy.onlinehome.de>,
Uwe Wolfram  <uwe.wolf...@dummy.onlinehome.de> wrote:

Star would perform better.

It definitely helps to first run:

fastfs /path_to/todir fast

Then run:

        star -copy -sparse -acl -p fs=128m -xdir -no-fsync -time -C fromdir todir

ftp://ftp.berlios.de/pub/star/alpha/
http://www.blastwave.org/packages.php

Do not forget to run

fastfs /path_to/todir slow

afterwards.

This should be 40% + faster than with ufsdump/ufsrestore.

The main reason why ufsdump is so slow is that it uses 25 year old strategies
for buffering. Star is not limited to 6xb block size buffers. With the example
above, you get 128 MB buffer.

--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
      j...@cs.tu-berlin.de                (uni)  If you don't have iso-8859-1
      schill...@fokus.fraunhofer.de       (work) chars I am J"org Schilling
URL:  http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily



 Tue, 26 Sep 2006 19:52:25 GMT   
 copying a lot of files

Well, 1.5 MB/s is still a bummer considering the hardware should be able
to run at about 10 MB/s.

Uwe



 Wed, 27 Sep 2006 19:37:30 GMT   
 copying a lot of files
In article <20040410133730.7ee97250.uwe.wolf...@dummy.onlinehome.de>,
Uwe Wolfram  <uwe.wolf...@dummy.onlinehome.de> wrote:
...

Did you test and mesure?

I don't know how you come to 10 MB/s....

If you are copying over an old and slow SCSI bus that is limited to 10 MB/s,
you need to know that the sum of read and write operations is limited to
10 MB/s which is 5 MB/s copy speed max.

If you are copying from one disk to the same disk, then you need to know that
you have additional seeks that you would not get when copying between different
disks.

Also the implementation of the filesystem in the kernel may be a limiting
factor. If you are using a too old version of Solaris, you need to know that
there has been a single mutex for all directory operations that limited the
FS write throughput.

--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
      j...@cs.tu-berlin.de                (uni)  If you don't have iso-8859-1
      schill...@fokus.fraunhofer.de       (work) chars I am J"org Schilling
URL:  http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily



 Fri, 29 Sep 2006 21:53:16 GMT   
 copying a lot of files

The busses are limited to 20MB/sec. 10MB is just a wild guess from what
I get from the disks copying large files. BTW, ufsdumping the filesystem
to LTO drives maxes out at 8MB/sec. So the dump|restore combo must be
a limiting factor.

Uwe



 Sun, 01 Oct 2006 02:01:54 GMT   
 
   [ 22 post ]  Go to page: [1] [2]

Similar Threads

1. Help using awk and bash script to copy and rename lots of index.htm files

2. Server crashes copying lots of data

3. I want to not copy exclude one file when i Copy

4. copy all files in a dir to another renaming them as they are copied

5. Split a big txt file to lots of HTML file ~~

6. Lots o lots o questions!!

7. Lots and lots of messages

8. Intelligent mouse freezing in copy files through KDE file manager

9. copy umsdos file system to ext2 file system

10. Copy files using filenames from text files with shell script or bash script


 
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by ST Software