Jump to content

Photo

Recursive Copy for MyDOS


74 replies to this topic

#26 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 14,603 posts
  • Location:United Kingdom

Posted Thu Feb 12, 2009 4:51 AM

My system used a free IOCB for each level in the tree; clearly this is going to cause problems with deeply nested folders. Surely it's possible in MyDOS to NOTE the point reached in the directory before the recursive call, then you'd just have to close the IOCB and do the recursive call, passing the path of the folder just found in the directory listing. When the routine exits, local variables will hold the NOTE value: just re-open the directory file and set the file pointer.

Edited by flashjazzcat, Thu Feb 12, 2009 4:54 AM.


#27 danwinslow OFFLINE  

danwinslow

    River Patroller

  • 2,593 posts

Posted Thu Feb 12, 2009 9:07 AM

I don't like the temp file idea either, on aesthetics, unless it were to ramdisk, but even then its kinda cheesy. The note values sounds interesting...I thought that was kind of a pointer to a location like lseek() etc...does that work with directories? I know they are files on linux and so forth but I didn't think they were files on Atari dos's. I dimly remember something about a VTOC sector(s) so maybe thats what you/drac are referring to, offsets within the directory sectors?

I think Rybags is right about managing a list of filenames in memory wouldn't be too bad...might not support really pathological cases of nesting, but it could probably work. You could get a thousand or so filenames in 16k, for instance. Kind of a brute force approach but might work.

Drac mentions a good point about recursion...its really not necessary and usually isn't worth it in terms of resources.

#28 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 14,603 posts
  • Location:United Kingdom

Posted Thu Feb 12, 2009 9:28 AM

Drac mentions a good point about recursion...its really not necessary and usually isn't worth it in terms of resources.

But the source code can look rather elegant! :) In SDX, I know offsets in directory files (certainly unformatted ones) can be noted and pointed just like any other file, but whether this holds true with MyDOS (or the AtariDOS 2.5 FS as a whole), I don't know. SDX uses sector maps (MyDOS uses sector links) and can position the drive head pretty quickly to any point in the file. We don't need the random access SpartaDOS can do, though: as long as we can make a note of where we left off, and re-open the directory later on and point to the same place again, a recursive routine will be easy to accomplish in C or some such.

There was another way I once tried to write a routine just like this, which was to build a list in memory of all the files on the disk, complete with their paths. This was easy to do without recursion, using a couple of pointers. It's also probably the best way to do it if you're going to be doing disk swaps. Bear in mind a list of destination paths will also be required, as well as filename transformation routines to deal with wildcards.

#29 drac030 OFFLINE  

drac030

    Stargunner

  • 1,836 posts
  • Location:Warszawa, Poland

Posted Thu Feb 12, 2009 10:03 AM

But the source code can look rather elegant! :) In SDX, I know offsets in directory files (certainly unformatted ones) can be noted and pointed just like any other file, but whether this holds true with MyDOS (or the AtariDOS 2.5 FS as a whole), I don't know.


I can assure you that the iterative code looks equally elegant, and there is no risk of overflowing the 6502 stack. For directory offsets, of course, under SpartaDOS X the program stacks the offsets relative to the beginning of the directory, as NOTE/POINT works like lseek() in Unix. But, as you say, it does not make much difference under MyDOS, where NOTE/POINT use physical sector numbers and offsets, it is enough to stack the result of the NOTE call, and pass it later to POINT, the result will be the same.

#30 Shawn Jefferson OFFLINE  

Shawn Jefferson

    Stargunner

  • Topic Starter
  • 1,987 posts
  • Location:Victoria, Canada

Posted Thu Feb 12, 2009 8:00 PM

If you use NOTE/POINT in MyDOS, would you then need to handle reading the directory files yourself? ie, you couldn't use OPEN #1, 6, 0 CIO call?

Well, I just tested it with a quick program in BASIC, and NOTE doesn't seem to give you any usable values using the CIO call above.

Edited by Shawn Jefferson, Thu Feb 12, 2009 8:13 PM.


#31 drac030 OFFLINE  

drac030

    Stargunner

  • 1,836 posts
  • Location:Warszawa, Poland

Posted Fri Feb 13, 2009 1:25 AM

Well, I just tested it with a quick program in BASIC, and NOTE doesn't seem to give you any usable values using the CIO call above.


Hmm, yes, I didn't think about it, but MyDOS probably lacks the "direct directory access" mode, which is needed here in SDX. So, another solution would be to stack the directory entry numbers. After the program gets one level up and pulls the number, it will simply have to skip this number of entries and start processing the next one (unless this is an end of the directory).

#32 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 14,603 posts
  • Location:United Kingdom

Posted Fri Feb 13, 2009 5:55 AM

Hmm, yes, I didn't think about it, but MyDOS probably lacks the "direct directory access" mode, which is needed here in SDX. So, another solution would be to stack the directory entry numbers. After the program gets one level up and pulls the number, it will simply have to skip this number of entries and start processing the next one (unless this is an end of the directory).

If we can't note/point to the directories, I think it would be quicker to iteratively (or recursively) read all the directories on the disk into memory before doing anything else. Then we can step through them and process them any way we like without having to re-read or skip data we've already read. My idea was to first build a list of all matching files on the disk anyway, so I think it makes sense. I might have a go at this over the weekend if I get time. I hope we get lots of different programming ideas here. :)

#33 8bitguy1 OFFLINE  

8bitguy1

    Moonsweeper

  • 347 posts
  • Location:Edmonton, Alberta

Posted Fri Feb 13, 2009 5:56 AM

Would it not make sense to look at the approach used in Copy Files'n'Folders from BossX written in Turbo Basic I believe and then try to construct/improve on that in different language?

#34 drac030 OFFLINE  

drac030

    Stargunner

  • 1,836 posts
  • Location:Warszawa, Poland

Posted Fri Feb 13, 2009 6:32 AM

If we can't note/point to the directories, I think it would be quicker to iteratively (or recursively) read all the directories on the disk into memory before doing anything else.


But this, in extreme case, will eat all the memory, so your copying buffer will get very short. It is also possible that all the directories won't fit in the memory. I think it is not necessary anyways, "seek" through the directory (using the GET RECORD function) to the specified entry number will be fast enough imho. Remember that MyDOS directories can only have 64 entries each, so it is a matter of reading ~1.5 KB of text data (formatted directory, that is). Directory offset can be held in one byte, so the stack will be even smaller than in SpartaDOS. The rest is a question of proper path manipulation (adding stuff at the end of it, when going down the directory tree, or cutting it down, when going up).

Edited by drac030, Fri Feb 13, 2009 6:34 AM.


#35 Rybags OFFLINE  

Rybags

    Gridrunner

  • 16,145 posts
  • Location:Australia

Posted Fri Feb 13, 2009 6:48 AM

Maybe get tricky and keep a Directory "cache".

Have smart allocation dependant on how much free RAM is available. Working out entries that can be flushed shouldn't be too much problem.

#36 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 14,603 posts
  • Location:United Kingdom

Posted Fri Feb 13, 2009 11:37 AM

But this, in extreme case, will eat all the memory, so your copying buffer will get very short. It is also possible that all the directories won't fit in the memory. I think it is not necessary anyways, "seek" through the directory (using the GET RECORD function) to the specified entry number will be fast enough imho. Remember that MyDOS directories can only have 64 entries each, so it is a matter of reading ~1.5 KB of text data (formatted directory, that is). Directory offset can be held in one byte, so the stack will be even smaller than in SpartaDOS. The rest is a question of proper path manipulation (adding stuff at the end of it, when going down the directory tree, or cutting it down, when going up).

Good point: the overhead of seeking through the directories will be low. I once wrote an XCOPY command years ago (which I've completely lost), and for some reason, keeping everything in memory worked best. I don't know if that will be the case on a second attempt!

Maybe get tricky and keep a Directory "cache".

Have smart allocation dependant on how much free RAM is available. Working out entries that can be flushed shouldn't be too much problem.

Yep: storage space for file caching can increase as we work through directories (we can dispose of what's been used). I have a fairly clear idea of how I'd approch this: a structure for each source directory comprising the complete path of the directory, followed a list of all the matching filenames. We then loop iteratively through each structure, creating a new folder on the destination drive if necessary (and appending a dest dir path to the destination folder spec if one was supplied), then applying a filename transform algorithm to each name in case the user provided a non-ambiguous destination file mask (such as "*.BAK").

We have to have "*.*" as the source DIR because we need to read all the directory names as well, so we need our own wildcard matching routine (already done). A menu-based utility would be nice but will take longer to write (I envision having a go at writing something in assembler).

Edited by flashjazzcat, Fri Feb 13, 2009 11:42 AM.


#37 drac030 OFFLINE  

drac030

    Stargunner

  • 1,836 posts
  • Location:Warszawa, Poland

Posted Fri Feb 13, 2009 12:29 PM

I once wrote an XCOPY command years ago (which I've completely lost), and for some reason, keeping everything in memory worked best.


Sure, it will work - for small directory trees. But imagine having 64 subdirectories in the root dir, each one containing 64 files (4096 files - perfectly possible on a hard drive). To load all this to memory you need 130 kilobytes, and this is just one level of recursion.

#38 bf2k+ OFFLINE  

bf2k+

    Stargunner

  • 1,773 posts
  • Location:Boot Factory BBS 2k+

Posted Fri Feb 13, 2009 4:23 PM

I once wrote an XCOPY command years ago (which I've completely lost), and for some reason, keeping everything in memory worked best.


Sure, it will work - for small directory trees. But imagine having 64 subdirectories in the root dir, each one containing 64 files (4096 files - perfectly possible on a hard drive). To load all this to memory you need 130 kilobytes, and this is just one level of recursion.


Writing a .tmp is looking better all the time :cool: .

I don't use MyDOS meself so don't mind me...

But I do have some hellacious big SpartaDOS harddrives and ATR's. Drive 1 on the BBS only has 9000 free sectors.

#39 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 14,603 posts
  • Location:United Kingdom

Posted Sat Feb 14, 2009 4:13 AM

Sure, it will work - for small directory trees. But imagine having 64 subdirectories in the root dir, each one containing 64 files (4096 files - perfectly possible on a hard drive). To load all this to memory you need 130 kilobytes, and this is just one level of recursion.

You're right, drac. I've come up with a better idea that doesn't involve temp files or anything: If we, say, allow up to ten files to be copied per "pass" (as in SDX's MENU program), then the only things we need to store in memory (apart from the file contents), are a maximum of the most recent ten directory entries. As complete files are written to the destination directory, finished entries can be pulled off the list, and new ones added to then end. When the source disk is inserted again (assuming disk swaps, but it will always work the same), we open the directory and skip all the entries up to the one after the last one in the list in memory.

I've started writing something, anyway, as a bit of a break from LW! :D I think it's an interesting programming challenge.

#40 Shawn Jefferson OFFLINE  

Shawn Jefferson

    Stargunner

  • Topic Starter
  • 1,987 posts
  • Location:Victoria, Canada

Posted Sat Feb 14, 2009 2:27 PM

But this, in extreme case, will eat all the memory, so your copying buffer will get very short. It is also possible that all the directories won't fit in the memory. I think it is not necessary anyways, "seek" through the directory (using the GET RECORD function) to the specified entry number will be fast enough imho. Remember that MyDOS directories can only have 64 entries each, so it is a matter of reading ~1.5 KB of text data (formatted directory, that is). Directory offset can be held in one byte, so the stack will be even smaller than in SpartaDOS. The rest is a question of proper path manipulation (adding stuff at the end of it, when going down the directory tree, or cutting it down, when going up).


Which sounds a lot like my original idea. Storing the directory entry number is better than storing the filename though. That's only one byte... I was thinking of writing this in C myself, which handles most of the tricky recursion. With some thought you can can probably get the amount of bytes needed on the C stack for each call down to 3 or 4 bytes... all other variables can be reloaded in the file copy loop.

Also, I think that your major bottleneck in a program like this is going to be the actual copying of data, and any re-reading of directories will amount to very little time... I think this is the best approach, since you get two benefits:

1. more memory for the copy buffer.
2. no temp files stored on disk.

#41 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 14,603 posts
  • Location:United Kingdom

Posted Wed Feb 18, 2009 1:33 PM

I've been possessed by this recursive file copier since the weekend now! :D Here's what I've managed so far:

Attached File  MyCopy1.zip   11.72KB   89 downloads

The (assembly) coding is a bit longhanded, so I think I could shave a few hundred bytes off the executable size. Still, if you turn off BASIC you get a copy buffer of about 30K. This program will handle disk swapping, but it's not implemented yet (more on this later). I haven't tested this much at all; I've just tried copying small directory trees from physical disks to RAMdisks and it seems to behave OK. I rely on others to try it exhaustively with MyDOS and tell me if it falls down.

Although recursion wasn't necessary (which is good news, given this is written in assembler), I would have used it if I'd written this in C (which I wish I had, given that the state of the assembly code doesn't on its own explain much about how I tackled the problem). In essence there's a string which contains the filespec of the source file; this string expands and contracts as we hit subdirectory names. When a subdirectory name is found, the position in the current directory is saved on a stack, the stack pointer is incremented, the subdirectory is opened, and the process is repeated. Disk swapping is catered for by a 20-entry filename cache; the destination files aren't written out until 20 entries have been read, or the copy buffer is full - whichever happens first. Other than that, I have a table of flags to say whether a given entry is a folder name (for an MKDIR command on the destination), or an ordinary file to be written out.

I mentioned disk-swapping before; this will be easy to implement but it led me to think about some potential problems with source and destination paths. Firstly, "D:" will have to be expanded to include the default drive so that we can establish whether the source and destination drives are the same. I haven't done this yet, but I don't think it will be too hard to derive the necessary information from MyDOS's internal tables. Secondly, we have the problem of - assuming the source and destination drives are the same and disk swapping is NOT required - the program trying to recursively copy the destination directory. Given that we can access a given path relatively in many different ways in the filespec (the default folder, parent folder, etc), it's important to detect if the program tried to employ a destination folder (which it may just have created) as a source folder. Doing this effectively is giving me a headache at the moment, but I think I'll have to interrogate MyDOS's Working Directory path and make both the source and destination paths explicit from the root directory. Once that's done, it should be possible to check when the beginning of the source path points to the destination folder.

Phew... this version should be good for a copy from drive 1 to drive 2, or to a RAMdisk, etc. Be as explicit as possible with source and dest filespecs - i.e. include the device and *.* or whatever at the end. You CAN specify, say, D8:*.BAK as the destination path and all the files will have a BAK extender on the destination drive.

I should add that years ago I wrote a recursive deletion program (DELTREE.COM) for SpartaDOS, together with TREE.COM, which displays a graphical representation of the directory structure. Happily, these are both written in CC65 and should be very easy to adapt for MyDOS. MYCOPY, DELTREE and TREE should make a nice trio of programs for MyDOS. I'll post the C source as soon as I'm able: someone might be able to adapt DELTREE into another recursive file copier...

Edited by flashjazzcat, Wed Feb 18, 2009 2:07 PM.


#42 danwinslow OFFLINE  

danwinslow

    River Patroller

  • 2,593 posts

Posted Wed Feb 18, 2009 2:14 PM

Nice! Now get back to work on that uber word processor! :)

Yes, post them if you can, I will add them into the CC65AtariLib at sourceforge.

Edited by danwinslow, Wed Feb 18, 2009 2:16 PM.


#43 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 14,603 posts
  • Location:United Kingdom

Posted Wed Feb 18, 2009 2:40 PM

Yes, post them if you can, I will add them into the CC65AtariLib at sourceforge.

Will do... I'm just looking through this stuff now and it's SO much more readable than my assembler coding! :) This is the DELTREE program, which might also form the basis for a nice recursive copier. The listing requires the use of the SpartaDOS CC65 library (to follow soon), but you can see how PC-like the calls are.

// DELTREE v 1.0

#include <stdio.h>
#include <dos.h>

#define VER "1.0"


char confirm,c;
char *cat,*cat3,*all;

int arg,i,count;

main(argc,argv)
 int argc;
 char *argv[];
$(

 char fspec[_MAX_PATH],temp[_MAX_PATH];

 cat="%s%s";
 cat3="%s%s%s";
 all=">*.*";
 arg=1;
 if(argc>1) $(
  if(argv[1][3]!='/') strcpy(fspec,argv[arg++]);
  if(arg<argc) $(
   if((argv[arg][3])=='/') $(
	switch(toupper(argv[arg][4])) $(
	 case 'N': confirm=1; break;
	 default: usage();
	$)
   $)
   else usage();
  $)
 $)
 else usage();
 i=strlen(fspec);
 if(i<4) usage();
 if(((fspec[3]=='\\') || (fspec[3]=='>')) && (fspec[4]==0)) $(
  printf("Erase entire directory tree");
  if(!y_n()) exit();
 $)
 getcwd(fspec,fspec+3);
 scan(fspec);
 if(strlen(fspec)>3) $(
  deldir(fspec);
  count++;
 $)
 if(count==0) printf("\nNo matching entries\n");
$)


scan(dir_path)
 char *dir_path;
$(
 struct find_t dta;
 char new_spec[_MAX_PATH];
 char del_path[_MAX_PATH];
 int done,j;

 sprintf(new_spec,cat,dir_path,all);
 done=_findfirst(new_spec,_A_SUB,&dta);
 while(!done) $(
  count++;
  sprintf(del_path,cat3,dir_path,">",dta._name);
  scan(del_path);
  deldir(del_path);
  done=_findnext(&dta);
 $)

 sprintf(new_spec,cat,dir_path,all);
 done=_findfirst(new_spec,_A_NSUB,&dta);
 while(!done) $(
  count++;
  sprintf(del_path,cat3,dir_path,">",dta._name);
  if(!confirm) $(
   printf("Delete %s",del_path);
   j=y_or_n();
  $)
  else j=1;
  if(j==1) $(
   printf("Deleting %s\n",del_path);
   remove(del_path);
  $)
  done=_findnext(&dta);
 $)
$)


y_n() $(
 int flag;
 printf("? [y,n]:");
 flag=tolower(kbdchar());
 if(flag=='y') $(
  printf("y\n");
  return(1);
 $)
 else $(
  printf("n\n");
  return(0);
 $)
$)
;
y_or_n() $(
 int flag;
 printf("? [y,n,q]:");
 flag=tolower(kbdchar());
 if(flag=='q') $(
  printf("q\n");
  exit();
 $)
 if(flag=='y') $(
  printf("y\n");
  return(1);
 $)
 else $(
  printf("n\n");
  return(0);
 $)
$)


deldir(path)
 char *path;
$(
 int s,y_n;
 if(!confirm) $(
  printf("Remove dir %s",path);
  y_n=y_or_n();
 $)
 else y_n=1;
 if(y_n==1) $(
  printf("Removing %s\n",path);
  s=rmdir(path);
  if(s==-1) $(
   printf("\nCan't delete directory!\n");
   exit();
  $)
 $)
 return(y_n);
$)


usage() $(
 printf("\nUsage: DELTREE [Dn:]path [/N]\n\n");
 printf("  N : Don't verify deletions\n");
 exit();
$)

...if you're wondering about the "//" style comment at the top of the program, I patched my copy of CC65 to allow that kind of comment... don't ask! :D

Edited by flashjazzcat, Wed Feb 18, 2009 2:43 PM.


#44 danwinslow OFFLINE  

danwinslow

    River Patroller

  • 2,593 posts

Posted Wed Feb 18, 2009 2:55 PM

Uh, well cc65 allows that (// comments) anyways...and I notice you are using K&R formatting and I don't know what the $( dimorph is. Yknow, you may be talking about the old Atari compiler cc65..? I am talking about the the new one at www.cc65.org.

#45 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 14,603 posts
  • Location:United Kingdom

Posted Wed Feb 18, 2009 3:06 PM

Uh, well cc65 allows that (// comments) anyways...and I notice you are using K&R formatting and I don't know what the $( dimorph is. Yknow, you may be talking about the old Atari compiler cc65..? I am talking about the the new one at www.cc65.org.

Yes - definitely talking at cross-purposes here. Sorry! The above listing is designed for the "old Atari" CC65. I haven't even checked the new one out, although clearly I need to... The $( / $) dimporphs were substitutes for the opening and closing curly brace, which the Atari8 doesn't have. The Atari CC65 didn't support // style comments.

How will the new CC65 take to my old K&R code? :D

#46 danwinslow OFFLINE  

danwinslow

    River Patroller

  • 2,593 posts

Posted Wed Feb 18, 2009 3:22 PM

Oh man, you are in for a treat. The new CC65 is really great...I dunno about the K&R, I think ANSI is supposed to handle it but even if not its easy enough to change, and the $( to /* and so forth is easy enough too.

#47 a8maestro OFFLINE  

a8maestro

    Chopper Commander

  • 162 posts
  • Location:Austin, TX

Posted Wed Feb 18, 2009 9:04 PM

snip....

Final note: Found a program named "Folder-Copy" by Rick Detlefsen on his homepage, the original program was written in Atari Basic, but he also created a compiled (*.COM) version with the MMG compiler - maybe this is just what you need... I did not try this program...

greetings, Andreas Koch.

P.S.: The program Folder-Copy is on the image HDtools2.ATR... (SpartaDOS format, so you may have to use Bewe or Sparta DOS and the menu program to copy it back onto a DOS 2.x or MyDOS disk)...


My folder copy needs work. And a more drag an drop setup. Started it, but holidays got in the way.
Rick D.

#48 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 14,603 posts
  • Location:United Kingdom

Posted Thu Feb 19, 2009 4:27 AM

Oh man, you are in for a treat. The new CC65 is really great...I dunno about the K&R, I think ANSI is supposed to handle it but even if not its easy enough to change, and the $( to /* and so forth is easy enough too.

I think you're right; I've just started reading through the CC65 docs and it looks superb. The old Atari CC65 was just about my ideal language (especially for command line utilities), but it was SO slow. Just imagine cross-compiling in the blink of an eye! Hopefully the SpartaDOS library should port across without too much trouble; most of it is written in assembler (RA65). I was quite excited to discover last night that I've even written a working memory management library will calloc/malloc/realloc/free functions and the like. :)

My folder copy needs work. And a more drag an drop setup. Started it, but holidays got in the way.


I had a browse through the plethora of copiers posted early in this topic and there are some great menu-based efforts. For a non-command line DOS, a menu-based program is surely a great way to go. I imagine something like the SDX MENU program but where you can tag directory entries as well as groups of files. You could either allow tagging of folders in the directory tree area, of simply list directory entries in the file window along with regular files. I think that would be a useful enhancement for MENU.COM natively, too. I always fancied writing something along the lines of PC-Shell for the Atari 8 (like SDX's MENU.COM but with a built-in text editor, sector editor, etc). Time to get back to LW for now, though. :D

If anyone has any ideas on the logic for doing the validation checks for my recursive copier I mentioned a few posts back, please let me know. The same goes for bugs. I might as well finish it; it's 90% done.

#49 _The Doctor__ ONLINE  

_The Doctor__

    Flux Capacitor Master Craftsman

  • 6,948 posts
  • Location:10-0-11-00:02

Posted Thu Feb 19, 2009 5:01 PM

can we have ape directory change support where it hits the directory twice to get ape to switch the directory. where you can set that double request mode for just that drive.

#50 Shawn Jefferson OFFLINE  

Shawn Jefferson

    Stargunner

  • Topic Starter
  • 1,987 posts
  • Location:Victoria, Canada

Posted Thu Feb 19, 2009 10:18 PM

I was quite excited to discover last night that I've even written a working memory management library will calloc/malloc/realloc/free functions and the like. :)


cc65 includes heap memory allocation functions already. They are sort of "expensive", like the formatted output (printf type functions) though.




0 user(s) are browsing this forum

0 members, 0 guests, 0 anonymous users