WatchDirectory home page
WatchDirectory Startseite (Deutsche Version)
Site WatchDirectory (Français)
  Welcome, Guest. Please Login or Register
YaBB - Yet another Bulletin Board
   
  HomeHelpSearchLoginRegister  
 
Page Index Toggle Pages: 1
Howto.. free space event (Read 8856 times)
jesper
YaBB Newbies
*
Offline


www.gdps.dk/pro ducts/watchDir/
GdPUtil.shtml

Posts: 17
Copenhagen
Howto.. free space event
Oct 17th, 2004 at 4:32pm
 
I am looking for ways to automate freeing space, for example
by starting a batch job to remove oldest files. I can not
get WatchDirectory to do this for me, but the ideas of
WatchDirectory seems close to it.. i am hoping either there
is a way i didnt find, or the feature could be added to an
upcoming release.

An event like "Free space below X Mb", possibly with a second
command event for "Free space below Y Mb". With Y << X the
second command should only happen if the first action failed
to free space and could in that case be sending an alert
e-mail requiring manual intervention to check why the first
command did not succeed. This setup could help start
different automatic jobs for freeing space, saving the
administrators time, and at the same time give a good
security against a disk running out of space.

I was thinking perhaps Windows has a way to remove "the oldest file"
in a directory, with a .exe file in /system32/ that i dont know
of Smiley. I stumbled across "Forty Ways To Free Up Disk Space"
and was surprised not to see a suggestion to "delete the oldest"
files and dirs:
http://www.microsoft.com/technet/prodtechnol/winntas/tips/techrep/freedisk.mspx

I would be using this auto free space feature to utilize
a diskdrive for as many web logfiles or backup archives available
as possible, always removing the oldest container first.

There are probably other software that can do this, for example below which has a nice "look to
the future" feature of what will be "deleted in X days". However i am looking for the all-in-one app like WatchDirectory, hoping it can solve the issue.
Anyway that app i am talking about has no demo and it is incredible expensive for a server:
http://www.virtualsoftware.com/ProdPage.cfm?ProdID=2812&Automatic-File-Deletion-...

I really hope WatchDirectory can do the job somehow Smiley
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2238
The Netherlands
Re: Howto.. free space event
Reply #1 - Oct 17th, 2004 at 6:44pm
 
At the moment, WD doesn't support events like that.

However, it should be possible to implement something like you describe...

If you let WD monitor the root of your drive using the FAST mode, and only fire events for FILENEW and FILECHNG (because those alter the diskspace availlable), you could test for available diskspace inside your .bat file.

I would be concerned about CPU usage here....

Perhaps you should just schedule a .bat file using the AT command (or the scheduler) to run every 30 minutes. Inside it check for diskspace and take action as needed.

Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
jesper
YaBB Newbies
*
Offline


www.gdps.dk/pro ducts/watchDir/
GdPUtil.shtml

Posts: 17
Copenhagen
Re: Howto.. free space event
Reply #2 - Oct 17th, 2004 at 7:28pm
 
Your suggestions noted thank you. But I lack a way to take action, in order to remove the oldest files. A .bat file or modified gdpUtil.exe Smiley

I was even hoping for a few checks on the action to perform before deleting the oldest file:

A. "Files to delete must be at least Z days old" so action doesnt remove brand new logfiles.

B. Always leave X files for a particular match, finding similar matched filenames by filtering out the date stamps of filenames [1]. Example of similar filenames:
abc-20040917-1318.zip timestamp: 2004-09-17 13:18
abc-20040817-1301.zip timestamp: 2004-08-17 13:01
abc-20040717-1312.zip timestamp: 2004-07-17 13:12

The oldest file would only be removed if X is < 3 and Z is < 60. This would avoid removing the last X good backups of a particular filename even if that file is in fact the oldest.

[1] I dont know how much CPU this matching would take.

A reporting feature of the events that triggered the actions  would be handy.. especially if rule A or B was to take effect, because that probably means the backup routine creating the zips are not running properly.

Phew.. I am full of ideas Smiley

Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2238
The Netherlands
Re: Howto.. free space event
Reply #3 - Oct 18th, 2004 at 12:04am
 
Thinking about this a bit more....
Checking for free disk space for a drive is not that expensive. My original concern is only valid when you check for space used at the directory level.

What I could do:
Add an option to gdputil (or watchDirectory) that will report the space available. In you watchDirectory .bat file you could take appropriate actions based on that...

As an example (note: this is not implemented yet):

Code:
GdPUtil -checkspace c: 100
if %errorlevel% == 0 goto :FreeDiskSpaceLessThan100MB
if %errorlevel% == 1 goto :FreeDiskSpaceMoreThan100MB
 


The above code would hardly cost any cpu (except for starting the .bat and gdputil).
Below the :FreeDiskSpaceLessThan100MB label, you would cleanup diskspace using the -modifiedolderthan switch of gdputil:
Code:
:FreeDiskSpaceLessThan100MB
GdPUtil -modifiedolderthan 14 c:\directory ...other switches...
GdPUtil -checkspace c: 100
if %errorlevel% == 0 goto :StillNotEnoughDiskSpace
GOTO :Quit
:StillNotEnoughDiskSpace
Call c:\WDBatFiles\wdAutoMail.bat
GOTO :Quit
 



does the above make sense?
Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
jesper
YaBB Newbies
*
Offline


www.gdps.dk/pro ducts/watchDir/
GdPUtil.shtml

Posts: 17
Copenhagen
Re: Howto.. free space event
Reply #4 - Oct 18th, 2004 at 1:02am
 
The -checkspace c 100 would be useful, but is still missing the action of "-deleteoldest" instead of "-deleteolderthan" that i have not seen anywhere.  Deleting the oldest instead of olderthan, will maximise the number of eg. logfiles available on a certain amount of diskspace.

Modification to your example:
GdPUtil -modifiedolderthan 14 c:\directory ...other switches...
Changes to:
GdPUtil -deleteoldest d:\directory\ 100 -subdirs -test...other switches...

This should remove files until there is 100 Mb free on d:\logfiles\, and if that fails, still send an e-mail in your example.

Adding the A and B check above is less relevant.


Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2238
The Netherlands
Re: Howto.. free space event
Reply #5 - Oct 18th, 2004 at 1:12am
 
Quote:
The -checkspace c 100 would be useful, but is still missing the action of "-deleteoldest" instead of "-deleteolderthan" that i have not seen anywhere.

I have seen it somewhere... but where.
maybe if I just start searching at www.a.com, then www.b.com.....
Or would it be faster to code it myself??

Quote:
Deleting the oldest instead of olderthan, will maximise the number of eg. logfiles available on a certain amount of diskspace.
....
This should remove files until there is 100 Mb free on d:\logfiles\, and if that fails, still send an e-mail in your example.

Yes, I see your point. I strongly advise you not to hold your breath this time, it will be implemented, but it could take a few days.

Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
jesper
YaBB Newbies
*
Offline


www.gdps.dk/pro ducts/watchDir/
GdPUtil.shtml

Posts: 17
Copenhagen
Re: Howto.. free space event
Reply #6 - Oct 18th, 2004 at 2:24am
 
Smiley

Alright if you make this, I hope i am not the only one to find it useful, thanks again for your help Gert!!
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2238
The Netherlands
Re: Howto.. free space event
Reply #7 - Oct 18th, 2004 at 8:43pm
 
Quote:
Deleting the oldest instead of olderthan, will maximise the number of eg. logfiles available on a certain amount of diskspace.
...
Changes to:
GdPUtil -deleteoldest d:\directory\ 100 -subdirs -test...other switches...

This should remove files until there is 100 Mb free on d:\logfiles\, and if that fails, still send an e-mail in your example.


One question...
Should GdPUtil remove the oldest files across all subdirectories, or look at each directory and there delete the oldest files.
I mean, assuming all files are the same size. C:\Temp\Log1 could have 10 files that were all modified yesterday, and C:\Temp\Log2 could have 10 files that were all modified today.
I guess you do not want to empty Log1, but spread those deletes over both directories (so each directory has 5 files left).

Correct?

Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
jesper
YaBB Newbies
*
Offline


www.gdps.dk/pro ducts/watchDir/
GdPUtil.shtml

Posts: 17
Copenhagen
Re: Howto.. free space event
Reply #8 - Oct 19th, 2004 at 3:59am
 
That might be an issue for some uses, but not in my particular case. All logfiles or backups in each subdir are created or rotated around the same time every day. I dont have more files in each /subdir/ with same timestamp.

The -deleteoldest result I would see in my dirs would be deleting about one file from each /Log1/, /Log2/, /Log3/ and if there still isnt enough free space, it would start over in dir /Log1/ etc.

Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2238
The Netherlands
Re: Howto.. free space event
Reply #9 - Oct 19th, 2004 at 4:06am
 
Ok, I understand.

I'lll implement something tomorrow.

Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2238
The Netherlands
Re: Howto.. free space event
Reply #10 - Oct 19th, 2004 at 8:58pm
 
Jesper,

A new version of GdPUtil is uploaded.
It has 2 new options:
Quote:
  -checkspace driveletter megabytes
     Checks the drive for available space. If the drive still has at least
     <megabytes> megabytes space will set the exit-code to 1, otherwise 0.
     Example:
       GdPUtil -checkspace C 100
  -cleanup directory megabytes [filemask] [-subdirs] [-preventempty[9]] [-test]
     Removes files from <directory> until at least <megabytes> free space
     is available on disk.
     If not enough space can be freed, GdPUtil will exit with errorlevel 1.
     filemask: only remove files that match this mask (defaults to *.*)
     -subdirs: also remove files from subdirectories
     -preventempty: will never delete the last file from a directory
           you can optionally append a number to this switch to set the
           number of files that should remain in each directory.
     -test: do not delete files, just report the files that would be deleted
     Example:
       GdPUtil -cleanup c:\temp 100 *.log -subdirs -preventempty10
           this will remove the oldest log files from all directories below
           C:\temp until there is at least 100 megabytes free space.
           At least 10 log files will remain in each directory.


Some notes:
The oldest files (accross all directories) will be deleted.

I think it can get a bit slow on large directory trees with lots of files in them.

I am not sure if the errorlevel reporting is very accurate:
Windows reports to GdPUtil the 'real' file size (for example 4 bytes), while the file will occupy more diskspace than that (disk clusters).
So, perhaps you should first run "-cleanup", followed by "-checkspace".

Enjoy,
Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
jesper
YaBB Newbies
*
Offline


www.gdps.dk/pro ducts/watchDir/
GdPUtil.shtml

Posts: 17
Copenhagen
Re: Howto.. free space event
Reply #11 - Oct 21st, 2004 at 12:40am
 
Hi Gert, this looks very very promising, thanks!

I will try it on a dir structure and report back how it works out, thank you!
Back to top
 
 
IP Logged
 
jesper
YaBB Newbies
*
Offline


www.gdps.dk/pro ducts/watchDir/
GdPUtil.shtml

Posts: 17
Copenhagen
Re: Howto.. free space event
Reply #12 - Oct 21st, 2004 at 1:27am
 
Somehow i can only get it to work with gdpUtil.exe in same dir as the dir i am cleaning.

Also, i cant get it working with -subdirs, it seems the /subdir/ is missing in delete attemp that then fails, eg.:

These are the subdirs:
D:\test\freespace\subdir1\
D:\test\freespace\subdir2\


Running:
D:\test\freespace>gdpUtil.exe -cleanup d:\test\freespace 3100 -subdirs -test

Free space on drive is 2941MB.
** TEST MODE ** Files are not deleted.
Attempting to free 159 megabytes.
Deleted D:\test\freespace\20040927-050943-full-tine-subdir1-backup.1.dar (0 bytes freed)
...
Deleted D:\test\freespace\20040927-050943-full-tine-subdir2-backup.2.dar (0 bytes freed)
...
WARNING: could not reclaim enough diskspace!
** TEST MODE ** No files were deleted.
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2238
The Netherlands
Re: Howto.. free space event
Reply #13 - Oct 21st, 2004 at 3:08am
 
oops,

sorry about that, I found what was wrong (I stored the filename instead of the complete filepath at some point).

New version uploaded.

Hope this one works for you, as I'll be going to DK tomorrow (Carlsberg Porter, here I come!).

mvh,
Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
jesper
YaBB Newbies
*
Offline


www.gdps.dk/pro ducts/watchDir/
GdPUtil.shtml

Posts: 17
Copenhagen
Re: Howto.. free space event
Reply #14 - Oct 21st, 2004 at 4:29am
 
Okay, cool, it seems to work flawless now:

c:\bat\freespace>c:\bat\freespace\gdpUtil.exe -cleanup D:\test\freespace 2970 -subdirs -test
Free space on drive is 2936MB.
** TEST MODE ** Files are not deleted.
Attempting to free 34 megabytes.
Deleted D:\test\freespace\subdir1\20040927-050943-full-tine-subdir1-backup.1.dar (26214400 bytes freed)
Deleted D:\test\freespace\subdir2\20040927-050943-full-tine-subdir2-backup.1.dar (26214400 bytes freed)
** TEST MODE ** No files were deleted.

Thanks for all the help so far!

When i find some way to support your company i will! And i will buy you a few Carlsberg if you come by Copenhagen one day Smiley
Back to top
 
 
IP Logged
 
Page Index Toggle Pages: 1