Just wanted to inform that during early October 2018 we will start testing out automation scripts (rexx/bat/tools) to try and process our first batch of files we have now completed scanning, decrunching and organizing for!
The first isolated batch of data consists of files we extracted out of ADF/DMS/ADZ disk images and which where detected as NON-DOS (more or less). We used "adfchk v0.2.2 by A. Eibach - 03 Dec 2009" to scan our ADF/ADZ/DMS images first, then logging the NON-DOS disks (which would also include special disks with maybe boot menu, but had a BAM, or not), it worked pretty well and at least we skipped alot of standard DOS disks this way (remember, all files on DOS disks are processed otherwise by my other tools, so nothing gets lost).
We scanned 372541 disks marked as such (out of my SOTDS database of 31 million files). Out of these disk images, we came across and extracted/decrunched by using "xfdScan" and "xfdDecrunch" a total of 776016 files. The "xfdScan" scanned the Disk Image as-is, and extracted out numbered files. After that we run a "xfdDecrunch" on those files and we ended up with a batch of files.
Update 10 October 2018
After putting Pro-Wizard4PC to the batch test, we found out it produces alot of junk files, mostly due to the fact it also can detect crunched material (even broken so) and produces kind-of-fuzzy-matching to indicate it found a "mod" module for instance when in reality the file was pure and simple junk!
So we had to implement some junk removal routines on the results produced by Pro-Wizard4PC. It basically was a simple job, as we do a character count of the entire file it extracted and if its below 100 chars in variations, chances are the file is not a proper module/music file most likely, and we managed to remove alot of junk this way. Hopefully it will work and we don't loose any music tracks, but even then, the 3 other ripping routines we gonna perform might pick it up again anyway :-)
Update 07 October 2018
Been running 8 x instances of WinUAE on the same PC for some days, and its very stable and steadily rips out tunes. We have processed around 75000 files already, and found close to at least 2500 songs so far. In future we would probably go much higher with several PC's of something like 20-40 instances of WinUAE doing its job :-) - Check the updated video below for processing porn!
I just love to see lots of windows of processing/flickering windows - just like any cool 80's sci-fi, hacker, cyberspace movie, hahaha!
Additionally, the Pro-Wizard4PC automation is under testing and seems to do its job, see 2nd image below for more information!
Update 04 October 2018
Experimenting with 6 x instances of WinUAE in a manageable resolution at 400x300 windows. See screenshot below!
Update 03 October 2018
After some tweaking of scripts and batch files we can confirm our first batch of 1015 directories with 750 files in each are currently scanning and ripping out tunes using the "EaglePlayer w/Eagleripper" approach!
So far we scanned 750*3 files and ripped out a little over 100 tunes, mostly M.K. (Soundtrakcer/Protracker) files!
It is at this point clear we need a massive firepower of multiple instances of WinUAE running 24/7 to make this as fast as possible, because the first batch of 776016 files is _NOTHING_ compared to the whole amount of files which are most likely around 31 million files * 4 instances of ripper techniques = 120 million files to be in total processed!
Actually we are still extracting and decrunching files on our first server which have been running since April 2018. A couple of weeks ago we set up a 2nd server, which are yet to be configured during October, but we expect full power blow horse proessing to initiate for full effect during October 2018. Watch this website and space, we're gonna break some records :-)
The video below shows WinUAE 4.01 in JIT mode + warp mode running my first test of automated batch ripping.
Play Video: 8 x Instances of WinUAE running my automated REXX/Batch Ripping
The song seems to be damaged, but fixable. And this is The Birth of ENDGAME fruits come to life, it just so happens it was called "logon", as this is what we really did right now. Logon onto the SOAMC= ENDGAME Matrix - the most insane ripping music project ever attempted by anybody, human OR aliens!
The module file was found on: (reference to my SOTDS database).
!Gigantic_Collection_9_DVD\Amiga - Applications\Digitech Mega Demo (19xx)(Digitech).adf
The tune and group was from Sweden, so hello neighbours!
My EaglePlayer with Eaglerippers batch/rexx magic in action!
Problem #1: Seems to detect fine and save found modules, allthough would stop if compressed (broken actually) files would show up.
Solution: Delete all XFD, XFR, Compressors, XPK related libs from the DH0:Libs as we do NOT want Eagleplayer try to decompress anything. Further, my batch auto generation tool had to avoid file headers of "XPKF", "IMP!", "PP20" and "PX20" to make automation go smooth.
It should be noted; at this point whatever was decrunchable was already performed, so any files acting to be compressed/crunched during this ripping operation is actually 100% broken/password protected files which will be avoided/ignored at all costs = nothing we can do about it :-)
Here's the "Pro-Wizard4PC" automation running with my own tool programmed in PureBasic for Windows. Not much to look at, but at least its automating Pro-Wizard4PC command line tool and captures the STDOUT of that program for analyze what was found and to rename the file output into our format.
We have decided to completely ignore any crunched datafiles that Pro-Wizard4PC will find unless its a Tracker Packer type of file = music. Those ignored files are most likely bad and false positives and most likely just byte matching junk that cannot be decompressed anyway!.
Remember, we have 3 other rippers running on the same file, so we are sure any possible music format would eventually be picked up by those.
Currently ignored are: rnc imploder powerpacker23 powerpacker3 bytekiller crunchmaniadata dragpack100 megacruncher powerpacker4 tnmcruncher11 crnd etc...etc.. We will not update this list here, list above is just an example of what type of files we will fully ignore.