WWWGrab is a web page data extraction and database generation tool, or "web scraper". It scans URL lists in a database, fetches the listed web pages and parses them with the DTBuild data transformation engine. WWWGrab can run sequences of URL scans and SQL database operations, allowing for multiple passes over data generated "on the fly" (at run time).
WWWGrab parsers are created with the DTBuild data transformation workshop. At run time WWWGrab gets a web page and sends it to the DTBuild engine, which transforms the web page with the specified parser.
WWWGrab is controlled by a list of tasks specified in a database. There are two types of task:
1. scan a URL list,
2. execute an SQL list.
The user can combine any number of URL scans and SQL executions in a task list. For example, a task list could:
* scan an initial list of URLs,
* generate a new list of URLs,
* modify the generated URL list with SQL,
* scan the generated+modified URL list,
* generate another URL list,
* etcetera.
The combined flexibility of WWWGrab and DTBuild enables a wide variety of web data transformation tasks. Consult DTBuild help for more information.
WWWGrab / DTBuild features:
* Recursive capabilities (enabling parsing of nested HTML/XML tags, comments, etc.)
* Wide-string (Unicode) input / output capability
* ODBC interface that displays database layout info (table and field names) to the user
* ODBC interface allowing construction of SQL statements with a combination of user-defined data and recognized data
* Trace mode to show correspondence between input and nodes (for debugging)
* User-defined function interface allowing execution of custom DLL code ...
Configuration assistance is available.