Scenario: Catching flow metrics from a Job in JETL or Talend

Catching flow metrics from a Job in JETL or Talend 

tMySQLInput : select * from film

tFlowMeter

 tMap

tFlowMeter

 tLogRow

 tFlowMeterCatcher

 tFileOutputDelimited

CSV output

Logging in JETL or Talend ETL

Logging in Talend ETL

Family : Logs & Errors

"Jaspersoft ETL"

Version: 5.5.1
Build id: r118616-20140618-0118

tStatCatcher
Operates as a log function triggered by the StatsCatcher Statistics check box
of individual components, and collects and transfers this log data to the output
defined.

tFlowMeter
The number of rows is then meant to be caught by the tFlowMeterCatcher
for logging purpose.

tFlowMeterCatcher
Operates as a log function triggered by the use of a tFlowMeter component
in the Job.

tLogCatcher:
Operates as a log function triggered by one of the three: Java
exception/PerlDie, tDie or tWarn, to collect and transfer log data.

tLogCatcher :  amclogs
CREATE TABLE `amclogs` (
   `moment` datetime DEFAULT NULL,
   `pid` varchar(20) DEFAULT NULL,
   `root_pid` varchar(20) DEFAULT NULL,
   `father_pid` varchar(20) DEFAULT NULL,
   `project` varchar(50) DEFAULT NULL,
   `job` varchar(255) DEFAULT NULL,
   `context` varchar(50) DEFAULT NULL,
   `priority` int(3) DEFAULT NULL,
   `type` varchar(255) DEFAULT NULL,
   `origin` varchar(255) DEFAULT NULL,
   `message` varchar(255) DEFAULT NULL,
   `code` int(3) DEFAULT NULL
 )

tFlowMeterCatcher: amcmeter
CREATE TABLE `amcmeter` (
   `moment` datetime DEFAULT NULL,
   `pid` varchar(20) DEFAULT NULL,
   `father_pid` varchar(20) DEFAULT NULL,
   `root_pid` varchar(20) DEFAULT NULL,
   `system_pid` bigint(8) DEFAULT NULL,
   `project` varchar(50) DEFAULT NULL,
   `job` varchar(50) DEFAULT NULL,
   `job_repository_id` varchar(255) DEFAULT NULL,
   `job_version` varchar(255) DEFAULT NULL,
   `context` varchar(50) DEFAULT NULL,
   `origin` varchar(255) DEFAULT NULL,
   `label` varchar(255) DEFAULT NULL,
   `count` int(3) DEFAULT NULL,
   `reference` int(3) DEFAULT NULL,
   `thresholds` varchar(255) DEFAULT NULL
 )

tStatCatcher: amcstats
CREATE TABLE `amcstats` (
   `moment` datetime DEFAULT NULL,
   `pid` varchar(20) DEFAULT NULL,
   `father_pid` varchar(20) DEFAULT NULL,
   `root_pid` varchar(20) DEFAULT NULL,
   `system_pid` bigint(8) DEFAULT NULL,
   `project` varchar(50) DEFAULT NULL,
   `job` varchar(50) DEFAULT NULL,
   `job_repository_id` varchar(255) DEFAULT NULL,
   `job_version` varchar(255) DEFAULT NULL,
   `context` varchar(50) DEFAULT NULL,
   `origin` varchar(255) DEFAULT NULL,
   `message_type` varchar(255) DEFAULT NULL,
   `message` varchar(255) DEFAULT NULL,
   `duration` bigint(8) DEFAULT NULL
 )

tFlowMeter: meter
CREATE TABLE `meter` (
   `moment` datetime DEFAULT NULL,
   `pid` varchar(20) DEFAULT NULL,
   `root_pid` varchar(20) DEFAULT NULL,
   `father_pid` varchar(20) DEFAULT NULL,
   `project` varchar(50) DEFAULT NULL,
   `job` varchar(255) DEFAULT NULL,
   `context` varchar(50) DEFAULT NULL,
   `priority` int(3) DEFAULT NULL,
   `type` varchar(255) DEFAULT NULL,
   `origin` varchar(255) DEFAULT NULL,
   `message` varchar(255) DEFAULT NULL,
   `code` int(3) DEFAULT NULL
 )

Difference between tFlowMeter and tFlowMeterCatcher
tFlowMeter : It counts the number of rows 
tFlowMeterCatcher : It catches the number of rows counted by tFlowMeter



-- flowmeter
# moment, pid, root_pid, father_pid, project, job, context, priority, type, origin, message, code
'2017-07-25 15:30:01', 'OuupyP', 'OuupyP', 'OuupyP', 'TALENDETLSCENARIOS', 'test1', 'Default', '4', 'tWarn', 'tWarn_1', 'this is a warning', '42'
'2017-07-25 15:30:01', 'OuupyP', 'OuupyP', 'OuupyP', 'TALENDETLSCENARIOS', 'test1', 'Default', '4', 'tWarn', 'tWarn_1', 'this is a warning', '42'

-- flowmetercatcher
# moment, pid, father_pid, root_pid, system_pid, project, job, job_repository_id, job_version, context, origin, label, count, reference, thresholds
'2017-07-25 15:30:01', 'OuupyP', 'OuupyP', 'OuupyP', '9880', 'TALENDETLSCENARIOS', 'test1', '_sUBX4HEXEeeLF_5AItd_KA', '0.1', 'Default', 'tFlowMeter_6', 'row7', '1', NULL, ''
'2017-07-25 15:30:01', 'OuupyP', 'OuupyP', 'OuupyP', '9880', 'TALENDETLSCENARIOS', 'test1', '_sUBX4HEXEeeLF_5AItd_KA', '0.1', 'Default', 'tFlowMeter_6', 'row7', '1', NULL, ''

Download : Click me


References: 
https://www.rilhia.com/tutorials/using-talend-job-create-amc-database-schema