DataGenX - Atul's Blog: Director

Showing posts with label Director. Show all posts

Tuesday, 12 April 2016

Export DataStage job log from Director Client

by Atul Singh / in Client, Datastage, Director, export, Jobs, Log, tips /

These steps are not for exporting log for more than 3-5 jobs as these are manual steps and need to repeat for each job (Yes, each job individually) and NO CUSTOMIZATION available in export.

** If want to customized export, use dsjob command in your script.

1. Open the job log view for one of the jobs in Director Client.
2. Choose Project > Print All entries > Full details > Print to file, and then enter a file name in which you want to save the log.

Like the below page to get update
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

yet another Shell script to access DataStage job logs

by Atul Singh / in Datastage, Director, dsjob, Jobs, Log /

Introduction

We have various data warehouses hosted on AIX/Linux/Unix operating systems. DataStage Director is one of the ETL monitoring & scheduling tool used in numerous data warehouses. In case of ETL job failures, we need to login to DataStage Director and check the log for error messages. The ETL job logs can tend to span into several pages with various types of informative messages. Usually we need to locate the error message under ‘Fatal’ message type. Doing this task by searching the DataStage log manually can be very time-consuming and tiring.

This shell script facilitates the user to access the error messages from the ease of the AIX/Linux screen. The script also has a facility to email the filtered log messages to the user’s mailbox.

Tool Logic

 Accept jobname and other parameters while script execution.
 Establish proper environment settings for local DataStage use
 Locate event ids for fatal errors in the ETL joblog.
 Extract detail log for fatal errors
 Mail the filtered job log with the exact fatal error message to the user.

Usage

getDSjob

5 Tips For Better DataStage Design #4

by Atul Singh / in #5, Client, conversion, data, Datastage, design, designer, Director, Integer, Log, Parameter, Pointer, tips, tricks, type /

1) While using AsInteger() function in datastage transformer always trim the imput column before passing it to function because if there are extra spaces or unwanted characters which generates zeros when actual integer values are expected. We should use APT_STRING_PADCHAR=0x20 (space) env var for fixed field padding.

2) The Len(col) will return the incorrect length if the input column is having some non-ASCII or double byte characters in it. So check your NLS settings for the job to fix this.

3) To remove embedded spaces from decimal data, use StripWhiteSpace(input.field) function to remove all spaces.

4) To get the datastage job no, Open the log view of the job in datastage director and double click on any entry of the log. The job number will be listed under the field "Job Number:"

5) Set these 2 parameters APT_NO_PART_INSERTION, APT_NO_SORT_INSERTION to TRUE to avoid datastage to insert partitioning or sorting method to improve the job performance at compile job. This will remove the warning also "When checking operator: User inserted sort "<name>" does not fulfill the sort requirements of the downstream operator "<name>""

Behavior of Multi-Instance job in Director Client

by Atul Singh / in Client, Concept, Datastage, Director, Id, Instance, Invocation, Job, Log, Monitor, Multi, Purge /

Multi-Instance Job:
DataStage is supporting Multi-Instance jobs which can be run at same time with different invocation id.
Today, we will discuss the behavior of multi-instance datastage job in Director.

Running Jobs:
When we run the multi-instance job, it will ask for Invocation Id to be passed, when the job is running, it will display a new job in director in format <JOB_Name>.<Invok_Id>. Nothing change in original job, it is still in compiled status. So, if we invoke job 3 times with 3 invocation id, it will generate 3 jobs in director -

Jobname.InvkId1
Jobname.InvkId3
Jobname.InvkId3

Monitoring Jobs:
We can monitor each invoked job as it is been generated and visible in the Director with invocation id. But the tool is using the same RT_LOGnn file to write the job log for all invocation id, So we can see the n instance in director and its log in the Director but in backend, it is a single file. We can monitor, stop and check individual job log.

Deleting Jobs:
If we delete a job instance from director, it will be deleted and other instances will remain there But the job log for this instance is still with RT_LOGnn file ( we can access through from Datastage Command Line but not in Datastage Director as job instance has been deleted).

Purging Job logs:
If we purge the job log in datastage, it will delete the job instances as well as job logs from RT_LOGnn file. So the difference here is that the Director delete action only deletes records from RT_STATUS whereas the purging mechanism deletes records from RT_LOG.

DataGenX - Atul's Blog