Showing posts with label dsjob. Show all posts
Showing posts with label dsjob. Show all posts

Sunday, 19 November 2017

Get Job Parameters for all DataStage Job


Feels awesome while writing this post after 7 week, In this post I am going to share a script which will fetch the job parameters for all the datastage job available in given project.

This week, one of my colleague ask to compare all the jobs parameter from one project (older) to another project (migrated),  So, wrote down this script which worked like charm for me and hoping will work for you as well :-)





=


Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Friday, 14 July 2017

Utility to Get Last Run Log of DataStage Job


Love the new feature of IBM DataStage to fetch the last run log of any jobs with the help of "dsjob" command.
In older version of DataStage, it is very tedious to get the last run log but from/after v9.1 IBM added an additional feature in dsjob command. Lets see hows this works -


  • To Fetch Last Run Log:
  • To Fetch Second Last Run Log
  • To Fetch Third Last Run Log
  • To Fetch Nth Last Run Log





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Saturday, 1 April 2017

Get Over Running DataStage Job Details #iLoveScripting


With the script below, we will get a list of jobs which are taking more time to complete than last run time. By some tweaks, we can use this script to monitor any kind of process.






Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Wednesday, 7 December 2016

Import the jobs from DS windows client #iLoveScripting


As we have discussed a script which can export the datastage jobs from your client system (http://bit.ly/2frNPKj) likewise we can write another one to import the jobs. Let's see how -

DsImportJobsClient.bat :

This Script read all the *.dsx job name from the specified Directory and Sub-Directory and import to the Specified project. It can also build (Only BUILD) the existing package created on Information Server Manager and send it to the specified location on client machine.

To use the build feature you need to make sure the package has been created with all the needed jobs, saved and closed. Only update to the selected job will be taken care automatically. To add/delete a job, you need to do manually.

Modify the Import.properties and ImportJobList.txt file and Go the .bat dir and then execute the importAndBuild.bat.




Import.properties :


ImportJobList.txt :


DsImportJobsClient.bat :





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Sunday, 6 November 2016

Export the jobs from DS windows client



Datastage jobs Export/Import are occasional activity (Deployment time :-)) for a developer But it becomes very tedious if the job list are long or it's daily routine to export or import jobs.

So I have written a batch script (windows script) which we can execute from the Client Machine (where datastage clients are installed) and automate this process.



DSjobExportClient.bat :
Export Script read the job name from the file (ExportJobList.txt) and exports the jobName.dsx from the project to the export base location and maintain the folder structure specified in the ExportJobList.txt file. File "ExportJobList.txt" and “Export.properties” should be updated before running the export script.

Copy the above file on any location of your DS windows client machine and update the "ExportJobList.txt" and “Export.properties” files.
-    Export.properties
-    ExportJobList.txt
-    ExportJobs.bat   



Export.properties :


ExportJobList.txt :


DsExportJobsClient.bat :




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Tuesday, 15 March 2016

yet another Shell script to access DataStage job logs



Introduction


We have various data warehouses hosted on AIX/Linux/Unix operating systems. DataStage Director is one of the ETL monitoring & scheduling tool used in numerous data warehouses. In case of ETL job failures, we need to login to DataStage Director and check the log for error messages. The ETL job logs can tend to span into several pages with various types of informative messages. Usually we need to locate the error message under ‘Fatal’ message type. Doing this task by searching the DataStage log manually can be very time-consuming and tiring.

This shell script facilitates the user to access the error messages from the ease of the  AIX/Linux screen. The script also has a facility to email the filtered log messages to the user’s mailbox.


Tool Logic 


Accept jobname and other parameters while script execution.
Establish proper environment settings for local DataStage use
Locate event ids for fatal errors in the ETL joblog.
Extract detail log for fatal errors
Mail the filtered job log with the exact fatal error message to the user.


Usage 

getDSjob







Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Sunday, 17 January 2016

Count of Jobs - A Quick DataStage Recipe



What to Cook:
How to count number of jobs in DS project

Ingredients:
Use dsjobs "-ljobs" command



How to Cook:
Go to DS Administrator "projects" tab
Click on command button
Enter following command to execute:
SH -c "dsjob -ljobs <Project Name> | wc -l"

<Project Name> - Enter your project name






Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://groups.google.com/forum/#!forum/datagenx

Friday, 23 October 2015

authfile in dsjob command


While working on Datastage CLI commands, in some cases when you are working on server which is having multiple DataStage server install Or your user id is don't have sufficient permission to login on DataStage, we have to give DataStage server details like DataStage server host, username, password etc in the command line. But, this creates a small issue, anyone can see the username and password details which you have used in the command line by checking the server command history or other ways.

                           IBM Datastage provides a way to secure your username and password with authfile parameter. Today, we will see what it is and how we can use in our command line statement.



authfile:



How to use in commands:

dsjob -authfile c:\auth_file.txt -lprojects
dsjob -authfile c:\auth_file.txt -ljobs dstage1





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://groups.google.com/forum/#!forum/datagenx

Thursday, 8 October 2015

dsjob #3 - Managing Datastage Jobs from Command Line




0. Sourcing DSHOME in Windows or Linux
In Windows - Add the DSENGINE/bin path to Windows PATH

dsjob-path

In Linux - Add the DSENGINE/bin path to Linux PATH or source the dsenv file



1. Listing Project:
dsjob –lprojects         ---List of all projects
dsjob -authfile filename -lprojects  ---If needed username and password for dsjob

dsjob-listproject

2. Listing Jobs:
dsjob –ljobs project    ---List of all Jobs under a project
dstage1 - project name

dsjob-listjobs

3. Listing Stages:
dsjob –lstages [useid] project job|job_id --- List of all stages in a job
xfm7 -- jobname

dsjob-lstages

4. Listing Links:
dsjob –links [useid] project job|job_id stage  --- List of the links within a job design
xfm7 - jobname
Transformer_1 - stagename

dsjob-llinks

5. Listing Parameters:
dsjob –lparams [useid] project job|job_id  ---List of all parameters used within a Job
This job is not having any parameter so <none> in output.

dsjob-lparams

6. Displaying Job Information:
dsjob –jobinfo [useid] project job|job_id

dsjob-jobinfo

7. Displaying stage information
dsjob –stageinfo [useid] project job|job_id

dsjob-stageinfo

8. Displaying Link Information
dsjob –linkinfo [useid] project job|job_id stage link

dsjob-linkinfo

9. Adding a Log Entry
dsjob –log [–info | –warn] [useid] project job|job_id

dsjob-log

10. Displaying a Short Log Entry
dsjob –logsum [–type type] [ –max n ] [useid] project job|job_id

dsjob-logsum

11. Generating a Report:
dsjob –report [useid] project job|job_id [report_type]

dsjob-report


You can find many more post like this HERE



Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://groups.google.com/forum/#!forum/datagenx

Tuesday, 6 October 2015

dsjob #2 - Managing Datastage Jobs from Command Line


Continued from dsjob - Managing Datastage Jobs from Command Line #1






Listing Projects, Jobs, Stages, Links, and Parameters :



We can list projects, jobs, stages, links, and job parameters using the

dsjob command. The different versions of the syntax are described in
the following sections.

The following syntax works for listing information about a Datastage job as below:

  •     dsjob –lprojects         ---List of all projects
  •     dsjob –ljobs project    ---List of all Jobs under a project
project is the name of the project containing the jobs to list.


  •     dsjob –lstages [useid] project job|job_id --- List of all stages in a job
1.    useid specify this if we intend to use a job alias (jobid) rather than a
2.    job name (job) to identify the job.
3.    project is the name of the project containing job.
4.    job is the name of the job containing the stages to list. To identify a
5.    job invocation, use the format job.invocation_id.
6.    job_id is an alias for the job that has been set using the dsjob -jobid
7.    command


  •     dsjob –links [useid] project job|job_id stage  --- List of the links within a job design
1.    useid specify this if we intend to use a job alias (jobid) rather than a
2.    job name (job) to identify the job.
3.    project is the name of the project containing job.
4.    job is the name of the job containing stage. To identify a job
5.    invocation, use the format job.invocation_id.
6.    job_id is an alias for the job that has been set using the dsjob -jobid
7.    stage is the name of the stage containing the links to list.


  •     dsjob –lparams [useid] project job|job_id  ---List of all parameters used within a Job



Setting an Alias for a Job:


The dsjob command can be used to specify own ID for a

DataStage job.Other commands can then use that alias to refer to the
job.

Syntax: dsjob –jobid [my_ID] project job


Explanations of the available options

  1. my_ID is the alias we want to set for the job. If we omit my_ID, the command will return the current alias for the specified job. An alias must be unique within the project, if the alias already exists an error message is displayed project is the name of the project containing job.
  2. job is the name of the job. To identify a job invocation, use the format job.invocation_id

    
Retrieving Information:

The dsjob command can be used to retrieve and display the available information about specific projects, jobs, stages, or links. The different

versions of the syntax are described in the following sections.

Displaying Job Information


Syntax: dsjob –jobinfo [useid] project job|job_id


Explanations of the available options:

  1. useid specify this if we intend to use a job alias (jobid) rather than a job name (job) to identify the job.
  2. project is the name of the project containing job.
  3. job is the name of the job. To identify a job invocation, use the format job.invocation_id.

The following information is displayed:
  1. The current status of the job
  2. The name of any controlling job for the job
  3. The date and time when the job started
  4. The wave number of the last or current run (internal DataStage reference number)
  5. User status

Displaying Stage Information:

Syntax: dsjob –stageinfo [useid] project job|job_id stage


Explanations of the available options:

  1. useid specify this if we intend to use a job alias (jobid) rather than a
  2. job name (job) to identify the job.
  3. project is the name of the project containing job.
  4. job is the name of the job containing stage. To identify a job
  5. invocation, use the format job.invocation_id.
  6. job_id is an alias for the job that has been set using the dsjob -jobid command.
  7. stage is the name of the stage.

The following information is displayed:
  1. The last error message reported from any link to or from the stage
  2. The stage type name, for example, Transformer or Aggregator
  3. The primary links input row number

Displaying Link Information:

Syntax: dsjob –linkinfo [useid] project job|job_id stage link


Explanations of the available options:

  1. useid specify this if we intend to use a job alias (jobid) rather than a
  2. job name (job) to identify the job.
  3. project is the name of the project containing job.
  4. job is the name of the job containing stage. To identify a job
  5. invocation, use the format job.invocation_id.
  6. job_id is an alias for the job that has been set using the dsjob -jobid command. 
  7. stage is the name of the stage containing link.
  8. link is the name of the stage.

The following information is displayed:
  1. The last error message reported by the link
  2. The number of rows that have passed down a link


Accessing Log Files:



Adding a Log Entry

Syntax: dsjob –log [–info | –warn] [useid] project job|job_id



Explanations of the available options:
  1. info specifies an information message. This is the default if no log entry type is specified.
  2. -warn specifies a warning message.
  3. useid specify this if we intend to use a job alias (jobid) rather than a
  4. job name (job) to identify the job.
  5. project is the name of the project containing job.
  6. job is the name of the job that the log entry refers to. To identify a job invocation, use the format job.invocation_id.
  7. job_id is an alias for the job that has been set using the dsjob -jobid command.

Displaying a Short Log Entry

Syntax: dsjob –logsum [–type type] [ –max n ] [useid] project job|job_id


type specifies the type of log entry to retrieve.

If –type is not specified, all the entries are retrieved.


Importing Job Executables:

The dsjob command can be used to import job executables from a

DSX file into a specified project.

Syntax: dsjob –import project DSXfilename [-OVERWRITE] [-JOB[S] jobname …] |[-LIST]



  1.  DSXfilename is the DSX file containing the job executables.
  2. -OVERWRITE specifies that any existing jobs in the project with the same name will be overwritten.
  3. -JOB[S] jobname specifies that one or more named job executables should be imported (otherwise all the executable in the DSX file are imported).
  4. -LIST causes DataStage to list the executables in a DSX file rather than import them.

Generating a Report:

The dsjob command can be used to generate an XML format report

containing job, stage, and link information.

Syntax: dsjob –report [useid] project job|job_id [report_type]


Report_type is one of the following:


 BASIC – Text string containing start/end time, time elapsed and status of job.

 DETAIL – As basic report, but also contains information about individual stages and links within the job.
 LIST – Text string containing



Examples are available HERE





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://groups.google.com/forum/#!forum/datagenx

Sunday, 4 October 2015

dsjob #1- Managing Datastage Jobs from Command Line



Datastage Command Line Interface:

The DataStage CLI comprises three groups of commands, those concerned with running DataStage Jobs, those concerned with administering DataStage projects, and those concerned with searching jobs.



dsjob command:
 
The dsjob command can be used from the UNIX command line to execute of the following tasks, All output from the dsjob command is in plain text without column headings on lists, or any other sort of description. This enables the command to be used in shell or batch scripts without extra processing.


These options are described in the following topics:


  •      The logon clause
  •      Starting a job
  •      Stopping a job
  •      Listing projects, jobs, stages, links, and parameters
  •      Setting an alias for a job
  •      Retrieving information
  •      Accessing log files
  •      Importing job executables
  •      Generating a report

Logon clause:

By default, the DataStage CLI connects to the DataStage server engine on the local system using the user name and password of the user invoking the command. We can specify a different server, user name, or password using the logon clause.

Syntax :

[–server servername][–user username][–password password]

servername specifies a different server to log on to.
username specifies a different user name to use when logging on.
password specifies a different password to use when logging on.

We can use the logon clause with any dsjob command.

Starting a Job:

We can start, stop, validate, and reset Datastage jobs using the –run option from the UNIX command line

Syntax :

dsjob –run
[ –mode [ NORMAL | RESET | VALIDATE ] ]
[ –param name=value ]
[ –warn n ]
[ –rows n ]
[ –wait ]
[ –stop ]
[ –jobstatus]
[–userstatus]
[–local]
[–opmetadata [TRUE | FALSE]]
[-disableprjhandler]
 [-disablejobhandler]
[useid] project job| job_id




Explanations of the available options

mode specifies the type of job run. NORMAL starts a job run, RESET resets the job and VALIDATE validates the job. If –mode is not specified, a normal job run is started.

param specifies a parameter value to pass to the job. The value is in the format name=value, where name is the parameter name, and value is the value to be set. If we use this to pass a value of an environment variable for a job (as we may do for parallel jobs), we need to quote the environment variable and its value, for example -param '$APT_CONFIG_FILE=sayan.apt' otherwise the current value of the environment variable will be used.

warn n sets warning limits to the value specified by n

rows n sets row limits to the value specified by n.

wait waits for the job to complete.

stop terminates a running job.

jobstatus waits for the job to complete, then returns an exit code
derived from the job status.

userstatus waits for the job to complete, then returns an exit code derived from the user status if that status is defined.
-local use this when running a DataStage job from within a shellscript on a UNIX server.

-opmetadata use this to have the job generate operational meta data as it runs.  If we specify FALSE, the job will not generate operational meta data.

-disableprjhandler use this to disable any error message handler that has been set on a project wide basis.

-disablejobhandler use this to disable any error message handler that has been set for this job.

-useid specify this if we intend to use a job alias (jobid) rather than a job name (job) to identify the job.

-project is the name of the project containing the job. job is the name of the job. To run a job invocation, use the format job.invocation_id.

job_id is an alias for the job that has been set using the dsjob -jobid command.


Stopping a Job:

We can stop a Datastage job using the –stop option from UNIX command prompt

Syntax:
dsjob –stop [useid] project job| job_id


 

Next part is available HERE




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://groups.google.com/forum/#!forum/datagenx