1) What are the types of Parallelism? There are 2 types of Parallel Processing. They are: Pipeline Parallelism :
It is the ability for a downstream stage to begin processing a row as soon as an upstream stage has finished processing that row (rather than processing one row completely through the job before beginning the next row). In Parallel jobs, it is managed automatically. For example, consider a job (srcà Transformer à Tgt) running on a system having three processors: The source stage starts running on one processor, reads the data from the source and starts filling a pipeline with the read data. Simultaneously, the Transformer stage starts running on another processor, processes the data in the pipeline and starts filling another pipeline. At the same time, the target stage starts running on another processor, writes data to the target as soon as the data is available. Partitioning Partitionin g Parallelism:
Partitioning parallelism means that entire record set is partitioned into small sets and processed on different nodes. That is, several processors can run the same job simultaneously, each handling a separate subset of the total data. For example if there are 100 records, then if there are 4 logical nodes then each node would process 25 records each. This enhances the speed at which lo ading takes place.
2) What are partitioning methods available in PX? The Partitioning methods available in PX are: a) Auto: It chooses the best partitioning method depending on:
The mode of execution of the current stage and the preceding stage. The number of nodes available in the configuration file.
b) Round robin: Here, the first record goes to the first processing node, the second to the second processing node, and so on. This method is useful for resizing partitions of an input dataset that are not equal in size to approximately equal-sized partitions. Data Stage uses ‘Round robin’ when it partitions the data initially. c) Same: It implements the Partitioning method same as the one used by the preceding stage. The records stay on the same processing node; that is, data is not redistributed or repartitioned.
Same is considered as the fastest Partitioning method. Data Stage uses ‘Same’ when passing data between stages in a job. d) Random: It distributes the records randomly across all processing nodes and guarantees that each processing node receives approximately equal-sized partitions. e) Entire: It distributes the complete dataset as i nput to every instance of a stage on every processing node. It is mostly used with stages that create lookup tables for their input. f) Hash: It distributes all the records with identical key values to the same processing node so as to ensure that related records are in the same partition. This does not necessarily mean that the partitions will be equal in size.
When Hash Partitioning, hashing keys that create a large number of partitions should be selected. Reason: For example, if you hash partition a dataset based on a zip code field, where a large percentage of records are from one or two zip codes, it can lead to bottlenecks because some nodes are required to process m ore records than other nodes. g) Modulus: Partitioning is based on a key column modulo the number of partitions. The modulus partitioned assigns each record of an input dataset to a partition of its output dataset as determined by a specified key field in the input dataset. h) Range: It divides a dataset into approximately equal-sized partitions, each of which contains records with key columns within a specific range. It guarantees that all records with same partitioning key values are assigned to the same partition.
Note: In order to use a Range partitioned, a range map has to be made using the ‘Write range map’ stage. http://www.allwalkin.blogspot.com/http://www.allwalkin.blogspot.com/ i) DB2: Partitions an input dataset in the same way that DB2 would partition it. For example, if this method is used to partition an input dataset containing update information for an existing DB2 table, records are assigned to the processing node containing the corresponding DB2 record.
3) What is the difference between OLTP and OLAP?
OLTP systems contain normalized data where as OLAP systems contain de-normalized data.
OLTP stores current data where as OLAP stores current and history data for analysis.
The query retrieval is very fast in OLTP when compared to the OLAP systems because in OLTP all data is stored in one table and in OLAP data is stored in multiple tables. 4) What is the difference between star schema and snow-flake schema?
In star schema, dimension tables are denorm alized where as in snow flake schema, d imension tables are normalized.
5) Where we need partitioning (In processing or somewhere)
Partitioning is needed in processing. It means we need Partitioning where we have huge volumes of data to process.
6) If we use SAME partitioning in the first stage which partitioning method it will take?
DataStage uses ‘Round robin’ when it partitions the data initially.
7) If we check the preserve partitioning in one stage and if we don’t give any partitioning method (Auto) in the next stage which partition method it will use?
In this case, the partitioning method used by the preceding stage is used.
Preserve Partitioning indicates whether the stage wants to preserve the partitioning at the next stage of the job. Options in this tab are: Set – Sets the Preserve partitioning flag. Clear – Clears the preserve partitioning flag. Propagate – Sets the flag to Set or Clear depending on the option selected in the previous stage.
8)
Why we need datasets rather than sequential files?
A Sequential file as the source o r target needs to be repartitioned as it is(as name suggests) a
single sequential stream of data. A dataset can be saved across nodes using partitioning method selected, so it is always faster when we used as a source or target. The Data Set stage allows you to store data being operated on in a persistent form, which can then be used by other DataStage jobs. Data sets are operating system files, each referred to by a control file, which by convention has the suffix .ds. Using datasets wisely can be key to good performance in a set of linked jobs.
9)
Why we need sort stage other than sort-merge collective method and perform sort option in the stage in advanced properties?
Sort Stage is used to perform more complex sort operations which are not possible using stages Advanced tab properties. Many stages have an optional sort function via the partition tab. This means if you are partitioning your data in a stage you can define the sort at the same time. The sort stage is for use when you don't have any stage doing partitioning in your job but you still want to sort your data, or if you want to sort your data in descending order, or if you want to use one of the sort stage options such as "Allow Duplicates" or "Stable Sort". If you are processing very large volumes and need to sort you will find the sort stage is more flexible then the partition tab sort.
10)Why we need filter, copy and column export stages instead of transformer stage?
In parallel jobs we have specific stage types for performing specialized tasks. Filter, copy, column export stages are operator stages. These operators are the basic functional units o f an orchestrate application. The operators in your Orchestrate application pass data records from one operator to the next, in pipeline fashion. For example, the operators in an application step might start with an import operator, which reads data from a file and converts it to an Orchestrate data set. Subsequent operators in the sequence could perform various processing and analysis tasks. The processing power of Orchestrate derives largely from its ability to execute operators in parallel on multiple processing nodes. By default, Orchestrate operators execute on all processing nodes in your system. Orchestrate dynamically scales your application up or down in response to system configuration changes, without requiring you to modify your application. Thus using operator stages will increase the speed of data processing applications rather than using transformer stages.
11) Describe the types of Transformers used in DataStage PX for processing and uses? Difference:
A Basic transformer compiles in "Basic Language" whereas a Normal Transformer compiles in "C++". Basic transformer does not run on multiple nodes whereas a Normal Transformer can run on multiple nodes giving better performance. Basic transformer takes less time to compile than the Normal Transformer.
Usage:
A basic transformer should be used in Server Jobs. 12) What will you do in a situation where somebody wants to send you a file and use that file as an input or reference and then run job?
Use wait for file activity stage between job activity stages in job sequencer.
13) How did you handle an 'Aborted' sequencer?
By using check point information we can restart the sequence from failure. if u enabled the check point information reset the aborted job and run again.
14) What are Performance tunings you have done in your last project to increase the performance of slowly running jobs?
1. Using Dataset stage instead of sequential files wherever necessary. 2. Use Join stage instead of Lookup stage when the data is huge. 3. Use Operator stages like remove duplicate, Filter, and Copy etc instead of transformer stage. 4. Sort the data before sending to change capture stage or remove duplicate stage. 5. Key column should be hash partitioned and sorted before aggregate operation. 6. Filter unwanted records in beginning of the job flow itself.
15) What is Change Capture stage? Which execution mode would you use when you used for comparison of data?
The Change Capture stage takes two input data sets, denoted before and after, and outputs a single data set whose records represent the changes made to the before data set to obtain the after data set. The stage produces a change data set, whose table definition is transferred from the after data set’s table definition with the addition of one column: a change code with values encoding the four actions: insert, delete, copy, and edit. The preserve-partitioning flag is set on the change data set. The compare is based on a set of key columns, rows from the two data sets are assumed to be copies of one another if they have the same values in these key colum ns. You can also optionally specify change values. If two rows have identical key columns, you can compare the value columns in the rows to see if one is an edited copy of the other. The stage assumes that the incoming data is key-partitioned and sorted in ascending order. The columns the data is hashed on should be the key columns used for the data compare. You
can achieve the sorting and partitioning using the Sort stage or by using the built-in sorting and partitioning abilities of the Change Capture stage.
We can use both Sequential as well as parallel modes of execution fo r change capture stage.
16)What is Peek Stage? When do you use it?
The Peek stage is a Development/Debug stage. It can have a single input link and any number of output links. The Peek stage lets you print record column values either to the job log or to a separate output link as the stage copies records from its input data set to one or more output data sets, like the Head stage and the Tail stage. The Peek stage can be helpful for monitoring the progress of your application or to diagnose a bug in your application.
17)What is the difference between Join and Lookup? a. If reference table is having huge amount of data then we go for join where as if the reference table is having less amount of data then we go for lookup.
b. Join performs all 4 types of joins (inner join, left-outr join, right-outer join and full-outer join) where as lookup performs inner join and left-outer join only.
c. Join don’t have reject link where as lookup is having a reject link.
d. Join uses hash partition where as lookup use entire partition.
18) What is RCP? How it is implemented?
DataStage is flexible about Meta data. It can cope with the situation where Meta data isn’t fully defined. You can define part of your schema and specify that, if your job encounters extra columns that are not defined in the Meta data when it actually runs, it will adopt these extra columns and propagate them through the rest of the job. This is known as runtime column propagation (RCP). This can be enabled for a project via the DataStage Administrator, and set for individual links via the Outputs Page Columns tab for most stages or in the Outputs page General tab for
Transformer stages. You should always ensure that runtime column propagation is turned on. RCP is implemented through Schema File. The schema file is a plain text file contains a record (or row) definition.
19)What is row generator? When do you use it?
The Row Generator stage is a Development/Debug stage. It has no input links, and a single output link. The Row Generator stage produces a set of mock data fitting the specified metadata. This is useful where we want to test our job but have no real data available to process.
Row Generator is also useful when we want processing stages to execute at least once in absence of data from the source.
20)What are Stage Variables, Derivations and Constants?
Stage Variable - An intermediate processing variable that retains value during read and doesn’t pass the value into target column. Derivation - Expression that specifies value to be passed on to the target column. Constant – Conditions that are either true or false that specifies flow of data with a link. The order of execution is stage variables-> constraints->derivations 21) Explain the Types of Dimension Tables?
Conformed Dimension: If a dimension table is connected to more than one fact table, the granularity that is defined in the dimension table is common across between the fact tables. Junk Dimension: The Dimension table, which contains only flags. Monster Dimension: If rapidly changes in Dimension are known as Monster Dimension. De-generative Dimension: It is line item-oriented fact table design.
22) What is the significance of surrogate key in DataStage?
Surrogate key is a substitution for the natural primary key. It is just a unique identifier or number for each row that can be used for the primary key to the table. The only requirement for a surrogate primary key is that it is unique for each row in the table. It is useful because
the natural primary key can change and this makes updates more difficult. Surrogated keys are always integer or numeric.
23)
What is the difference between sparse lookup and normal lookup?
Sparse Lookup:
If the reference table is having more amounts of data than Primary table data, then better to go for sparse lookup Normal Lookup:
If the reference table is having less amount of data than primary table data , then better to go for normal In both cases reference tables should be entire partioned and primary table should be hash partioned.
24)
Can we capture the duplicates in Datastage? If yes, how to do that?
Yes. We can capture duplicates by using sort stage. In sort stage, we have a property called “CreateKeyChangeColumn”. It will give 1 for original record and 0 for duplicates. Then by using transformer or Filter stage we can capture duplicates in one file and non-duplicates in another file.
