Databricks Secret Scopes

 https://hevodata.com/learn/databricks-secret/#51

Snowflake - Copy single file from Snowflake to S3

How to generate a single file in snowflake while copying data from Snowflake to S3/Blob/Etc.,

Add option single = true 

This option would avoid the 0_0_0 at the end of the file name.

Code snippet :

COPY INTO s3://<bucket>/test_data/file1.csv

FROM TESTDB..TEST1

FILE_FORMAT = (type = 'CSV')

SINGLE = TRUE

Reference:

https://docs.snowflake.com/en/sql-reference/sql/copy-into-location.html#unloading-data-to-a-single-file

DAX (Tabular Cube) : Top N Calculation

DAX (Tabular Cube) : Top N Calculation


https://www.youtube.com/watch?v=szYkXhjOWXY

Read Data from SharePoint List to SQL Server

I followed this blog for one of my use-case. It helpful.

http://whitepages.unlimitedviz.com/2014/03/using-the-odata-source-connector-with-sharepoint-online-authentication/

Run Excel from Windows Credential to Connect SSAS Cube

Option 1:

Create batch file using these below script

@echo off
SET /P username=Please enter the your username:  

runas /netonly /user:DAS\%username% "C:\Program Files (x86)\Microsoft Office\Office15\EXCEL.EXE"

Step 1- Copy the above script in notepad and modify as per your Domain name and Path of Excel exe.

Step 2- Save as batch file e.g "Run_Excel.bat"

Step 3 - now double click the batch file to run the excel and provide the name and password

Option 2:

Also you can run the script directly in Powershell

runas /netonly /user:DAS\Liyas "C:\Program Files (x86)\Microsoft Office\Office15\EXCEL.EXE"


by doing this you can run the excel in Windows credentials and connect the Analysis Service Cube.

Offline Reporting with SSAS Cube and Excel 2007 or Above

This link would help for Implement the offline reporting using Microsoft BI Capabilities. There are cases Sales Reps will be working in Remote locations where they would not have access to the Internet.
So Its a common requirement to access the data and create self service report with offline.

http://office.microsoft.com/en-in/excel-help/work-with-offline-cube-files-HP010087040.aspx

SSAS OLAP Cube - Role Based Data Security

In my case there was a Scenario to link the Store Dimension with Domain Users and provide the data security in Excel, PPS Dashboard for the users who has the access only to the Stores.

This article was quite helpful.

http://www.rdacorp.com/2009/01/advanced-dimension-data-security-with-sql-server-2008/

Here the Key thing is pass the Application Domain user name into Cube as follows..

NONEMPTY([Store].[StoreName].MEMBERS,
(STRTOMEMBER("[User].[User Name].&[" + USERNAME+ "]"), 
[Measures].[User Store Count]
))

Note: This script should be entered in Role -> Dimension Data(Cube Dimension not the OLAP Database Dimension) -> Advance tab.

In case you specify this script in Dimension Data tab under Cube Dimension then you will get this error.

"The browser is disabled because custom MDX expressions defined in the Advanced tab were not validated due to the following problem:

A connection cannot be made. Ensure that the server is running."

Create Dynamic Get storedprocedure for All the tables

--- This can be customized as per the Requirments ------ -- This Script used to create the Get(select all columns) Procedure for all the database tables. ---------------------------------------------------------------------------------------------------- SELECT ' CREATE PROCEDURE [USP_GET_' + TBL.NAME + ']' + + ' AS BEGIN SELECT ' + LEFT(COLUMNS,LEN(COLUMNS)- 1) + CHAR(10) + ' , GETDATE() AS INSERTED_DATE FROM ' + TBL.NAME + ' END GO ' FROM SYS.TABLES AS TBL CROSS APPLY ( SELECT UPPER('[' + COL.NAME + ']') + ' ,' FROM SYS.COLUMNS AS COL WHERE COL.OBJECT_ID = TBL.OBJECT_ID FOR XML PATH('')) AS TABLE_COLUMN(COLUMNS) ORDER BY TBL.NAME ASC

Sql Table variable Function for Handling Multiple Value Parameter in SSRS

Sql Table variable Function for Handling Multiple Value Parameter in SSRS

http://geekswithblogs.net/HighAltitudeCoder/archive/2010/06/06/sql-server-split-function.aspx

*** Just for self reference in future ****

Handle SSIS lookup failure on Date Dimension Date column NULL value

Handle Lookup failure when look up null value in Date dimension.

Problem:

Lookup fail when look up null value in Date dimension.

I have date Dimension table with one row for Null value. In my OLTP Order transaction data some of the date data type column has null value.

I am getting the lookup key not found error and those rows are redirected into reject table. Actually both null was not matched up and not able to found the NULL value dimension key.

Solution:

So in this blog I have explained the customized date look up script with the following screen shots..

Note(Alternate solution):

I don’t want to replace the OLTP date column NULL value with some default date (like 1/1/1900). If I do so in the reporting layer I have to replace the default date (1/1/1900) into NULL. Otherwise this will be misleading in the report also I feel this approach is overhead.

Database & SSIS : SQL Server 2008

Table        ColumnName                            DataType
Date_D    Date(lkp column)                        Date      
Order       ShippingDate(Source column)     Date      

 
select date_Key, date from [dbo].[Date_D]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
select * from (select date_Key, isnull(date,'1/1/1900') as date from [dbo].[Date_D]) [refTable]

where [refTable].[date] = ISNULL(?,'1/1/1900')
 
 
 
 
 
 
 
 
 
 
   

I am not sure about the performamce of Date dimenison look up. I am working on this...Since its a date dimension volume is not the consideration.

 

I hope this would be helpful to all.

Create Measure for DateTime DataType in SSAS

*** Self reference entry by Liyasker **
There is a requirement to create the Measure for User InTime Hrs, OutTime Hrs, CheckInTime, CheckOutTime, AVGCheckInTime with HH:MM AM/PM format.

But SSAS measure will support only for Integer and Numeric. Our case I have to expose the cube measure in to MS EXCEL 2007. There we can’t do any conversion.

I got the right help from the following blog of John (Thanks John :)). Please go through this, this will help you.

http://sqlblogcasts.com/blogs/drjohn/archive/2007/12/03/ssas-working-with-date-and-time-measures-to-calculate-average-elapse-time.aspx

I create a Facttable view and did the float conversion like below
,CONVERT(FLOAT,CAST(InTime_InHrs AS DateTime)) + 2 AS OfficeInTime
,CONVERT(FLOAT,CAST(OutTime_InHrs AS DateTime)) + 2 AS OfficeOutTime
,CONVERT(FLOAT,CheckInDateTime) + 2 AS InDateTime
,CONVERT(FLOAT,CheckOutDateTime) + 2 AS OutDateTime

Then I apply the Time format in SSAS Measure Properties.

I hope this will help to calculate DateTime Measure in SSAS.
 
 
 
 
 
 
 
 
 
 

Pentaho Transformation Deployment

How to deploy Pentaho Transformation(Database Connection)from Developement to Staging and Production?
Please look at this, I hope you can do this.
http://wiki.pentaho.com/display/EAI/Beginners+FAQ

I was looking more than 2 weeks to to find the right resource for the deployment :)

Pentaho - Fact Incremental Load

Pentaho - Fact Incremental Load

1. Create MapingHistory table to maintain the ETLRun Date


2. Load the MaxDate from Source and LastETLRunDate from MapingHistory Table(Destination).
A)Select MaxSource and Use the proper date convertion
Select job => SetVariables Transformation to hold the "MaxSource" date Value

3. Do the same for get the MAX ETL Date

4. Create Mapping to load FactData

a) Substitutevariable in the Source Script and choose "Replace Variable in script"
b) Use "Execute SQl Script" Tranfromation under the Scripting component 
and Select Variable Substitution option

5. Create job to Integrate the GetVariable and FactIncrementalLoad Transformation

Pentaho Looping

Pentaho Looping

I have a scenario to load the staging table from different sources. So i have to execute the same mapping more the one times based on number of rows avaliable in the Control Table.

Table : ConnectionInfo
ServerName      DatabaseName
Server1              DB1
Server2              DB2
Server2              DB3

I followed this URL and i could implement the Pentaho Looping
http://www.youtube.com/watch?v=rEG5yU8oLgI

Step1:"LoadConnections.ktr"

Create Transformation to select number of rows(Databasename,serverName) and copy the rows in to result set

Step2: "LoadConnectionInfo.ktr"

Create Transformation to Assign connection Information to variable 

Step3:"Load StgEntity.ktr"

a) Create Static Source Connection to load the data from source.

b) Create mapping to transfer the data from Source to Destination

c) Then change the connection name in to Variable.

Step4) "LoadVariable and ExecuteTransformation.kjb"

Create  the job to Integrate the (Step2)-LoadConnectionInfo.ktr and (Step3) - LoadStgEntity.ktr

Step5: Create job to integrate  Step1 - "LoadConnections.ktr"  and Step4 - "LoadVariable and ExecuteTransformation.kjb"

And configure the job properties like below

Here (Step4 - "LoadVariable and ExecuteTransformation.kjb")  job will be executed by the number of rows return from the (Step1- "LoadConnections.ktr")  transaction

I hope this will hellp you to do the pentahoo looping.

Parse Hierarchy(TreeView) XML using Outer Apply

/*

This sample will help you to parse Tree View XMl of Explicit format.
If you Apply Cross Apply on for Outer Apply then it will not display the child which does not have 3 Levl

If you store this XML in to temp table then you can update the Parent child clasification in to table.

*/
-- Please Replace [ and ] by < and >
DECLARE @HierarchyDetail XML

SET @HierarchyDetail =
'[treeview]


[treeview-nodes]

[treeview-node node-text="Account -1001" node-value="1001"]

[treeview-nodes]

[treeview-node node-text="Account -1075" node-value="1075"]

[treeview-nodes]

[treeview-node node-text="Account - 1132" node-value="1132" /]

[/treeview-nodes]

[/treeview-node]

[/treeview-nodes]

[/treeview-node]

[treeview-node node-text="Account - 1000" node-value="1000"]

[treeview-nodes]

[treeview-node node-text="Account -1077" node-value="1077"]

[treeview-nodes]

[treeview-node node-text="Account -1134" node-value="1134" /]

[/treeview-nodes]

[/treeview-node]

[/treeview-nodes]

[/treeview-node]

[treeview-node node-text="Account - 1080" node-value="1080"]

[treeview-nodes]

[treeview-node node-text="Account - 1101 " node-value="1101"]

[treeview-nodes]

[treeview-node node-text="Account - 1135" node-value="1135" /]

[/treeview-nodes]

[/treeview-node]

[/treeview-nodes]

[/treeview-node]

[treeview-node node-text="Account - 1114" node-value="1114"]

[treeview-nodes]

[treeview-node node-text="Account - 1124 " node-value="1124"]

[treeview-nodes]

[treeview-node node-text="Account - 1137" node-value="1137" /]

[treeview-node node-text="Account - 2937 " node-value="2937" /]

[/treeview-nodes]

[/treeview-node]

[/treeview-nodes]

[/treeview-node]

[/treeview-nodes]

[/treeview]'

Select Rows from Table1 NOT IN Table2 (t1.a = t2.a and t1.b = t2.b)

--Source(Table1 & Table2)

--Expected OutPut

DECLARE @T1 TABLE(A INT, B INT,C VARCHAR(10))
DECLARE @T2 TABLE(A INT, B INT,C VARCHAR(10))
INSERT INTO @T1(A,B,C)
VALUES (1,1,'SS1')
,(1,2,'SS2')
,(1,3,'SS3')


INSERT INTO @T2(A,B,C)
VALUES (1,1,'SS1')
,(1,2,'SS2')


--Method1:
SELECT A,B
FROM @T1
EXCEPT
SELECT A,B
FROM @T2
--Method2:
SELECT A.A,A.B
FROM @T1 A
left JOIN @T2 B ON A.A = B.A
AND a.B = b.B
WHERE b.B is null
--Method3:
SELECT * FROM @T1 AS T1
WHERE NOT EXISTS (SELECT 0
FROM @T2 AS T2
WHERE T2.A = T1.A
AND T2.B = T1.B
)
 
/*
 
I hope these will give the expected Result, Now i am looking over this which is best?
 
*/

String Concatenation Using XML Path

/*
Description : String concatenation using XML Path
This will help you to concatenate the reference table column values by using cross apply
*/
DECLARE @Table1 TABLE(CountryID INT,CountryName VARCHAR(50))
DECLARE @Table2 TABLE(StateID INT IDENTITY(1,1)
                                 ,CountryID INT
                                 ,StateName VARCHAR(50))

INSERT INTO @Table1(CountryID,CountryName) VALUES
(1 , 'USA'),
(2 , 'INDIA'),
(3 , 'AUSTRALIA')


INSERT INTO @Table2(CountryID,StateName) VALUES
(1,'Texas'),
(1,'Washington'),
(1,'New Yark'),
(1,'Colorodo'),
(2,'TamilNadu'),
(2,'Mumbai'),
(2,'Delhi'),
(2,'Banglore'),
(3,'Victoria'),
(3,'Tasmania'),
(3,'Queensland'),
(3,'New South Wales'),
(3,'Western Australia')


SELECT Table1.CountryID
,Table1.CountryName
,LEFT(StateNames,LEN(StateNames)- 1) AS StateNames
FROM @Table1 as Table1
CROSS APPLY
( SELECT StateName + ', '
FROM @Table2 AS Table2
WHERE Table2.CountryID = Table1.CountryID
FOR XML PATH('') ) AS State(StateNames)

--Desired Result

--* This Insert Statement will work in 2008 *

Parent - Child Hierarchy in SQL Server 2008 Reporting Service with Operational Source

Parent >> Child Hierarchy in SQL Server 2008 Reporting Service with Operational Source


I followed the below articles and i could build the Parent child Hierarchy in SSRS 2008 using Relational Database Source.

If you are ok with this look and feel then you can follow this.










http://www.mssqltips.com/tip.asp?tip=1939

http://blogs.msdn.com/b/robertbruckner/archive/2008/10/14/using-analysis-services-parent-child-hierarchies-in-reports.aspx


There are 4 Steps need to be done to do the SSRS 2008 Parent Child Hierarcchy report with ODS Source
Step 1: Create Datasource Connection  and Dataset

Datasource: [AdventureWorksDW]

SELECT DimEmployee.EmployeeKey

,DimEmployee.ParentEmployeeKey

,DimEmployee.FirstName

,ISNULL(SUM(SalesAmountQuota),0) AS SalesAmount

FROM DimEmployee

LEFT JOIN FactSalesQuota ON FactSalesQuota.EmployeeKey = DimEmployee.EmployeeKey

GROUP BY DimEmployee.EmployeeKey

,DimEmployee.ParentEmployeeKey

,FirstName

ORDER BY 4 DESC

How to debug SSIS Script task breakpoints in 64bit Environment ?

Please look at the Faruk Celik Blog.This will hep you.

I had the same proplem in SSIS 2008 64 bit Environment. Scrript task does not get in to Break Point. I over come this issue by following the bellow article. Thanks Faruk Celik.

http://blogs.msdn.com/farukcelik/archive/2010/03/17/why-the-breakpoints-that-i-set-in-my-script-task-not-script-component-in-the-data-flow-never-hits.aspx

 Thanks

Export Table Row in to CSV File

Step1: Create Procedure with Select Statement

Create Procedure pcget_ETLErrorlog
AS
BEGIN
   SELECT * FROM ETLErrorlog
END

Step 2:
DECLARE @SQL VARCHAR(2000)
DECLARE @ExportResult INT


SET @SQL = 'bcp "EXEC [MyDatabase].[dbo].pcget_ETLErrorlog" queryout "C:\Configuration V1\Global Alert\ETLAlert.CSV" -c -t, -T -S ServerName'
EXEC master..xp_cmdshell @SQL

Note: Procedure can be replaced directly by Select Statment "SELECT * FROM ETLErrorlog"
This will return the list of Output rows. If we dont want the out put rows then we can run the above query like this...


EXEC @ExportResult = master..xp_cmdshell @SQL,NO_OUTPUT
SELECT @ExportResult

Note: 
If the OutPut Result is 0 then It means no error. This will help you to validate and proceed the next step.