SQL Insert via mappings table

All we need is an easy explanation of the problem, so here it is.

As part of a SSIS ETL process, I have multiple staging tables. I have used xxxx just to resemble data:
e.g.
tblStage1

Field_ABC  Field_ZXC  Field_QWE
 xxxx       xxxxxxxxx   xxxxx


tblStage2

Field_PLM   Field_WER   Field_YTE
 xxxxx       xxxxxx      xxxxxxx

I then have the ‘real’ tables that they need to be inserted into:

tblDest1
FldABC    FldZXC    FldQWE


tblDest2
FldPLM  FldWER   FldYTE

Using info from the Information Schema I have built a mapping table that details which table and field from the staging tables map to which real table:
tblMapping:

StageTble   StageColumn   RealTable    RealCol    OrdinalPosition
tblStage1   Field_ABC     tblDest1     FldABC     1
tblStage1   Field_ZXC     tblDest1     FldZXC     2
tblStage1   Field_QWE     tblDest1     FldQWE     3
tblStage2   Field_PLM     tblDest2     FldPLM     1
tblStage2   Field_WER     tblDest2     FldWER     2
tblStage2   Field_YTE     tblDest2     FldYTE     3

How do I loop through each staging table and insert into the real tables\columns based on the values in tblMapping?

Would this be possible via a Cursor and a Loop?

Any help would be greatly appreciated.

How to solve :

I know you bored from this bug, So we are here to help you! Take a deep breath and look at the explanation of your problem. We have many solutions to this problem, But we recommend you to use the first method because it is tested & true method that will 100% work for you.

Method 1

Perhaps you could create an SSIS script process to dynamically build the insert commands and execute them.

--Demo setup
drop table if exists MappingTable;
create table MappingTable 
    ([StageTble] varchar(9), [StageColumn] varchar(9), [RealTable] varchar(8), [RealCol] varchar(6), [OrdinalPosition] int)
;

INSERT INTO MappingTable
    ([StageTble], [StageColumn], [RealTable], [RealCol], [OrdinalPosition])
VALUES
    ('tblStage1', 'Field_ABC', 'tblDest1', 'FldABC', 1),
    ('tblStage1', 'Field_ZXC', 'tblDest1', 'FldZXC', 2),
    ('tblStage1', 'Field_QWE', 'tblDest1', 'FldQWE', 3),
    ('tblStage2', 'Field_PLM', 'tblDest2', 'FldPLM', 1),
    ('tblStage2', 'Field_WER', 'tblDest2', 'FldWER', 2),
    ('tblStage2', 'Field_YTE', 'tblDest2', 'FldYTE', 3)
;

SSIS script task

--SSIS script task
--NOTE: the actual EXECUTE sp_executesql @cmd is commented out
Declare @Cmd nvarchar(max)
;with DistinctTables as
(
select distinct [RealTable], [StageTble] from MappingTable
),
InsertCommands as
(
select *,
'insert into ' + [RealTable] + '(' +
STUFF((
        SELECT ',' + [RealCol]
        FROM MappingTable t1
        where t1.RealTable = drt.RealTable
        ORDER BY OrdinalPosition
        FOR XML PATH(''), TYPE).value('.', 'VARCHAR(MAX)'),1,1,'')
+ ')' + 
' select ' + 
STUFF((
        SELECT ',' + [StageColumn]
        FROM MappingTable t1
        where t1.RealTable = drt.RealTable
        ORDER BY OrdinalPosition
        FOR XML PATH(''), TYPE).value('.', 'VARCHAR(MAX)'),1,1,'')
+ ' from ' + [StageTble]  as InsertCommand
 from DistinctTables drt 
)
select @cmd = 
STUFF((
        SELECT ';' + char(10) + [InsertCommand]
        FROM InsertCommands
        FOR XML PATH(''), TYPE).value('.', 'VARCHAR(MAX)'),1,1,'')
+';' from InsertCommands ic

print @cmd
--EXECUTE sp_executesql @cmd

Produces the following dynamic SQL

insert into tblDest1(FldABC,FldZXC,FldQWE) select Field_ABC,Field_ZXC,Field_QWE from tblStage1;
insert into tblDest2(FldPLM,FldWER,FldYTE) select Field_PLM,Field_WER,Field_YTE from tblStage2;

Note: Use and implement method 1 because this method fully tested our system.
Thank you 🙂

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0