How to Import Multiple Text Files into SQL Server with SSIS?

Import multiple text files – csv or txt – into a SQL Server table using an SSIS package and a dynamic loop called the for each loop.

This tutorial explains how to import multiple text files into SQL Server with SSIS using a single package and a dynamic loop to detect flat files. To start with, all text files (.txt, .csv, etc…) must be in the same folder. This automatic integration uses dynamic variables and allows to integrate many files with the same structure at the same time. To load several flat files dynamically, use an SSIS loop, i.e. the SSIS Foreach Loop Container component.

This package uses the same components as the previous SSIS tutorial to import a single flat file into a SQL table. It is therefore the same data flow used, but this time with variables in the dynamic file path.

Step 1: Download text files to import using SSIS and create a table (Premium)

First, to access our resources, to support us and much more, subscribe to our SSIS Full training available on Udemy.

The files are stored like this:

• C:\data, contains these CSV files:
  • Customers_Data_1.csv
  • Customers_Data_2.csv
  • Customers_Data_3.csv

Then create the destination table. The table uses the same structure as the tutorial for loading a flat file and fills a column with the full path to the source file.

Then, to access target table creation code, enroll in our SSIS Full training available on Udemy.

/* 

Then, to access target table creation code, enrol in our SSIS Full training available on Udemy.

*/ 

Step 2: Create the file loop and the SSIS data flow

First, use a loop from the control flow tab. Then select the dynamic Foreach Loop Container and drag it onto the workspace.

Step 3: Create the SSIS dynamic variable to store the file path

Create a variable named FlatFilePath of type String at the package level. To display the variables, click on the package and press F4 or right-click in the data flow area. Here the variable is called FlatFilePath and the Data Type is String.

Set up the flat file connection as detailed in this tutorial with a data flow to load a flat file. It is the same principle but this time the dataflow allows to load several text files with a single SSIS package. Then modify the ConnectionString expression to use the dynamic variable instead of the hard path.

Use the variable created before so that at each passage in the loop, the file name is dynamic.

Step 4: Configure the SSIS Foreach File Loop container

Then double-click on the Foreach Loop Container sequence. In the Collection tab :

1. Choose the folder where the flat files are located: C:\Data
2. Then specify the name of the flat files to be loaded with SSIS: Customers_Data*.csv
3. Choose the full name of the file with the path: Fully qualified

Then, in the Variable Mappings tab, link the FlatFilePath variable to index 0. This link dynamically transmits the name of the current file to the variable and therefore to the file connection.

Step 5: Add the file name to import using SSIS derived column

Edit the derived column component and add the following two derived columns for:

1. Add the Customer ID in integer format
2. Add the source file name from the dynamic variable

Step 6. Run the SSIS package to import the multiple text files with SSIS and check data in SSMS

Now start the loading. The SSIS package loads the flat files into the database one by one, i.e., one data flow executed per file detected in the source folder. In this case, 3 iterations occur.

Clicking in the data flow, we notice that 16 lines are integrated in each iteration.

Finally, check the results with the 2 following SQL queries. The result is consistent with 16 rows embedded for each source file, with no errors.

SELECT [Filename], COUNT(*)
  FROM [dbo].[Customers_with_filename]
  GROUP BY [Filename]; 

SELECT * 
  FROM [dbo].[Customers_with_filename];

Conclusion on multiple text files import with SSIS

To avoid errors when importing of multiple text files into a SQL Server table with SSIS, check the encoding of the files to process. Also, the mapping of the variable is mandatory to make the package dynamic and avoid manual development.

More on data import and export with SSIS

• Create a basic data flow with SSIS
• Create a connection to a SQL Server database in SSIS
• Import an text file into a SQL database with SSIS
• Create a package to import Excel data into SQL Server

SSIS Online Training