Import multiple CSV text files into a SQL table with SSIS

Import multiple csv text files into a SQL Server table with an SSIS dynamic loop.

This tutorial explains how to import multiple CSV text files into a SQL table with a single package that uses a dynamic loop to detect the files. To start with, all text files (.txt, .csv, etc…) must be in the same folder. This automatic integration uses dynamic variables and allows to integrate many files with the same structure at the same time. To load several flat files dynamically, use an SSIS loop, i.e. the SSIS Foreach Loop Container component.

This package uses the same components as the previous SSIS tutorial to import a single flat file into a SQL table. It is therefore the same data flow used with variables in the dynamic file path.

1. Download the text files to import and create the table with SSMS

First, download the three example files:

The files are stored like this:

• C:\data, contains these CSV files:
  • Customers_Data_1.csv
  • Customers_Data_2.csv
  • Customers_Data_3.csv

Then create the destination table. The table uses the same structure as the tutorial for loading a flat file and fills a column with the full path to the source file.

IF NOT EXISTS (
   SELECT * FROM sys.objects 
   WHERE object_id = OBJECT_ID(N'[dbo].[Customers_with_filename]') 
      AND type in (N'U')
)
CREATE TABLE [dbo].[Customers_with_filename](
   [CustomerID] [int] NOT NULL,
   [FirstName] [nvarchar](20) NULL,
   [LastName] [nvarchar](20) NULL,
   [City] [nvarchar](20) NULL,
   [Country] [nvarchar](50) NULL,
   [filename] [nvarchar](50) NULL,
   CONSTRAINT [CustomerswithFileNamePKCustomerID] 
      PRIMARY KEY CLUSTERED ([CustomerID] ASC)
)
GO

2. Create the file loop and the SSIS data flow

First, use a loop from the control flow tab. Then select the dynamic Foreach Loop Container and drag it onto the workspace.

3. Create the SSIS dynamic variable to store the text file names

Create a variable named FlatFilePath of type String at the package level. To display the variables, click on the package and press F4 or right-click in the data flow area.

Set up the flat file connection as detailed in this tutorial with a data flow to load a flat file. It is the same principle but this time the dataflow allows to load several text files with a single SSIS package. Then modify the ConnectionString expression to use the dynamic variable instead of the hard path.

Use the variable created before so that at each passage in the loop, the file name is dynamic.

4. Configure the Foreach File SSIS loop container

Then double-click on the Foreach Loop Container sequence. In the Collection tab :

1. Choose the folder where the flat files are located: C:\Data
2. Then specify the name of the flat files to be loaded with SSIS: Customers_Data*.csv
3. Choose the full name of the file with the path: Fully qualified

Then, in the Variable Mappings tab, link the FlatFilePath variable to index 0. This link dynamically transmits the name of the current file to the variable and therefore to the file connection.

5. Add the current text file name to import in a new derived column

Edit the derived column component and add the following two derived columns for:

1. Add the Customer ID in integer format
2. Add the source file name from the dynamic variable

6. Run the SSIS package to import the files and check the data with SSMS

Now start the loading. The SSIS package loads the flat files into the database one by one, i.e., one data flow executed per file detected in the source folder. In this case, 3 iterations occur.

Clicking in the data flow, we notice that 16 lines are integrated in each iteration.

Finally, check the results with the 2 following SQL queries. The result is consistent with 16 rows embedded for each source file, with no errors.

SELECT [Filename], COUNT(*)
  FROM [dbo].[Customers_with_filename]
  GROUP BY [Filename]; 

SELECT * 
  FROM [dbo].[Customers_with_filename];

7. Conclusion on the multiple text files import with SSIS

To avoid errors during the import of multiple text files into a SQL table with SSIS, it is generally important to check the encoding of the files to be processed. Also, the mapping of the variable is mandatory to make the package dynamic and avoid manual development.

More on data import and export with SSIS

• Create a basic data flow with SSIS
• Create a connection to a SQL Server database in SSIS
• Import an text file into a SQL database with SSIS
• Create a package to import Excel data into SQL Server

SSIS Tutorials