DEV Community

John Baker
John Baker

Posted on

Extracting SQL code from SSIS dtsx packages with Python lxml

Cross posted from my blog Analyze the Data not the Drivel

Lately, I’ve been refactoring a sprawling SSIS (SQL Server Integration Services) package that ineffectually wrestles with large XML files. In this programmer’s opinion using SSIS for heavy-duty XML parsing is geeky self-abuse so I’ve opted to replace an eye-ball straining1 SSIS package with half a dozen, “as simple as possible but no simpler”, Python scripts. If the Python is fast enough for production great! If not the scripts will serve as a clear model2 for something faster.

I’m only refactoring3 part of a larger ETL process so whatever I do it must mesh with the rest of the mess.

So where is the rest of the SSIS mess?

SSIS’s visual editor does a wonderful job of hiding the damn code!

This is a problem!

If only there was a simple way to troll through large sprawling SSIS spider-webby packages and extract the good bits. Fortunately, Python’s XML parsing tools can be easily applied to SSIS dtsx files. SSIS dtsx files are XML files. The following code snippets illustrate how to hack these files.

First import the required Python modules. lxml is not always included in Python distributions. Use the pip or conda tools to install this module.

# imports
import os
from lxml import etree
Enter fullscreen mode Exit fullscreen mode

Set an output directory. I’m running on a Windows machine. If you’re on a Mac or Linux machine adjust the path.

# set sql output directory
sql_out = r"C:\temp\dtsxsql"
if not os.path.isdir(sql_out):
Enter fullscreen mode Exit fullscreen mode

Point to the dtsx package you want to extract code from.

# dtsx files
dtsx_path = r'C:\Users\john\AnacondaProjects\testfolder\bixml'
ssis_dtsx = dtsx_path + r'\ParseXML.dtsx'
Enter fullscreen mode Exit fullscreen mode

Read and parse the SSIS package.

tree = etree.parse(ssis_dtsx)
root = tree.getroot()
Enter fullscreen mode Exit fullscreen mode

lxml renders XML namespace tags like <DTS:Executable as\}Executable. The following gathers all the transformed element tags in the dtsx package.

# collect unique element tags in dtsx
ele_set = set()
for ele in root.xpath(".//*"):
Enter fullscreen mode Exit fullscreen mode

Using transformed element tags of interest blast over the dtsx and suck out the bits of interest.

# extract sql code in source statements and write to *.sql files 
total_bytes = 0
package_name = root.attrib['{}ObjectName'].replace(" ","")
for cnt, ele in enumerate(root.xpath(".//*")):
  if ele.tag == "{}Executable":
    attr = ele.attrib
    for child0 in ele:
      if child0.tag == "{}ObjectData":
        for child1 in child0:
          sql_comment = attr["{}ObjectName"].strip()
          if child1.tag == "{}SqlTaskData":
            dtsx_sql = child1.attrib["{}SqlStatementSource"]
            dtsx_sql = "-- " + sql_comment + "\n" + dtsx_sql
            sql_file = sql_out + "\\" + package_name + str(cnt) + ".sql"
            total_bytes += len(dtsx_sql)
            print((len(dtsx_sql), sql_comment, sql_file))
            with open(sql_file, "w") as file:
print(('total sql code bytes',total_bytes))
Enter fullscreen mode Exit fullscreen mode

The code snippets in this post are available in this Jupyter notebook: Extracting SQL code from SSIS dtsx packages with Python lxml. Download and tweak for your dtsx nightmare!

  1. I frequently run into SSIS packages that cannot be viewed on 4K
    monitors when fully zoomed out. 

  2. Python’s readability is a major asset when disentangling

  3. Yes, I’ve railed about the word “refactoring” in the past but I’ve moved on and so should you. “A foolish consistency is the hobgoblin of little minds.” 

Discussion (1)

gtossou profile image

Interesting read. THANKS;
Is there a way to run the ssis package directly from python ?