Pragmatic Works: BI Documenter 4.0

I attended a 30 minute webinar this morning hosted by Pragmatic Works founder Brian Knight to introduce the newest release of their BI Documenter tool.  The newest version introduces three new areas of functionality to an already very useful toolset.

  1. The ability to create and compare snapshots of objects
  2. Impact analysis across SQL Server, SSIS, SSAS, and SSRS
  3. Data profiling within the created documentation

Some of these features are discussed in this short video:

In addition to the three big ticket items listed above, there also seemed to be some rather major improvements to the user interface, an addition of SSIS package pictures to the documentation created, and  the ability to create and document database diagrams within the tool set.  Quite a list of new features!

Bottom-line, this looks like a solid tool that will really be helpful to better leverage all of the metadata associated with SQL Server, SSIS, SSAS, and SSRS.  Taking this a step further and adding SharePoint/Performance Point integration as well will really round out the offerings and truly provide one place to gather all pertinent business intelligence and data lineage information.

Like it? Tell others!
  • Facebook
  • del.icio.us
  • Technorati
  • TwitThis
  • Google Bookmarks
  • LinkedIn
  • Live
  • StumbleUpon
  • Digg
  • Identi.ca

Simple SSIS #1: Using the Data Viewer

Note: This is the first installment of what I hope turns into a series of posts highlighting very simple tasks within SSIS.  It seems like in a lot of cases there is a lot of information out there on the very complex or complicated tasks, but very little on the basics.  This is my effort to help out others that are working on learning SSIS but have not yet mastered all aspects of the tool.

The Data Viewer functionality within SSIS is a very useful feature which can be used to solve data related problems within a package.  With that, the main use and most common application of this functionality is troubleshooting data issues during the SSIS development lifecycle.

In an effort to allow developers the ability to see data as it flows through the SSIS pipeline, SSIS utilizes the Data Viewer.  The Data Viewer can be added to any path within the package data flow and it can display the data output in a variety of formats.

Setting up the Data Viewer is a Simple SSIS task – simply right click on the data path (success or failure) and select “Data Viewer” from the menu.  You can then easily add a Data Viewer to the stream, run the package, and see the data output. 

Setting Up the Grid Data Viewer and Results

The following video shows this process and the results using the Grid Data Viewer.  The grid is the most commonly used of the four types of Data Viewers and allows the developers to see raw data in columns and rows.

httpv://www.youtube.com/watch?v=PCGWU_UBeMg

It is important to note that when the package is run with a data viewer in place, you can either allow the data to continue down the path to completion by clicking the “Detach” button on the data viewer window.

Showing the Histogram Data Viewer Results

Along with the Grid Data Viewer, developers can also make use of the Histogram Data Viewer.  This viewer is used for numeric columns and can help give insight as to how data is distributed across a range.  The Histogram Data Viewer can be seen in action in the following video.

httpv://www.youtube.com/watch?v=pKCZ-DYbkVU

Showing the Scatter Plot Data Viewer Results

Another Data Viewer that can be used with numeric data – actually two numeric columns are needed – is the Scatter Plot Data Viewer.  This viewer can help uncover relationships between the two selected numeric columns.  Once again, the following video will show the configuration and results of the Scatter Plot Data Viewer.

httpv://www.youtube.com/watch?v=275_zW4pPy4

Showing the Column Chart Data Viewer Results

Unlike the Histogram and Scatter Plot Data Viewers, the Column Chart Data Viewer can be used with numeric or non-numeric data.  The purpose of this viewer is to show a summary of the values within a particular column in the data flow.  As in prior sections, this viewer is shown in action in the following video.

httpv://www.youtube.com/watch?v=lLN5opE7GlQ

Like it? Tell others!
  • Facebook
  • del.icio.us
  • Technorati
  • TwitThis
  • Google Bookmarks
  • LinkedIn
  • Live
  • StumbleUpon
  • Digg
  • Identi.ca

Nice 5 Minute Scrum Overview

My project manager has been kicking around the idea of using the Scrum methodology for some upcoming data warehouse projects and as such, I decided I better do a little research to figure out how it will apply to me as a developer.  In looking for something on the web to quickly explain the key concepts, I stumbled across this video from Brian and Devin Knight at Pragmatic Works. 

While the video isn’t all inclusive and left me with a lot of questions, I think it did a good job of illustrating the basics – not just in academic terms, but in a real life scenario that I was able to apply to my current situation.

Definitely worth the 5 minutes…

httpv://www.youtube.com/watch?v=nUsDZ4CBu6I

Like it? Tell others!
  • Facebook
  • del.icio.us
  • Technorati
  • TwitThis
  • Google Bookmarks
  • LinkedIn
  • Live
  • StumbleUpon
  • Digg
  • Identi.ca

SSIS: Where & How to Deploy?

One would think deploying SSIS packages would be a fairly straight forward process.  Heading in to this I would have never guessed that the question, “What is the best way to deploy SSIS files to the SSIS Server?” would pose so much discussion or raise so many questions. However, as I’m finding with most things I research and work on with SSIS development, there is NEVER a single best answer for these types of questions.  Really, it comes down to reading, analyzing, and finding the “best” solution for you and your situation – there is no one best way.  I wish it were more clear cut, but unfortunately there is a lot of grey area to swim through.

For a little background, there are basically three (but really only two) main options for deploying SSIS packages:

  1. File System
  2. SQL Server
  3. SSIS Package Store (really just a version of #1)

Each of these options has certain advantages and disadvantages and none of them are the clear cut winner. Go figure.

Deploying to the File System

The key advantage of this solution is that it is drop dead simple.  All you need to do is take the dtsx files you’ve created in BIDS locally and move them to a directory in the SSIS Server of your choice.  You can then use the DTEXEC executable on the server to run the dtsx files from there.  Additionally, you can schedule them to run via the SQL Agent and set any necessary parameters or command line setting necessary for the file to run correctly.  This is nice because there is no need for complicated configurations or added security, just put the files out there and let them run. 

Another benefit to the file system deployment is backup and recovery.  As long as the file system that holds your packages is properly backed up, recovery from a major melt-down is simple and straight forward.  An additional benefit is the ability to recover single files from the backup rather than the entire directory.

Easy, right?  Yeah.  I agree, it seems like a no-brainer, but – like anything -  there are a few downsides to consider as well.

For starters, any security you would like in place needs to be handled at the file/folder level using Active Directory to assign the access to the directory folders and associated files.  While this isn’t a huge issue nor necessarily bad, I think the SQL Server option provides a better implementation model, increases flexibility, and improves security.  Along with security, there is limited audit ability included in utilizing a file systems deployment.  While it may be possible to track who put a new file in place or overwrote an existing file with a newer version, it is not ideal.   

Deploying to SQL Server

This is definitely the most complicated deployment option, but it does offer the best security options and execution performance that is on par with the other options.  There are also SSIS specific SQL Server roles that can be granted to allow (or not allow) people to see, execute, and interact with SSIS packages stored in the database.  Additionally, the packages can be encrypted within the database which allows for yet another layer of package security.  Not bad.

Backing up SSIS packages stored in the database can be accomplished by backing up the database.  While this is simple and most DBA’s will already have this in place, it does come with its drawbacks.  The primary issue with this is that it is not possible to restore or recover a single SSIS package by using the database backup – the only option would be to restore ALL of the SSIS packages stored in the data base.  This can be rather easily overcome by utilizing a source control product like Microsoft Visual Source Safe to store current version of all packages stored in the database.

Deploying to SQL Server seems to be the best option for our environment.  While it doesn’t solve every problem – and actually creates a few of its own – it seems to me to be the most easily manageable deployment method of the options given.

Deploying to the SSIS Package Store

As the SSIS Package store is really just file location within the SQL Server installation, using the SISS Package Store for deployment has all of the same challenges and advantages as utilizing the file system.  The location of the SSIS Package Store is Program Files\Microsoft SQL Server\90\DTS\Packages directory. 

Resources

I gathered the above information from a variety of resources, most all of which are listed below.

Additionally, I found the following table from Pragmatic Works insightful at a glance and useful to get a basic idea of the important issues and how the main deployment options perform in regards to those issues.

Functionality

Best in File System

Best in MSDB

Security   x
Backup and Recovery x  
Deployment x  
Troubleshooting x  
Execution Speed x x
Availability x  
Like it? Tell others!
  • Facebook
  • del.icio.us
  • Technorati
  • TwitThis
  • Google Bookmarks
  • LinkedIn
  • Live
  • StumbleUpon
  • Digg
  • Identi.ca

SQL Server 2008 R2 Video – SQL Server News Hour: Donald Farmer on Managed Self Service BI and SQL Server 2008 R2

Here is a new video posted to YouTube from the SQL Server News Hour…

httpv://www.youtube.com/watch?v=TPyCeRbp9k0

Like it? Tell others!
  • Facebook
  • del.icio.us
  • Technorati
  • TwitThis
  • Google Bookmarks
  • LinkedIn
  • Live
  • StumbleUpon
  • Digg
  • Identi.ca