SQL Chops Episode #1 - Table Of All Dates

by bpanjavan 17. February 2012 18:47

This script is for a table function of all dates for a given range.

USE [MyDB]
GO
/****** Object:  UserDefinedFunction [dbo].[fn_GetCalendarDateTableAllRange]    Script Date: 01/23/2012 11:12:40 ******/
IF  EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[fn_GetCalendarDateTableAllRange]') AND type in (N'FN', N'IF', N'TF', N'FS', N'FT'))
DROP FUNCTION [dbo].[fn_GetCalendarDateTableAllRange]
GO
USE [MyDB]
GO
/****** Object:  UserDefinedFunction [dbo].[fn_GetCalendarDateTableAllRange]    Script Date: 01/23/2012 11:12:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE FUNCTION [dbo].[fn_GetCalendarDateTableAllRange]
(    
@minDateInclusive DATE 
,@maxDateInclusive DATE 
)
RETURNS     
@CalendarDateAllRange TABLE
(CalendarDate DATE    )
AS
BEGIN
IF @minDateInclusive > @maxDateInclusive
RETURN
DECLARE @loopCurrentDate DATE
SET @loopCurrentDate = @minDateInclusive
WHILE @loopCurrentDate <= @maxDateInclusive
BEGIN
INSERT INTO @CalendarDateAllRange VALUES  (@loopCurrentDate)
SET @loopCurrentDate = DATEADD(DAY,1,@loopCurrentDate)
END
RETURN
END
GO

This can be useful if I want report of how many customers enrolled with my service on each day of the month.
Lets say I had a table of customer enrollments with a date of enrollment for each.

Customer Enrollment Table
Customer Enrollment Table

Grouping these without leveraging my date table does not show me the dates that we had no enrollments, and does not illustrate the clear gaps.

Select Enrollment Date Counts
Enrollment Counts by Date

Using excel I get a graph that visually lies to me

Enrollment Graph 1
Enrollment Graph #1

But by using our comprehensive date table as our base, we get a full chronological view.

Select Enrollment Date Counts Full
Enrollment Counts by Date with Full Dates

Using excel I can get a more truthful graph.

Enrollment Graph 2
Enrollment Graph #2

But this function only gives me an un-indexed inmemory table variable which will not scale for a larger time period.
In that case, the following script creates the permenant table that we would need and applies a primary key to the single column which automatically creates a clustered index.

 

USE [MyDB]
GO
/****** Object:  Table [dbo].[CalendarDateAllRange]    Script Date: 02/17/2012 22:20:18 ******/
IF  EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[CalendarDateAllRange]') AND type in (N'U'))
DROP TABLE [dbo].[CalendarDateAllRange]
GO
USE [MyDB]
GO
/****** Object:  Table [dbo].[CalendarDateAllRange]    Script Date: 02/17/2012 22:20:19 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[CalendarDateAllRange](
[CalendarDate] [date] NOT NULL,
CONSTRAINT [PK_CalendarDateAllRange] PRIMARY KEY CLUSTERED 
(
[CalendarDate] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
-- Now populate the dates in the table
DECLARE    @minDateInclusive DATE 
DECLARE @maxDateInclusive DATE 
SET @minDateInclusive = '1/4/2010'
SET @maxDateInclusive = '5/2/2012'
DECLARE @loopCurrentDate DATE
SET @loopCurrentDate = @minDateInclusive
WHILE @loopCurrentDate <= @maxDateInclusive
BEGIN
INSERT INTO CalendarDateAllRange VALUES  (@loopCurrentDate)
SET @loopCurrentDate = DATEADD(DAY,1,@loopCurrentDate)
END
SELECT * FROM CalendarDateAllRange

But lets be clear that only under rare circumstance should you create a foreign key reference to this table. There would be no benefit to doing so and you would only be hindering performance of your child table.

Scripts available below
fn_GetCalendarDateTableAllRange.sql
DropAndCreateCalendarDetailTable.sql

Tags:

SQL Server Integration Services

Programming in SSIS

by bpanjavan 22. January 2012 11:13


In this example I’m using SSIS to write a .NET-like executable application that grabs data and “processes” it. Using SSIS is a nice alternative if your application is not “business-logic” heavy, and that way you can avoid a code-heavy program that may be more flexible but less robust in the process.

    Background: Energy Suppliers must leverage pipeline transportation systems to transmit Natural Gas to local Utilies to be distributed to customers. Each day, the pipeline accepts nominations for transportation the following day. The Supplier must identify all counter-parties and indicate the date and amount to be delivered. Additionally, they must properly calculate the line-loss from delivery to destination, which is typically 5%.

Here are the steps at a very high level:

  • Query for an un-submitted “open” set of gas nominations that I need to submit to the pipeline.
  • Use the Nomination Set to load the nominations themselves.
  • One-by-one, submit each nomination to the gas pipeline’s web service
  • Log the output of each submission
  • Close the Nomination Set

 

 

Setup

We’ll start with two tables:
    Nomination – The table to house nominations to be submitted
    Nomination Set – The “father table” to hold specifications for groups of Nominations

Nomination Tables
Nomination Tables
Sample Data from these tables here.


    At any time, I can create a new Nomination Set specifying parameters around the nominations that I want to encapsulate with this set. This Nomination Set will have its Closed flag set to FALSE until I process the Nomination Set.

 

 

Create SSIS Package and Setup Variables

New Package
New Package
    Variables in SSIS can serve several purposes. I’ve most often seen them used as configuration variables, specifying connection strings and URL’s, but you can also use them as local variables for your current working set of data.

Variables
Variables
  • NominationSetID – The ID of the current Nomination Set that we’re processing.
  • ScheduleDate – Loaded from the Nomination Set. Used as a filter when querying Nominations to submit.
  • TransportationServiceURL – Loaded from the Nomination Set. Will be used by our web service task as the URL to call.
  • TraderCode - Loaded from the Nomination Set. Used as a filter when querying Nominations to submit.
  • NominationSector - Loaded from the Nomination Set. Used as a filter when querying Nominations to submit.
  • NominationData - Will be used as an in-memory table for iterating through our nominations (more on this later).
  • Loop_CurrentNomination – Used within our ForEach Loop. Represents the current Nomination Data to submit.
  • Loop_NominationOutput – Used within our ForEach Loop. Represents the output from submitting our nomination to the web service.

 

 

Load an Open Nomination Set

    I’ll use an Execute SQL task tied to an OLE DB connection to query for an open Nomination Set.

Load Open Nomination Task
Load Open Nomination Task

This is a simple SQL Statement:
SELECT TOP 1 
NominationSetID 
,[NominationSector]
,[PipelineEndpointURL]
,[TraderCode]
,NominationSetDate = CONVERT(DATETIME,[NominationSetDate])
,[Closed]
FROM [dbo].[NominationSet]
WHERE Closed = 0

    Load the result-set into our package variables for ongoing use. In order to do this we must set the Result Set to Single Row.

Set Result Set To Single Row
Set Result Set To Single Row

Then you must map your resulting output columns back to your pre-defined variables.

Map Result Set To Variables
Map Result Set To Variables

Debugging can help verify that your mappings assign correectly. Right-click on your Execute SQL Task and select Edit Breakpoints

Edit Breakpoints
Edit Breakpoints


Using the Watch Window, we can verify the values of our variables

Using the Watch Window to verify our variable assignments
Using the Watch Window to verify our variable assignments


 

 

Load Nominations

    Now we run a query to load the Nomination data table rows that we will iterate through.
SELECT [NominationID]
,NominationData = 
[Shipper_ContractNumber]
+ ' | ' + [DeliveringParty_ContractNumber]
+ ' | ' + CONVERT(VARCHAR(50),[StartDate],112)
+ ' | ' + CONVERT(VARCHAR(50),[StopDate],112)
+ ' | ' + [ReceiptMeterNumber]
+ ' | ' + CONVERT(VARCHAR(50),[ReceiptAmount])
+ ' | ' + [DeliveryMeterNumber]
+ ' | ' + CONVERT(VARCHAR(50),[DeliveryAmount])
+ ' | ' + [ReceivingPartyContractNumber]
,[ScheduleDate]
,[Sector]
,[TraderCode]
FROM [dbo].[Nomination]
WHERE
ScheduleDate = ?
AND Sector = ?
AND TraderCode = ?

We start by creating another Execute SQL Task.
Because we want the full table, we set the Result Set output to Full record set and assign it to the variable NominationData.

Full Record Set
Set Result Set to Full Record Set


Set Output To NominationData
Set Output to our NominationData variable


 

 

Loop through Dataset and Process

    Now we can use a Foreach Loop Container to iterate through our Nomination Data Set. For each iteration of the loop, we need to capture the current Nomination Data value by mapping it to our variable Loop_CurrentNomination

Foreach Container
Add a Foreach Container


Configure Enumeration
Point enumeration to the NominationData collection from previous step


Map Current Nomination Data
Map the current row to our variable Loop_CurrentNomination

 

 

Submit the Nomination

    I’m using a simple .NET WCF Web Service that takes in the NominationData as a string and returns a confirmation message. I won't go into the details of a Web Service Task or ASP.NET WCF in this post, because I just want an example of "processing data".
Configure the SSIS Web service task to point to the URL and WSDL of the exposed service.

Configure Method Call to Web Service
Set Method Call and pass in our variable Loop_CurrentNomination


To capture the result, I can retrieve the return value of my output into another SSIS variable, Loop_NominationOutput.

Set Output Variable
Set output to our variable: Loop_NominationOutput


Because I'm hosting my web service, I can debug through the method call and verify the Nomination Data input:

Debugging Web Service
Debugging our Web Service Call


 

 

Logging the Output

    At this point, I have successfully submitted my nomination and want to log the output.
I can use a Data Flow task with a destination Output File as my log.

Add Data Flow Task to Foreach Loop
Add Data Flow Task to our Foreach Loop


Our Data Flow task consists of 3 steps.

Data Flow Task
Our Data Flow Task to Log our output


In this case, we're only using our first step of the Data Flow task as a place-holder data-source. We'll be getting our data to output in the next step:

Data Flow Task Step 1
Data Flow Task Step 1


In the second step of our Data Flow Task, we grab the variables we want to output to the log.
The expression for our NominationOutput column is complex, but just understand we're exracting the output result string from our web service call.

Data Flow Task Step 2
Data Flow Task Step 2


For Step 3, I start by configuration a Flat-File Connection Manager on my local drive. Then I can add my Flat-File output step and map my columns appropriately.

Data Flow Task Step 3
Data Flow Task Step 3 - Map Columns to Flat File


At this point, if I run my SSIS package I can see the output results of my web service submission call to my LogOutput.txt file

Log Output
Log file with our Nomination submission results


 

 

Closing our Nomination Set

    Finally, to finish the process we'll run another Execute SQL Task to close our Nomination Set. Our query is a simple UPDATE statement. Notice the '?' marking the parameter input. We get our input from our variable NominationSetID

Close Nomination Set
Query to Close Nomination Set


Configure our input to our query
Configure our input to our query


And now, one full cycle will query to find an un-closed Nomination Set, submit all Nominations through the exposed Web Service, log the output to our log file, and update the Nomination Set record.

Full source code for both the SSIS package and WCF Web Service can be found here.

 

 

Conclusion

    I realize that my rollercoaster is missing a few “struts” for complete stability (i.e. Error logging, transactional integrity). Error logging can be achieved through alternate error-routing in SSIS, which is another simple drag-and-drop task. Transactional Integrity is also a capability, but for this example it’s not necessarily achievable because I am calling a web-service, which would require a distributed roll-back on the service-end. But for everything else, I stand-by my assertion that SSIS has proven to be an incredibly powerful tool for automating rudimentary tasks (i.e. I/O, data retrieval and output). If I’m writing complex domain-driven code, I prefer and obviously recommend .NET, where you can modularize your code, wrap Unit Tests, and do much more. But for something this simple I was able to achieve what I need without a single line of code (scripting and querying is NOT code :) …).

 

Tags: , ,

Data | Development | SSIS | SQL Server Integration Services

IIS7 Permissions – IUSRS and IIS_IUSR

by bpanjavan 17. December 2011 19:42

Setting up access control permissions on Windows 7 for a new website hosted under the wwwroot is not difficult, because you don’t have to do anything. But if you’re hosting a website from a virtual directory outside of the wwwroot you might get this error:

IIS7 Configuration Error
The requested page cannot be accessed because the related configuration data for the page is invalid

 

    After close examination this is a permission issue. You may also get this error if you’ve only solved half the problem:

IIS7 Permission Error
You do not have permission to view this directory or page because of the access control list (ACL) configuration or encryption settings for this resource on the Web server

 

You need to grant permission to 2 entities using Windows Explorer security controls:
  • IUSRS - The new default account used for anonymous authentication
  • IIS_IUSRS – A new built-in group serving as a container for all application pool identities

    By default, these objects have access to wwwroot (IUSR has access under the Users group).

wwwroot permissions

They need the same permissions for your respective website’s directory.
Instructions below:

 

  1. Navigate to your website’s directory. Right-click and bring up the Properties menu. Under the Security menu, click Edit to change permissions.

  2. Security Menu

  3. Click Add to add another object to the access list. Type in “IUSR” and click Check Names to verify. Click Ok.

  4. Add User Permission

  5. By default the typical “Read” permissions should be checked. Repeat step 2 typing in the group “IIS_IUSRS” instead.

  6. Read Permissions

  7. Click Ok to close the properties dialogue.

Now you’re ready to rock’n’roll with a website hosted under IIS 7! For a more detailed explanation of the new entities please refer to the article below:

Understanding Built-in User and Group Accounts in IIS

Tags: , , , ,

IPhone Development - The Insta-Email App

by bpanjavan 24. September 2011 15:19


Insta-Email

    The Head-First book series is incredible, possibly the most not-boring set of technical resources I've ever come across. Of course they have one for IOS Development, decked out and up-to-date with XCode 4 too!

Headfirst iPhone
HeadFirst iPhone Development

    The Insta-Email app is one of the first "complete" apps that you create as an exercise. It demonstrates the use of Views, View-Controllers, IBActions and IBOutlets, and the first iPhone "data-control".

IDE
XCode 4 IDE

Application Lifecycle

Choose an Activity. Then choose your Feeling
Choose Activity and Feeling

Press "Send Email" to inform your significant other!
Press Send Email

The Email is sent with the message!
Email Sent

The Root View (UI)

UI Views
The Sub-Views that comprise the Root View
The Root View consists of
  • A few Labels
  • A UIPickerView for choosing the Activity and Feeling
  • A UIButton for sending the Email

Outlets and Actions

    The way you wire-up events and access your Views ("controls" for you Microsoft people) is through IBActions and IBOutlets. The IBAction sendButtonTapped is declared as the delegate method to intercept an action from the UI. The action is then wired up using Interface Builder.

Wire Button Action
Wiring the button-click to the IBAction

From there, the code to send the Email is executed (won't go into the details of how that's done for now).

#pragma mark
#pragma mark IBActions
-(IBAction) sendButtonTapped:(id)sender
{
int actionIndex = [self.emailPicker selectedRowInComponent:0];
int feelingIndex = [self.emailPicker selectedRowInComponent:1];
NSString* actionMessage = [activities_ objectAtIndex:actionIndex];
NSString* feelingMessage = [feelings_ objectAtIndex:feelingIndex];
NSString* theEmailMessage = [NSString stringWithFormat:@"I'm %@ and feeling %@ about it.", actionMessage, feelingMessage];
//NSLog(@"%@", theEmailMessage);
if ([MFMailComposeViewController canSendMail])
{
MFMailComposeViewController* mailController = [[MFMailComposeViewController alloc] init];
[mailController setMailComposeDelegate: self];
[mailController setSubject:@"Hello Xuan!"];
[mailController setMessageBody:theEmailMessage isHTML:NO];
[self presentModalViewController:mailController animated:YES];
[mailController release];
}
else
{
NSLog(@"Sorry you need to setup mail first");
}
}

    The IBOutlet allows me to connect a variable declared in my Header File to a control in my View. Without IBOutlets, the code in my ViewController would never be able to access or manipulate anything on my View. To do this, I must declare a variable in my Header File of the same type as the View I want to connect. Then I use Interface Builder to connect the two.

Wire IBOutlet to UIPicker
Wiring the View-Controller to the UIPickerView using IBOutlet

This allows my code to access the UIPickerView at will.

//
int actionIndex = [self.emailPicker selectedRowInComponent:0];

A View that needs a Data Source

Working with a data-driven view in Cocoa Touch is very different from a data-driven control in ASP.NET. In the latter, all you really need is a data-source collection and you're good-to-go. With the former, you must designate a class to implement each of the protocols (ALMOST like an interface in .NET) required by the view to function.

The first is the UIPickerViewDataSource. This protocol features the following method signatures for helping to define the rows and components in the view:

  • numberOfComponentsInPickerView
  • pickerView:numberOfRowsInComponent

 

//
#pragma mark - 
#pragma mark Picker Datasource Protocol
- (NSInteger) numberOfComponentsInPickerView:(UIPickerView *)pickerView
{
return 2;
}
-(NSInteger) pickerView:(UIPickerView *)pickerView numberOfRowsInComponent:(NSInteger)component
{
if (component == 0)
{
return [activities_ count];
}
else
{
return [feelings_ count];
}
}
Our implementation of the UIPickerViewDataSource

The second is the UIPickerViewDelegate, which provides methods for defining templating and the content that populates the view.

  • pickerView:rowHeightForComponent
  • pickerView:widthForComponent
  • pickerView:titleForRow:forComponent
  • pickerView:viewForRow:forComponent:reusingView
  • pickerView:didSelectRow:inComponent
//
#pragma mark -
#pragma mark Picker Delegate Protocol
-(NSString *) pickerView:(UIPickerView *)pickerView titleForRow:(NSInteger)row forComponent:(NSInteger)component
{
if (component == 0)
{
return [activities_ objectAtIndex:row];
}
else
{
return [feelings_ objectAtIndex:row];
}
}
Our implementation of the UIPickerViewDelegate

Did I mention that not ALL methods are required to be implemented? THAT's why protocols are NOT like interfaces in .NET. In this case the one I bolded is the only one I need for this example.

 

    In accordance with the Apple Human Interface Guide, this was more like a Utility application. The next application on our list will be a Navigation-based application.

Tags: , ,

Development | iOS | Objective-C

Project Lumina

by bpanjavan 3. August 2011 14:01

     This is a project that I've been working on in my spare time. My development team has a shiny plasma on the wall the Silverlight Toolkit has some very nice data visualization tools and I've been looking for an opportunity so I just put 2 and 2 and 2 together.


Paradigms and Paradigm Controls

     The flow is based on the concept of Paradigms, which are contained in a Paradigm Deck. For this example we load a Paradigm Deck called "SALES-DEMO". This deck consists of 2 Paradigms, one entitled "Year Over Year Earnings". This Paradigm contains 2 Paradigm Controls. The one in the top zone is a graph of year-over-year earnings for a company from 2010 to 2011. The bottom control contains a table with the same information shown in the top control. The second Paradigm entitled "Sales By Channel" contains 2 Paradigm Controls. Notice that the left Paradigm Control in this Paradigm uses the same template table as the bottom Paradigm Control in the last Paradigm. These controls are dynamically driven off of a separate meta-data store.


The Technology

     This project was built on Silverlight 4 (originally I used Silverlight 5 but the runtime is not officially available to the common people). I also used the Silverlight 4 Control Toolkit available from CodePlex, specifically the Data Visualization controls.


The Design

     This was my first experiment with the Model-View-View Model pattern. The most challenging thing however was dynamically loading the views and graphs off of a separate data store. More detail on that in a later post.

Tags: , ,

Silverlight

Learning iOS Part I - From C# to Objective-C

by bpanjavan 29. May 2011 21:46

NOTE - This post is NOT an introductory guide to Objective-C. What may resemble "Guidance notes" are merely my learning accounts

    My co-workers started an iOS study-group 5 weeks ago. I won't stay in my comfortable .NET shell because you never know where the demand-side of the market will go, and where-ever it should be 5 to 10 years from now, I plan to stay on the supply-side. I didn't start sooner because of laziness and fear of the unknown, two natural tendencies that would serve us well to defy. But thanks to peer pressure I'm on this road and I'm starting a series of posts to recap my accounts.

I'm in a slightly unique position, because my only experience with C++ was one semester in high-school before transferring to Java. I'm one of those youngsters that old-timers gripe about, never having to manage my own memory. But now I'm finally witnessing relics of C, the great ancestor of C#.


Understanding What I'm Dealing With

    Objective-C is to C what C# 3.0 is to C# 2.0. By that I mean Objective-C is an extension of C that provides Small-talk "messaging" features to the base language, though the runtime is the same. Likewise C# 3.0/3.5 adds features to C# 2.0, such as automatic properties, extension methods, and LINQ.


Separation of Files - The Header and Body (a.k.a. interface from implementation)

    The first example of "why do I have to do this when the compiler already knows what I want?". The header file provides a forced documentation of your class members. Figuring out class members is not a "one-glance" routine in .NET. But inspecting the header file is. And because the header file is the place for instance variable declarations, it truly forces the implementation file to be a place for pure implementation.


Smalltalk Messaging

    Freedom from the burden of type-checking? It's funny, and ironic. My friend told me the history of programming languages in which we've gone back and forth from loosely typed languages that let you "pass anything to everything" like C, to strongly typed languages where not only is there a net RIGHT under the high wire, but captain compiler won't let you out of the nest without a thorough type-check. But the messaging syntax allows you to decide your level of enforcement:

  • If you want the object-oriented (OO) mutable style of C# then you use the dot (.) syntax. But the messaging syntax provides a dynamic-typed functional approach. I guess what impresses me is that I've never seen the two incorporated into the same language
  • I can't yet vouch for the Apple developers' claim that this is more readable, but I intend to press on using it if this is truly the more common standard this side of the fence


Memory Management

    I'm guessing that computer science majors wound up relating memory management in C back to semaphores from Operating Systems. It actually happened the other way around for me. I was watching the Stanford Video series and heard someone mention semaphores after learning about allocation and release counters. I Google-ed semaphore and found the resemblance uncanny.

Simple vs. complex types
You don't have to allocate or release simple variable types, since they don't use pointers. You do have to for complex types. Object instances are managed with pointers, so anything that was pointed to (and thus resides on the heap) must be de-allocated.
I was surprised to find out that there IS a garbage collector, but it will only collect on released memory. Any pointer that is allocated must be de-allocated or it will remain uncollected

Semaphore logic
In a nutshell, here's how it works. Just as multiple variables can reference the same instance in C#, multiple pointers can reference the same memory location. The run-time increments the allocation counter for every pointer that has been added. The run-time decrements the counter for every pointer that is released. When the last pointer is released, the memory is reclaimed.


Synthesizing Properties

    The cool version of Automatic Properties. I think it's ironic that with header files and pointer management, a lot of things are explicit. Accessors(not in the English dictionary?) through property synthesis are implicit. By synthesizing property members you just kind of know that the accessors are there.


Next Time...

    After that retrospect, I got to pondering the question of which language is "better". Alright, that'll be next time's topic.

Tags: , , ,

Commentary | Development | iOS | Objective-C

Silverlight 3/4 Hands-On Lab - Inventory Manager

by bpanjavan 22. May 2011 13:46

Sample Overview

Below is a summary of my work on one of the Silverlight 3 Hands-On Labs. This lab focuses mainly on data layout and retrieving data from LINQ to XML, or a WCF Service. My walkthrough contains the following concepts:

  1. Defining Layout using Grid and Canvas
  2. User Controls
  3. The DataGrid control
  4. Data Context-driven data binding
  5. The Factory Pattern (Design Pattern)
  6. Calling asynchronous methods and delegating a call-back method
  7. Configuring and calling a WCF Web Service
  8. LINQ to XML
  9. LINQ to SQL


Overview

To use the Inventory Manager, the user can search for products by SKU or by Name.
If the user searches by SKU, a single product's information will be displayed in a Grid layout.
If the user searches by Name, all products whose names contain the search text will display in a Data Grid.

Layout - A
The top section consists of a border wrapped around a Grid layout control (a common practice).

3a_Layout_Top

The middle section features the search controls and the bottom section features a Canvas surrounding a Grid of TextBlocks for displaying Product Information.

3b_Layout_Middle


Walkthrough - Search by SKU Against Mock Data-store

2_Walkthrough_SearchBySKU


The user follows the steps outlined in the overview, selecting "Search by SKU", typing the Product Number in the textbox, then clicking "Search". This triggers the event handler.

4_0_EventHandler

The Factory Pattern is used to return an object implementing the expected interface. The caller neither knows nor cares about the implementation details of the data provider. It just knows that whatever is returned implements the following interface

4_1_Factory

Below is the interface that my particular Inventory Manager must implement.

4_2_Interface


The class InventoryManagerMock uses LINQ to a static collection to retrieve mock data.

5_InventoryManagerMock


First, the call-back is wired to the event that signals the completion of the operation.Second the call-back is called when the event is fired. The object returned in the result is bound to the DataContext of our grid.

6_callback_bindcontexttoobject

Data Context-driven binding in Silverlight means you don't need a specific data control to support data binding. Just about any property of any control supports binding. The respective "source" is the most specifically defined data context.

Think of it like scope in terms of programming. In this case, I'm binding a Text property of my control to the property name ProductID. Silverlight will traverse up the control tree and find the most specifically defined scope in which the data context has been bound. In this case, the Grid has been bound to a ProductInfo object. Just as easily, the entire UserControl could have been bound to an object which would have served as my data source.

7_ContextBinding

The result is that my Inventory Detail Grid (gridInventoryDetails) shows the values of the ProductInfo object returned.

8_SKUSearch_BindingResult

Layout - B

The next walkthrough will be a search by name, which uses an alternative layout:

9_NewSectionWithDataGrid


For this I have a new UserControl called ProductListView.xaml.

10_UserControl_ProductListView


The UserControl contains one child control, the vapid Datagrid from Windows Forms which found its way into Silverlight.

11_DataGrid

For this example we auto-generate out columns to avoid the grunt-work of customizing them. I set my ItemsSource property to just "Binding" as opposed to "Binding [Some Property to bind to]" What this means is that my Datagrid will be bound to whatever object happens to be assigned to my local Data Context.


Walkthrough - Search by Name Against an XML Data Store

The user clicks on the radio button "Search by Name", types in a search term, and clicks "Search". Once again this triggers our event handler. This time we have a block to handle the search by name.

12_EventHandlerGetByName

But this time, we'll have our factory return a different type of Inventory Manager, one that goes against an XML data store instead.

13_FactoryReturnsXMLDataStore


Our Products.xml data store resides within our project:

14_XMLDataProducts

Our XML Inventory Manager InventoryManagerXML implements the IInventoryManager interface.

15_InventoryManagerXMLClass

And here are the method implementations used to search the XML data and return a generic list of ProductInfo. The Products.xml file is loaded using our utility class (included within the project) and we use LINQ-to-XML to search the record we need. We use a helper method to hydrate a generic list of ProductInfo and then raise our event to signal to the UI that we're done.


16_InventoryManagerXMLMethods

Walkthrough - Search by Name Against a WCF Service

Finally, we'll change our factory one more time to use an Inventory Manager that calls a separate WCF Service.

17_FactoryReturnsInventoryManagerService

Setting up a WCF service for Silverlight
In a simpler scenario you can setup your WCF service on the same site that is hosting your application. This will allow your Silverlight client to automatically authenticate back to its parent site. But for demonstration I'm going to set this up on a separate web site.

18_SeparateWebsite

Make sure you specifically choose the type "Silverlight-enabled WCF Service" when adding your service. This is still a WCF service, but the difference is in the configuration setup. Silverlight only supports basic HTTP binding, so your web.config must be configured NOT to explicitly use wsHTTPBinding like most WCF services would be.


19_XMLForWCFService

I'm admittedly unclear on the full details on that last portion. I've only scratched the surface on the possibilities that WCF can provide through custom meta-data, but knowing what you need from WCF vs. trying to grasp it all has been a catch-22 for most developers. That's why there have been attempts to simplify connectivity for the most common situations.

I add a Linq-To-SQL provider against the renowned AdventureWorks database. And I only want 2 tables: Product and ProductInventory. I call it AdventureWorks.dbml.

20_LINQToSQL

My end-point service interface will contain 2 methods, one for searching by SKU and the other for searching by name. Here's the interface:

21_IInventoryManager_WebInterface

To populate an object to return, I use a POCO (Plain old C Sharp) class called ProductInfo to return to the caller:

22_ProductInfoClass

For this example here is the implementation of the method GetInventoryByName(). I use my LINQ-to-SQL generator to search for my item by name and hydrate a generic list of ProductInfo instances.


23_LINQToSQL

Finally I need one more thing to make this work. In a simple scenario where my Silverlight application is served through the same website that contains my service, the client-application automatically authenticates to its parent website. But if I'm on a different web-site, I need a way to expose my access to my client application. This is achieved through the file clientaccesspolicy.xml.

24_ClientAccessPolicyFile

25_ClientAccessPolicyXML

Without going into detail, this implementation basically states that anyone from any domain URI to access my web service. The clientaccesspolicy.xml file must reside within the root of your web application.


Back in my Silverlight Application, I need a reference to my WCF service:

26_ReferenceProxy

Then, my InventoryManagerService class returned from my InventoryManagerFactory can instantiate a service proxy and make the respective call to my web service.

27_RaiseCompletedMethod

Conclusion

This is just one of a plethora of hands-on labs available for Silverlight. My favorite part about this one is that it truly covers at a medium level everything needed from front-end to back-end to get a full application working. I can make a generalization that there are 2 main parts to getting a Silverlight application up and running full-cycle:

  1. The UI - which primarily consists of learning XAML. In no priority I can think of the following topics to learn:
    1. Customizing layout - using controls like Grid, StackPanel, and Canvas
    2. Styles - XAML has its own version of CSS. Styling has essentially become a "pattern", in which you separate your UI styling from the controls themselves and reference your style libraries
  2. The Data - Data in Silverlight is tricky because the compact framework has nothing like ADO.NET, and good-luck finding an ORM framework that will work directly with Silverlight on the front-end. No, instead you really need to separate your data-layer (IMO a good thing). You have 2 main ways you can do this:
    1. WCF web services - This is how we retrieved data in this lab, and just as well you would update data through clearly defined API calls to your web-service
    2. Silverlight RIA services or O-Data - These options are for dealing with data as an ORM service. When you use O-Data, you now open your entire entity-model to your Silverlight application, a good or bad thing depending on your environment
      1. Shawn Wildermuth has a great blog post on Silverlight Data Access options

Full source code can be downloaded here.

Next Time
Likely not the very next post, but in the near future I intend to cover the Model-View-View Model design pattern

Unit Testing Part 2 - "Test Driven Development is not like Grammar Driven Literature"

by bpanjavan 12. March 2011 14:12

Unit Test Crash Dummy

Contrarian Software Development asserts that some of the best known software development practices are really some of the worst, and that developers follow these practices just because they are best practices. The author has done a great job at summarizing the arguments against Unit Testing. Below are my responses to two of his posts on why "Unit Testing Sucks". I'm only repeating back his summary points so I recommend reading HIS posts before you read mine.


1st Post - Unit Testing Sucks - November 8th 2008

1. "You cannot test all possible permutations..."

Take a Step Back
I'm having a hard time understanding why this is such a strong argument. Realize that this is an argument against testing in general. You can further abstract that this is really saying "Because A is not a silver bullet, we should not do A.".

The Counter-Argument
"True, but A does bring enough value to warrant the cost of doing A.". In other words, if A adds more value than cost, you should do A. In business people terms this is called a positive return on investment, or margin.

Back to Unit Testing
So coming back to Unit testing the question is not whether Unit testing can test everything. OBVIOUSLY it cannot, because there is no tangible definition for everything. The question is "Does the cost and effort of Unit testing warrant investing in it?"

Moving Forward
Let us stop shooting down a solution because it is not a silver bullet. Let's stop saying "You can't possibly test ALL permutations" or "It won't solve ALL of our problems" because WE KNOW THIS, and the proponents of such solutions NEVER make such claims (although they tend to sound like they do).


2a. "Whatever you figure out needs to be tested is stuff you would code to work correctly anyway..."

How do you Code Something to Work Correctly?
So I'm coding our payment processor. It needs to apply payments to invoices and update the customer's balance, and age the customer's balance correctly. This needs to be tested just as the quote says. So ...how do I know that I coded it correctly? I test it. Whether it's through formal Unit testing, or running informal scenarios, code needs to be tested.

Take a Step Back
All stuff that needs to work correctly needs to be tested. Whether it's a US Airways evacuation exercise, or a Toyota braking test, somebody somewhere somehow needs to do at least SOME level of testing.

Moving Forward
Let's stop debating whether something needs to be tested, and start debating on the best way to test it, or if you prefer "prove that that it works correctly".


2a. "The bugs that have always showed up in the applications I’ve worked with are unanticipated combinations of user actions or circumstances..."

Sidetrack - Programming Analysis
I took a course at the University of Texas called Analysis of Programs. This is one of those logic courses that despite its name we did not write a single program. One of the topics was something I'll call "Programming by Permutation". It's where you take all the inputs of your program (for example all parameters of a method) and iterate through all probable (not necessarily possible) combinations when testing your program for correctness.

Back to Unit Testing
Despite pejorative theory, the above is not as time consuming and wasteful as it seems. If you have a Unit Testing framework that supports Data-Driven tests than this is easy to accomplish. I'm not talking about all possible permutations here, just the practical stuff that you can think of. Following on my example earlier, I can't just test a payment applying to an invoice. I need to test if the payment is a reverse payment, if there IS no invoice, if the payment amount exceeds the invoice amount, etc. These are several unlikely yet possible permutations.

Take a Step Back
These are ALL improbable and yet possible production scenarios. Most IT shops do not test this and that's how they end up with "unanticipated combinations of user actions or circumstances". IT shops do not purposely avoid testing these scenarios by choice. It's because they feel that the time it would take to (Unit) test them is not worth the time it takes to deal with them when they happen (and they do happen) in production.

Moving Forward
So if the time it took to test scenarios, even the "unanticipated combinations" WAS minimal, then there's no reason NOT to test these. So whether the benefit is big, or the effort is small, either one of these makes testing worth it!


3. "The most important part of the application is the user interface. That’s the part of the program that the user interacts with—programs, after all, are meant to be used by people and not by unit tests..."

It Depends…
A LOT of developers shoot down this one. It really depends. But even generally speaking I don't agree with this. I think the most important part of the application is the user experience, which does include the interface but is NOT limited to it.

It's the Data Stupid
If you're planting a Zen-Garden , yes the interface is the most important. But on an accounting, billing, transaction processing, line-of-business, e-commerce, bulk-data, resource management, workflow-based, search-taxonomy, content management system, the data is just as important if not more.

Define "important"
I don't personally agree with this, but it's true. To most companies, money is important. Bad user interfaces cause frustration, but not loss of money (at least not directly). Bad data can cause SO many problems. It can cause lawsuits, which cost money. It can lower billing accuracy, customer satisfaction, transaction timeliness, etc. All of which cost money.

Moving Forward
Priority #1 is making your program work and work correctly. Making it look cool is unfortunately not a priority.


4. "Writing unit tests is incredibly boring. Unit testing advocates might respond, “well that’s too bad, it’s necessary.” But it’s not necessary, and forcing your programming staff to do really boring tasks results in sapping their motivation thus lowering their productivity..."

Arguing on the Subjective
I can't argue against this one, because it's an opinion. I can only vouch from experience. We introduced Unit testing at my job and got raving feedback. I'm paraphrasing but the #1 response I got was along the line of "I love that when I need to make a correction, I can just hit F5 (re-run) and watch the scenario play again!". I've had good experiences. Joe Contrarian has had bad experiences. We're technically at a stand-still .

It's all Relative
Setting up Unit tests might be boring and unfulfilling. But relatively speaking, what is more boring? Having to manually mock up scenarios EVERY time you work on code? Having to deal with the Production issues that result from not testing your code? Yes, testing can be more boring than running wild with new development. That's like saying going fishing is more fun than maintaining your fishing equipment.

Moving Forward
Testing and verification are necessary evils, and done wrong they can be time consuming. THAT is what Unit testing is about. Making testing easy, repeatable, and NOT time consuming, thereby allowing you to spend less time testing and more time developing. Sound like fun?


5. "The biggest problem I have with unit testing is its reliance on design patterns concepts. The standard test driven software project requires you to create various application “layers,” with each layer being an interface that receives other interfaces…."

Blaming the Enforcer
"I hate speed limits. They cause me to drive slower than I can, and therefore I get there slower when I know I can get there faster" Unit Testing does force you to put more thought into encapsulation. It's true that if I didn't have to encapsulate, use good object-oriented programming practices, write Unit tests, or go through this "QA process", I could more quickly write a successful program. Except that it wouldn't be successful. The fact that Unit testing enforces good coding practices is not a flaw, but a fringe benefit!

Taking a Step Back
But he brings up a good point. If I'm working on a program that really doesn't warrant encapsulation, then that's most likely the type of program that doesn't warrant Unit testing either. these apps are often called "throw-away" applications.

Moving Forward
Unit testing is not a silver bullet or a golden hammer. Systems that don't need to be correct, throw-away apps, and systems that are ALL UI based do not need Unit testing. But the remaining 97% of systems out there DO warrant some form of formal testing.


2nd Post - Unit Testing is Like… - November 7th 2008

"Test driven development is like grammar driven literature"

What it means
If literature was bound by proper semantics, its potential would be limited. This is because literature is not about grammatical correct-ness. It is about elegance, beauty, and open interpretation. Grammar would in effect, ruin literature just as Unit testing would ruin good software.

The Flaw
Except that software MUST be bound. It is bound by logical correct-ness. When accounts receivable complains that their customer balances are screwed up because of buggy software, I can't approach them as a literary programming poet and convince them that there are multiple interpretations of correctness, because there are NOT.

A Better Analogy
Much like legal documentation, software must avoid room for interpretation. 1+1 must equal 2. There must be a clearly defined speed limit. There must be a classification for what constitutes a planet and that is why we have exactly nine eight planets in our solar system and no more or less.


"Test driven development is like grammar driven legislation"

Tags: , ,

Commentary | Unit Testing

Unit Testing Part I - Why we don't Unit Test

by bpanjavan 28. December 2010 11:06

PassFaile

This is the first in a series of research, analysis, and examples I've been working on. I went to Wikipedia to get an idea of "what the people" thought about Unit Testing, and I noticed that this debate is a never-ending one. For the record I'm an extreme proponent of Unit Testing. If I could change 2 things about the world, the 2nd would be to force all IT shops to adopt Unit Testing (The 1st being the elimination of poverty). Before I go any further let's be clear on what Unit Testing is. This is straight from Wikipedia:

Unit Testing (Defn.)
The practice in which individual bits of source code are tested to determine if they are "fit for use".
 • A Unit is the smallest testable part of an application
 • Tests are created by Programmers or White Box Testers

My Thesis: The #1 reason IT shops don't Unit Test is because the value it adds is only obvious to developers (a.k.a. the people who don't control the schedule / budget). Beyond that the value to the business is not obvious, thus making the investment trivial.

The remainder of this post consists of excerpts from the same article mixed with my own commentary.


The Goal of Unit Testing

By isolating each part of a program and proving them correct, the result is that it finds problems early in development cycle

So by finding them early, you don't find them later. Finding them later is often more costly because:
 • It's likely already affected production data
 • You have to clean up after it AND re-create the scenario
 • Developers spend unplanned man-hours on fixing issues
 • Arguably the most crucial, your users / customers will have noticed it


Benefits


Facilitates Change
You can safely regression test after re-factoring to quickly identify changes that cause fault and fix

Value:
 • To Business: 3/5 - Great in long term, but indeterminable
 • To Developers: 5/5

This is THE #1 reason why we Unit Test. Unfortunately the most difficult to justify because:
  • The ROI (Return on Investment) is obscure to all but developers, and even then your team is likely to be in contention, not because they disagree per-say, but because it requires some discipline and coordination, and most development teams behave like dysfunctional families
  • Your stakeholders aren't concerned with future change, they're concerned with accomplishing an immediate goal. And unless an IT manager is truly spearheading the project, there's no room on the schedule for regression.

Simplifies Integration
Because individual parts are tested first before integration

Value:
 • To Business: 1/5
 • To Developers: 2/5

Probably the weakest argument of the bunch. It is true, but problems with integration testing can come from so many different factors (i.e. miscommunication, lack of detailed requirements, lack of coordination), I wouldn't tout this as a benefit of Unit Testing because it can only lead to unrealistic expectations.

Integration Testing has and will always rely heavily on human testing

But at some point Integration Testing is no longer "testing", but more "verification". I digress, because that's a whole other topic (i.e. blog post) all together.

Documentation
The Unit Test suite acts as a "living documentation". By contrast, narrative documentation is more susceptible to drifting from original implementation (becoming outdated)

Value:
 • To Business: 0/5
 • To Developers: 4/5

I didn't realize this one until I was a new addition to an existing team that actually did Unit Testing. This is especially true, and yet once again hard to justify because this is ONLY useful to developers and it adds no "business value" to the project whatsoever. My favorite part about this is that by design, the "documentation" that is your Unit Tests are guaranteed to be accurate. This doesn't receive a perfect score on developer value because it requires very well organized (i.e. readable) Unit Tests to be effective.


Design
Unit tests can take the place of formal design. Each test is essentially a design element, specifying classes / methods / behavior to be implemented

Value:
 • To Business: 0/5
 • To Developers: 5/5

Maybe it's just diction, but I really wouldn't go as far as to say "take the place of formal design". I do believe however that it does lead to better design, because it forces you to plan ahead. It's a fundamental characteristic of TDD (Test Driven Development) that writing test cases ahead of time forces you to design before you code. In this case, you literally "design by use-case". Planning is hard work, and in software development, "weeks of coding can save you hours of planning".

Unit Testing Limitations


The following point out the limitations of Unit Testing and thus can be considered arguments against the adoption of Unit Testing

Unit Testing can NOT test EVERY error in a program
True, but just because a solution doesn't find ALL problems doesn't mean you don't do it. By that logic, we shouldn't plan at all because no plan accounts for all scenarios!

By definition, Unit Testing ONLY tests the units themselves. Unit Testing:
 • Cannot test integration errors
 • Cannot test performance
These are both true statements. But I notice that these aren't so much arguments against Unit Testing, but simply a reinforcement that Unit Testing is not a silver bullet to all problems. I don't think automakers ever decided NOT to implement safety features because it wouldn't solve the problem of wreck-less drivers.

Unit Testing can show the presence of errors, not their absence
This is the classic "You don't know what you don't know" argument. Known pejoratively as the Black Swan Theory. It's true that stuff will happen that you just can't predict, but it's also true that the more you try to predict the things that will happen, the less that you "won't know what you don't know". And when those things DO happen and you have a well maintained suite of Unit Tests, you can augment those tests with the previously unforeseen issue to ensure that it never happens again.

Software is combinatorial. Ex: A boolean function can return true and false. As a result the developer must write 3-5 lines of test code for every piece of ACTUAL code.
The added benefit may not be worth so much effort

True. And because the benefits are indeterminable, how do you quantify that the benefits outweigh the time of investment? The biggest benefit of being overly thorough in your Unit Tests comes when developers and business personnel approach you and ask "Hey what happens when I do A, and then a B happens, but I didn't do C?". If you have a test suite handy, you can VERY quickly put this scenario together and keep it as part of your test suite. Eventually you'll sound like an I-Phone commercial:
 • User: "Have you thought of scenario A?"
 • Developer: "There's a test for that"
 • User: "And scenario B?"
 • Developer: "There's a test for that"
 • User: "And scenario X / Y / Z ?"
 • Developer "There's a test for that too."


The Code for unit test can be buggy too
Not just buggy, but if the test is just as wrong as the code, then the test might give you a "false negative". A co-worker of mine said this can be mitigated by ensuring that as you write your Unit Tests, you make them fail BEFORE you make them pass. If you're doing TDD (Test Driven Development), this is guaranteed to be true.
Also, if the entropy of your Unit Tests gets to the point that it can be potentially buggy, that's an implication that they are not broken out enough.

Unit Testing requires discipline in:
 • Maintenaince
 • Thorough Code Coverage
 • Developing a process of review and fixing broken tests

If "indeterminable benefit" is the #1 reason not to Unit Test, this is the #2 reason. It is yet another process that developers must follow. The aversion toward Unit Testing today is further exacerbated by the pitfalls that IT shops have encountered when failing to "do it right". In short, Unit Testing is something that must be done well in order for it to be remotely effective. I do concede in saying that if you're not going to invest in doing it right, it probably is NOT worth it. It doesn't help either that the best tools for Unit Testing in Visual Studio 2010 are only available in the Ultimate Edition.

Tags: , , ,

Unit Testing | Commentary

Garbage Collection Analysis - Part II

by bpanjavan 10. December 2010 17:15

Transformers Constructicons

This time we'll use a WPF Application to showcase the difference between deleting in-memory objects and garbage collecting on memory. I have a basic ListBox container bound to a collection of my class: ByteContainer.
My CRUD operations allow me to allocate mega-bytes of data to my desire.
To the right is my command to perform garbage collection. This button executes the method GC.Collect().
Also notice the bottom-right is a dynamic snapshot of my total process memory, which will essentially be the value of PerfMon at any time.

App Explained

The ByteContainer class is a very simple class which serves merely as a container around a byte array.

    namespace GarbageCollectionAnalysis.Surface
    {
        public class ByteContainer
        {
            private byte[] _data;

            public string Name { get; set; }

            /// <summary>
            /// Returns size in megabytes
            /// </summary>
            public int Size
            {
                get
                {
                    if (_data == null)
                        return 500;
                    return _data.Length / 1024 / 1024;
                }
            }

            public int DisplayLength
            {
                get
                {
                    // 150 is the max length of the rectangle
                    // 150 / 1000 = displaylength / size
                    return (int)((150D / 1000D) * Size);
                }
            }

            // max megabytes is 1000
            public void SetSizeInMegabytes(int size)
            {
                int sizeInBytes = size * 1024 * 1024;
                _data = new byte[sizeInBytes];
            }
        }

    }
    

I'll run 2 scenarios , the first of which I control the Garbage Collection

Scenario 1:
 • Allocate 500 MB
 • Allocate 200 MB
 • Allocate 300 MB
 • Delete the 500 MB and 300 MB blocks
 • Call Garbage Collection

Here are the results of Scenario I. Notice that when I deleted my memory blocks, in-process memory did NOT reduce until I called Garbage Collection

PerfMon 1st Run

Scenario 2:
 • Allocate 500 MB
 • Allocate 500 MB
 • Delete 1000 MB
 • Allocate 100 MB 10 times in a row

Here are the results for Scenario 2. I didn't call Garbage Collection myself, but because the first 1000 MB was already deleted, Garbage Collection made room for my new data only when it needed to.

PerfMon 2nd Run

Here are the results for Scenario 2. I didn't call Garbage Collection myself, but because the first 1000 MB was already deleted, Garbage Collection made room for my new data only when it needed to.

Full source code can be found HERE