jakelite

Encrypting Passwords in Java Property Files

2009-07-14T07:59:00Z

Configuration is an essential part of every software development project. Among other things, we store sensitive information such as database, ftp and other credentials in these configuration files. This may not be an issue for a small company. But if you're working for a large bank, this becomes a major audit issue. Hence, i give to you this post about encrypting passwords in Java Property files.

PropertyFile Templates

We need a way to externalize sensitive information from the property files and put a place holder for them. These place holders will be replaced when the build process gets triggered. For this we will be using StringTemplate.

I will not go into details on how one uses StringTemplate. But below is an an example configuration file with the said place holders.

//PropertyFile.st

ftp.username=$ftpUsername$
ftp.username=$ftpPassword$

Password File

As you can see, the property file template expects the variables $ftpUsername$ and $ftpPassword$ to be replaced by the actual value. This variables are stored in another property file. Lets just call it Passwords.prop and its content is shown below.

//Passwords.prop

ftpUsername=jake.stateresa
ftpPassword=omgwtfbbq!!!!

The Goal

The goal of this exercise is to be able to generate a property file based on the template Property.st and have the stubs replaced with the values in Passwords.prop. All variables ending with Password should be encrypted in the generated property file.

//Target.prop

ftp.username=jake.stateresa
ftp.username=uznecctWVkMqB9eMxnIJ7Q==

The source code below demonstrates the process of loading the Password file to a Properties object, encrypting all values in the Password file whose key ends with "Password", loading the template and passing the Properties object to the setAttributes method and writing the output of the template operation to a file.

1:  Properties passwordFileProperties = new Properties();  
2:  passwordFileProperties.load(new FileInputStream(passwordFilePath));  
3:  Set<String> stringPropertyNames = passwordFileProperties.stringPropertyNames();  
4:  for (String propertyName : stringPropertyNames) {  
5:       if(propertyName.endsWith("Password")){  
6:            String propertyValue = passwordFileProperties.getProperty(propertyName);  
7:            passwordFileProperties.setProperty(  
8:                      propertyName,   
9:                      encryptionHelper.encrypt(propertyValue));  
10:       }  
11:  }  
12:    
13:  StringTemplate template =   
14:       templateGroup.getInstanceOf(templateName);            
15:  template.setAttributes(passwordFileProperties);  
16:    
17:  FileWriter outputStream = new FileWriter(outputFilename);  
18:  BufferedWriter outputWriter = new BufferedWriter(outputStream);  
19:  outputWriter.write(template.toString());         
20:  outputWriter.close();

In your java program, you can read/decrypt the properties from the output file using the code below. In reality you would have used the values on an FtpClient class like the one commented out below.

1:    
2:  Properties encryptedProperties = new Properties();  
3:  encryptedProperties.load(new FileInputStream("Output.prop"));  
4:  String username = encryptedProperties.getProperty("ftp.username");  
5:  String encryptedPassword = encryptedProperties.getProperty("ftp.password");  
6:  String decryptedPassword = new EncryptionHelper().decrypt(encryptedPassword);  
7:    
8:  //FtpClient ftp = new FtpClient("localhost", 22, username, decryptedPassword);  
9:  System.out.print(String.format("Username: %s; Password: %s", username, decryptedPassword));  
10:

Final Notes

You can find the source code for this project here. Please make sure to download the latest StringTemplate libs from this location when you attempt to execute the project.

Feel free to comment on the given approach. This is my first post in Java so be gentle.

Cheers!

How to: Add CData or Text in a ConfigurationElement

2009-03-22T11:22:00Z

I was working on a MsScriptControl based ETL project. One of its requirements is to be able to use vbscript functions in the transformations it performs. It is fairly easy to hard-code functions then register them during start-up all at the expense of maintainability. I guess the better side of me prevailed and I opted for a more configurable solution.I looked around and found there was no available solution on how to do this; hence this post.

The Target Configuration

For the purpose of illustration, let’s pretend I need to create a system which required me to send emails and store its template and mail server details in the configuration section. The xml below shows the target configuration.

<mailConfiguration server="localhost" port="25">
  <mailTemplates>
    <mailTemplate 
        name="template1"  
        sender="user@localhost">
      <receivers>
        <contactInfo 
          name="Jake Sta. Teresa" 
          email="jake@jakelite.com" />
      </receivers>
      <body>
        <![CDATA[
        Dear Sir,
 
        This is a system generated message.
 
        Thanks,
        OMGWTFBBQ!!!
        ]]>
      </body>
    </mailTemplate>
  </mailTemplates>
</mailConfiguration>

And reading this configuration section will result in something like this.

Using the CDataConfigurationElement and CDataConfigurationPropertyAttribute Classes

I wouldn’t bore you with the details of working with .Net 2.0 configuration classes. I have another post dedicated to that.

The meat of this post is in the <body> element of the configuration. Its code representation is written below. The highlighted code emphasizes on the classes that are put into play in order to read a cdata or text node from a configuration element.

public class MessageBodyElement
    : cdataElement
{
    [ConfigurationProperty("content", IsRequired = true, IsKey=true)]
    [CDataConfigurationProperty]
    public string Content
    {
        get { return (string)(base["content"]); }
        set { base["content"] = value; }
    }        
}

The CDataConfigurationPropertyAttribute Class

The CDataConfigurationProperty attribute is only allowed to decorate properties. The code for it can be seen below.

[AttributeUsage(AttributeTargets.Property)]
public sealed class CDataConfigurationPropertyAttribute
    : Attribute
{
}

The CDataConfigurationElement Class

The class CDataConfigurationElement is a subclass of ConfigurationElement. In its constructor, it ensures the following:
•    A property decorated with a CDataConfigurationProperty attribute must be of type string. This will spare us of the trouble of conversion.
•    A property decorated with a CDataConfigurationProperty attribute must be decorated by a ConfigurationProperty attribute. This is only to play nice with the .Net 2.0 configuration classes. Subclassing ConfigurationProperty would have been a better approach but unfortunately its sealed.
•    A class must have only one property decorated with a CDataConfigurationProperty. This is fairly self explanatory. I mean, how do you know which CData element maps to which property?
•    A class which already has a property decorated with a CDataConfigurationProperty can not contain other properties that are decorated by ConfigurationProperty and whose type is subclass of ConfigurationElement. It looks like an orphaned CData element. Just look at this.

<mailConfiguration server="localhost" port="25">
  <mailTemplates>
    <mailTemplate 
        name="template1"  
        sender="user@localhost">
      <receivers>
        <contactInfo 
          name="Jake Sta. Teresa" 
          email="jake@jakelite.com" />
      </receivers>
      <![CDATA[
        Dear Sir,
 
        This is a system generated message.
 
        Thanks,
        OMGWTFBBQ!!!
      ]]>      
    </mailTemplate>
  </mailTemplates>
</mailConfiguration>

After all these validations are performed, the name defined in the corresponding ConfigurationPropertyAttribute of the CDataConfigurationPropertyAttribute is stored in the variable _cDataConfigurationPropertyName.

Overriding SerializeElement and DeserializeElement Methods

In order to read, write to the CData element, we need to override the SerializeElement and DeserializeElement methods. If no property of the CDataConfigurationElement subclass is decorated with a CDataConfigurationProperty attribute, the default serialization/deserialization logic is used. Below are the overrides used in the CDataConfigurationElement:

protected override bool SerializeElement(
    System.Xml.XmlWriter writer, 
    bool serializeCollectionKey)
{
    bool returnValue;
    if (string.IsNullOrEmpty(
        _cDataConfigurationPropertyName))
    {
        returnValue = base.SerializeElement(
            writer, serializeCollectionKey);
    }
    else
    {               
        foreach (ConfigurationProperty configurationProperty in 
            Properties)
        {
            string name = configurationProperty.Name;
            TypeConverter converter = configurationProperty.Converter;
            string propertyValue = converter.ConvertToString(
                    base[name]);
 
            if (name == _cDataConfigurationPropertyName)
            {
                writer.WriteCData(propertyValue);
            }
            else
            {
                writer.WriteAttributeString("name", propertyValue);
            }
        }                               
        returnValue = true;
    }
    return returnValue;
}
 
protected override void DeserializeElement(
    System.Xml.XmlReader reader, 
    bool serializeCollectionKey)
{
    if (string.IsNullOrEmpty(
        _cDataConfigurationPropertyName))
    {
        base.DeserializeElement(
            reader, serializeCollectionKey);
    }
    else
    {
        foreach (ConfigurationProperty configurationProperty in 
            Properties)
        {
            string name = configurationProperty.Name;
            if (name == _cDataConfigurationPropertyName)
            {
                string contentString = reader.ReadString();
                base[name] = contentString.Trim();
            }
            else
            {
                string attributeValue = reader.GetAttribute(name);
                base[name] = attributeValue;
            }
        }
        reader.ReadEndElement();
    }           
}

Conclusion

Well there you have it, A working implementation of how to include the text or cdata elements in a ConfigurationElement class. Sources are available here.

MethodInfo.Invoke vs MethodCall Performance on DynamicMethods

2009-02-16T14:55:00Z

I posted a question on cvega’s article on using CIL getters and setters to speed up the execution of heavily reflection-based code asking which one will perform better using DynamicMethod.Invoke or using DynamicMethod.CreateDelegate then calling that delegate. On which he promptly replied. But I tonight is one of those nights where I can't get myself to sleep so I decided to satisfy my curiosity on this matter.

So I immediately opened my trusty VS.Net IDE and started working on it. The following are the steps I made to come up with the benchmark results.

I declared two generic delegates:

delegate P GetPropertyDelegate<O, P>(O objectInstance);
delegate void SetPropertyDelegate<O, P>(O objectInstance, P propertyValue);    

I didn't want to change his existing methods so I decided to add some of my own to the existing classes. They are shown below:

    1 //ADDED to PropertyGetter
    2 public static P GetPropertyGeneric<O, P>(
    3              O instance, string propertyName)
    4 {
    5     GetPropertyDelegate<O, P> getDelegate = 
    6         (GetPropertyDelegate<O, P>)
    7             GetPropertyGetterMethod(instance, propertyName).
    8                 CreateDelegate(typeof(GetPropertyDelegate<O, P>));
    9     return getDelegate(instance);
   10 }
 

    1 //ADDED TO PropertySetter
    2 public static void SetPropertyGeneric<O, P>(
    3      O instance, string propertyName, P propertyValue)
    4 {
    5     SetPropertyDelegate<O, P> setDelegate = 
    6         (SetPropertyDelegate<O, P>)
    7             GetPropertySetterMethod(instance, propertyName).
    8                 CreateDelegate(typeof(SetPropertyDelegate<O, P>));
    9     setDelegate(instance, propertyValue);
   10 }

For the purpose of illustration, I created a class called MyClass which has one string Property MyProperty.

    1 public class MyClass
    2 {
    3     private string myVar;
    4 
    5     public string MyProperty
    6     {
    7         get { return myVar; }
    8         set { myVar = value; }
    9     }
   10 }

Then used the following code in my main method.

    1 MyClass c = new MyClass();
    2 
    3 //make sure the creation of the methods 
    4 //does not get computed as part of benchmark
    5 PropertySetter.SetPropertyValue(c, 
    6     "MyProperty", "OMGWTFBBQ!!!!");
    7 string q = PropertyGetter.GetPropertyValue(c, 
    8     "MyProperty").ToString();
    9 
   10 int limit = 1024 * 1024 * 2;
   11 Stopwatch sw = new Stopwatch();
   12 
   13 Console.Write("GetProperty: ");
   14 sw.Start();
   15 for (int i = 0; i < limit; i++)
   16 {
   17     string s = PropertyGetter.GetPropertyValue(c, 
   18         "MyProperty").ToString();
   19 }
   20 sw.Stop();
   21 Console.WriteLine(sw.Elapsed);
   22 
   23 sw.Reset();
   24 Console.Write("GetPropertyGeneric: ");
   25 sw.Start();
   26 for (int i = 0; i < limit; i++)
   27 {
   28     string s = PropertyGetter.
   29         GetPropertyGeneric<MyClass, string>
   30             (c, "MyProperty");
   31 }
   32 sw.Stop();
   33 Console.WriteLine(sw.Elapsed);
   34 
   35 sw.Reset();
   36 Console.Write("SetProperty: ");
   37 sw.Start();
   38 for (int i = 0; i < limit; i++)
   39 {
   40     PropertySetter.SetPropertyValue
   41         (c, "MyProperty", "OMGBBQFTW!!!");
   42 }
   43 sw.Stop();
   44 Console.WriteLine(sw.Elapsed);
   45 
   46 
   47 sw.Reset();
   48 Console.Write("SetPropertyGeneric: ");
   49 sw.Start();
   50 for (int i = 0; i < limit; i++)
   51 {
   52     PropertySetter.
   53         SetPropertyGeneric<MyClass, string>
   54             (c, "MyProperty", "OMGBBQFTW!!!");
   55 }
   56 sw.Stop();
   57 Console.WriteLine(sw.Elapsed);
   58 
   59 Console.ReadKey(false);

Conclusion

The following is the result of executing the program.

As it turns out CreateDelegate executes faster than Invoke. I think the reason is that the Invoke call results in something like a MethodInfo.Invoke while the delegate call is more analogous to an actual method call.

Of course the performance difference is negligible in scenarios which does not require calling the DynamicMethod 1024*1024*2 times in one go. This article is just meant to satisfy the curiosity of a man who finds it difficult to sleep at night. Sources can be downloaded from here.

Threading Patterns: Producer-Consumer Pattern Examples

2009-02-03T10:42:00Z

This is the third installment to my series of posts about threading patterns.

Today we will look into familiar scenarios where we can apply the producer-consumer pattern particularly applications of the ThreadedQueueBase<T> class.

Usage Pattern

Note that the threaded queue classes that I provided in the last article allows you to have an in memory queue of work items. Because of this nature the normal usage pattern is to start the thread during application start up. And close it during application shutdown. It also contains two Stop methods. The signatures are as follows:

    1 public new void Stop();

    2 public void Stop(PendingItemAction pendingItemAction);

The parameter-less Stop method is just a wrapper to a call to the second Stop method passing the parameter PendingItemAction.ProcessPendingItems

The second Stop method accepts a parameter of type PendingItemAction. The definition for this enumeration is found below. Calling this method with the PendingItemAction.ProcessPendingItems as parameter will cause the queue to process all pending items before exiting its thread-proc. While passing the parameter PendingItemAction.AbortPendingItems will cause the thread to exit immediatel

    1 public enum PendingItemAction
    2     {
    3         ProcessPendingItems,
    4         AbortPendingItems
    5     }

File Copying

We will now be revisiting the file copying mechanism that we have been working on for some time now. We will be modifying it in order to be able to use the ThreadedQueueBase<T> class.

    1 public class CopyFileQueue : ThreadedQueueBase<CopyInfo>
    2 {
    3     Logger _logger;
    4     public CopyFileQueue(Logger logger)
    5     {
    6         _logger = logger;
    7     }
    8 
    9     protected override void ProcessItem(CopyInfo item)
   10     {
   11         //check if the user called Stop
   12         if (StopRequested)
   13         {
   14             _logger.Log(LogLevel.Information, "User called Stop.");
   15             _logger.Log(LogLevel.Information, 
   16                 "Terminating thread while copying directory '{0}'.",
   17                 item.Source);
   18             return;
   19         }
   20 
   21         _logger.Log(LogLevel.Debug, 
   22             "Checking if '{0}' exists...", item.Destination);                
   23         if (!Directory.Exists(item.Destination))
   24         {
   25             _logger.Log(LogLevel.Debug, 
   26                 "Creating '{0}'.", item.Destination);                
   27             Directory.CreateDirectory(item.Destination);
   28         }
   29 
   30         _logger.Log(LogLevel.Information, 
   31             "CopyFiles from '{0}' to '{1}' {2}...",
   32             item.Source,
   33             item.Destination,
   34             item.Recursive ? "recursive" : "non-recursive");
   35 
   36         foreach (string file in
   37             Directory.GetFiles(item.Source))
   38         {
   39             string destination = Path.Combine(
   40                 item.Destination,
   41                 Path.GetFileName(file));
   42             _logger.Log(LogLevel.Information, 
   43                 "Copying file from '{0}' to '{1}'...",
   44                 file,
   45                 destination);
   46 
   47             _logger.Log(LogLevel.Debug, 
   48                 "Copying file from '{0}' to '{1}'...", 
   49                 file, 
   50                 destination);                            
   51             File.Copy(file, destination, item.Overwrite);
   52         }
   53 
   54         if (item.Recursive)
   55         {
   56             foreach (string directory in
   57                 Directory.GetDirectories(item.Source))
   58             {
   59                 string destination = Path.Combine(
   60                     item.Destination,
   61                     Path.GetFileName(directory) //get the directory 
   62                     );
   63 
   64                 _logger.Log(LogLevel.Debug, 
   65                     "Copying directory from '{0}' to '{1}'...", 
   66                     directory, 
   67                     destination);
   68                 EnqueueItem(
   69                     new CopyInfo(
   70                         directory,
   71                         destination,
   72                         item.Recursive,
   73                         item.Overwrite));
   74 
   75             }
   76         }            
   77     }
   78 }

As it turns out, not a lot is needed to be changed in the file copying method in order to take advantage of the new class. But this approach now en-queues new CopyInfos to be copied instead of actually recursing the directory structure. This kind of coding is an example of a non-recursive tree traversal.

Logging

The queuing mechanism introduced in the producer-consumer article can be used to create a Logger. The logger will asynchronously accept inputs. And depending on the configuration the log processor thread can write the entries to a backing store like a text/xml file or database.

LogLevel

Each log entry should specify a log level. This will indicate whether or not the log entry is created for Information, Warning, Debugging or Error Notification purposes.

    1 [Flags]
    2 public enum LogLevel
    3 {
    4     Information = 1,
    5     Debug = 2,
    6     Warning = 4,
    7     Error = 8,
    8     LogAll = Information | Debug | Warning | Error
    9 }

LogEntry

Each log entry will contain a log level, the date of creation, the log message. This class is shown below.

    1 public class LogEntry
    2 {
    3     private DateTime _dateCreated;
    4     private string _message;
    5     private LogLevel _logLevel;
    6 
    7      public LogEntry(LogLevel logLevel,        
    8         string message,
    9         params object[] args)
   10         : this(logLevel, message)
   11     {
   12         _message = string.Format(message, args);
   13     }
   14 
   15     public LogEntry(LogLevel logLevel,            
   16         string message)
   17     {
   18         _dateCreated = DateTime.Now;
   19         _message = message;
   20         _logLevel = logLevel;            
   21     }
   22 
   23     public DateTime DateCreated
   24     {
   25         get { return _dateCreated; }        
   26     }
   27 
   28     public string Message
   29     {
   30         get { return _message; }        
   31     }
   32 
   33     public LogLevel LogLevel
   34     {
   35         get { return _logLevel; }        
   36     }
   37 }

Logger

I will not bore you with details of file IO. If you need to have a look the sourcecode is available. The file logging mechanism featured in the sample code contains features such as rolling file mechanism, deleting files which are greater than a certain number of days old and configuring setup items like Log path, Filename, and date format to use.

Final Notes

The examples provided here just scratches the surface on what you can do with the ThreadQueueBase<T> class. I have packaged the codes in my last articles into a single VS2005 solution. Please feel free to incorporate my threading base classes in your own projects and send me an email about the interesting stuff that you create from it.

If you have any inquiries regarding the use of the threading base classes just let me know and I will be happy to answer them.

Code samples can be found here.

Cheers!

Threading Patterns: The Producer-Consumer Pattern

2009-01-12T06:03:00Z

This is the second installment to my series of posts about threading patterns.

In the previous article, we looked into the worker thread pattern. It models the concept of thread-per-message by creating one thread per instruction/message pair. This approach is effective in achieving some level of parallelism, but suffers the overhead of thread creation and clean-up each time a worker is spawned.

Overview

Today we will look closely on the producer-consumer pattern using .Net threading. This is done by having a producer identify the data to be processed and adding it to a to-do list. The consumer on the other hand looks into the to-do list, takes an item from it and performs the instruction on that item. Below is an image showing this relationship.

A consumer can also act as producer when its output is stored into another consumers’ to-do list. This execution will result in pipelined execution. It also models the concept of thread-per-instruction. Thread-per-instruction decouples the data to be processed from the actual execution of instructions performed on the data.

For performance reasons, one can also create multiple producers/consumers each accessing a shared queue of work items. This approach can only do so much to increase performance. Because like the worker thread approach, it suffers from the overhead of thread creation and clean-up. Also, the shared queue becomes a point of contention. You might end up with a bunch of threads who most of the time is sitting idle waiting for the queue synchronize.

My advice is tune the number of threads every time you try this approach. And know when you are already getting diminishing returns.

Some Hand-Waving

I do not condone doing the third approach simply because I believe it is not the right tool for the job. Hence, I will introduce you to the ThreadedQueueBase<T> class. This class ties the concept of the queue to the consumer thread. Because there will be one consumer per queue, this will allow us to write programs that run like images 1 and 2 and the multiple producer version of 3.

The ThreadedQueueBase<T> Class

Our friends in the Java world have the BlockingQueue. Too bad for us .Net doesn’t have a built-in one. In this section, we will extend the WorkerThreadBase from the previous article and create a class that contains an AutoResetEvent named _itemArrived and a Queue<T> named _itemQueue.

    1 private AutoResetEvent _itemArrived;
    2 private Queue<T> _itemQueue;

When the Work method (the thread method) starts, we will do the following:

   1. wait for the signal that an item has arrived
   2. continuously de-queue and process the items while it is not empty
   3. if the Stop or Dispose method has been called, exit the thread else go to step 1.

    1 protected override void Work()
    2 {
    3     do
    4     {
    5         _itemArrived.WaitOne();
    6         do
    7         {
    8             if (_itemQueue.Count > 0)
    9             {
   10                 lock (((System.Collections.ICollection)
   11                     _itemQueue).SyncRoot)
   12                 {
   13                     T item = _itemQueue.Dequeue();
   14                     if (item != null)
   15                     {
   16                         ProcessItem(item);
   17                     }
   18                 }
   19             }
   20         } while (_itemQueue.Count > 0);
   21         Thread.Sleep(0);
   22     } while (!Disposing && !StopRequested);
   23 }

We will also expose a public method called EnqueueItem which will accept the item, add it to the queue and send the signal to the Work method that an item has arrived.

    1 public void EnqueueItem(T item)
    2 {
    3     ThrowIfDisposedOrDisposing();
    4     _itemQueue.Enqueue(item);
    5     _itemArrived.Set();
    6 }

And lastly, we will create an abstract method called ProcessItem. This method is called by the Work method passing one item from the queue. This method should contain the instructions for processing one item.

    1 protected abstract void ProcessItem(T item);

Final Words

Today, we learned more about the ThreadedQueue class. And howit can perform just like the BlockingQueue except that it does not block the incoming items whenever the queue is full.

Next time, we will look into applying the ThreadedQueue class into familiar scenarios like logging or file copying.

Examples are located here. And source codes are located here.

Updates

12-January-2009: Reposted! Changed the title to "Producer-Consumer". Sorry about that. =D

10-February-2009: Added links to the samples and sources

5 Tips for Debugging Multi-threaded Code in Visual Studio.Net

2009-01-09T19:51:00Z

I have been talking about threading and threading patterns for a while now but somehow fell short on how to debug them. Today we will look into tips as well as tools available in the Visual Studio IDE that enables us to effectively debug multi-threaded applications like non-threaded ones.

Tip #1: Always Name Your Threads

Whenever you create a thread, remember to always give a name to it. You can give a thread a name by assigning a string value to the Name property. Below is a sample code on how to give a thread a name as taken from the constructor of WorkerThreadBase.

    1 protected WorkerThreadBase(string name,
    2     ThreadPriority priority,
    3     bool isBackground)
    4 {
    5     _disposing = false;
    6     _disposed = false;
    7     _stopping = new ManualResetEvent(false);
    8     _stopped = new ManualResetEvent(false);
    9 
   10     _workerThread = new Thread(threadProc);
   11     _workerThread.Name = name == null ? GetType().Name : name;
   12     _workerThread.Priority = priority;
   13     _workerThread.IsBackground = isBackground;
   14 }

Tip #2: Use the Threads Debug Window

You can open this window by clicking Debug->Windows->Threads or Ctrl+D, T in a debug session. In this window you can see the number of threads open for your application. You can also switch between threads by double clicking on a line in the table.

Below is picture of the Threads Debug Window

Tip #3: Use the Call Stack Debug Window

You can open this window by clicking Debug->Windows->Threads or Ctrl+D, C in a debug session. In this window you can see the different methods that got called before reaching the current breakpoint location. Each line in this table corresponds to a Stack Frame. You can double click a line in this window to view the corresponding method in that stack frame.

Below is picture of the Call Stack Debug Window

Tip #4: Use the Debug Location Toolbar

You can open the debug location toolbar by clicking View->Toolbars->Debug Location. This toolbar is the one that displays the Process, Thread and Stack Frame of the method being debugged.

You can also use this toolbar to switch between the live threads and navigate the call stack of the selected thread just like the Threads Debug Window and Call Stack Debug Window.

Below is picture of the Debug Location Toolbar for your reference.

Tip #5: Use Conditional Breakpoints to Simulate Single-threaded Execution

Choose strategic points where you want to inspect the state of the thread. Create breakpoints in the selected locations. On each of the breakpoints, click Right-Mouse-Button->Condition. In the textbox where you are supposed to enter a condition, enter

System.Threading.Thread.CurrentThread.Name = "<THREAD_NAME>".

Where <THREAD_NAME> is equal to the name of the thread you wish to inspect.

Below is picture of the Breakpoint Dialog for your reference.

IConfigurationSectionHandler Is Dead! Long Live IConfigurationSectionHandler!

2009-01-09T15:40:00Z

I was tasked to create the configuration functionality of the architectural components for an SOA project that I previously worked on. I took as much time as I can in designing the XML thinking that it will be very easy to process it and churn it into a Configuration class by implementing the IConfigurationSectionHandler interface. And just when I started to develop it, I found out that the IConfigurationSectionHandler interface has been deprecated in .Net 2.0.

I’m the type of guy who is OC about this kind of stuff. And so I head forth to find out a solution. The page in MSDN for the interface says that.

In .NET Framework version 2.0 and above, you must instead derive from the ConfigurationSection class to implement the related configuration section handler.

Digging deeper, I learned that they have moved on to a more declarative way of processing configuration sections. So what used to be a series of xml manipulations is now a bunch of configuration classes decorated with the necessary attributes.

In this article, I will try to cover the necessary steps in creating a ConfigurationSection using the .Net 2.0 configuration classes.

The Mark-Up

When creating .Net configurations, I normally start with the xml markup to use. For the purpose of illustration, let’s just say I want to save a list of users and be able to read it at runtime.

    1 <?xml version="1.0" encoding="utf-8" ?>
    2 <configuration>
    3   <configSections>
    4     <section name="project-configuration" 
    5              type="NewConfigurationSection.ProjectConfigurationSection, 
    6                 NewConfigurationSection"/>
    7   </configSections>
    8   <project-configuration>
    9     <users>
   10       <add key="jakelite" firstname="jake" lastname="sta teresa" />
   11       <add key="pinoy" firstname="juan" lastname="dela cruz" />
   12       <add key="kano" firstname="uncle sam" lastname="lagmay" />
   13     </users>
   14   </project-configuration>
   15 </configuration>

The UserElement

The elements inside the users' node should map to a ConfigurationElement class. Each attribute in the configuration xml will map to a property decorated with a ConfigurationPropertyAttribute.

    1 internal class UserElement : ConfigurationElement
    2 {
    3     public UserElement()
    4     {            
    5     }
    6 
    7     public UserElement(string key)
    8     {
    9         Key = key;
   10     }
   11 
   12     [ConfigurationProperty("key", IsRequired = true, IsKey = true)]
   13     public string Key
   14     {
   15         get { return (string)(base["key"]); }
   16         set { base["key"] = value; }
   17     }
   18 
   19     [ConfigurationProperty("firstname", IsRequired = true)]
   20     public string Firstname
   21     {
   22         get { return (string)(base["firstname"]); }
   23         set { base["firstname"] = value; }
   24     }
   25 
   26     [ConfigurationProperty("lastname", IsRequired = true)]
   27     public string Lastname
   28     {
   29         get { return (string)(base["lastname"]); }
   30         set { base["lastname"] = value; }
   31     }  
   32 }

The UserElementCollection

The users' node should map to a ConfigurationElementCollection class. I find this to be the hardest part in doing this so I created a generic class called ConfigurationElementCollection<T>. The collection should be decorated with a ConfigurationCollectionAttribute specifying the type of ConfigurationElement is contains.

    1 [ConfigurationCollection(typeof(UserElement),
    2     CollectionType = ConfigurationElementCollectionType.BasicMap)]
    3 internal sealed class UserElementCollection : 
    4     ConfigurationElementCollection<UserElement>
    5 {
    6 } 

The ConfigurationSection

As advised in the MSDN article, we should create a class that inherits from ConfigurationSection instead of the now deprecated IConfigurationSectionHandler interface. You will find the configuration section class below.

    1 public sealed class ProjectConfigurationSection : 
    2     ConfigurationSection
    3 {
    4     [ConfigurationProperty("users", 
    5         IsDefaultCollection = true, IsRequired = true)]
    6     internal UserElementCollection Users
    7     {
    8         get { return (UserElementCollection)this["users"]; }
    9     }
   10 }

Retrieving The Users

To retrieve the users contained in the configuration, you will have to open the configuration, then locate the configuration section of the target type. Then you can cast it to the target type and use the class in a strongly typed manner. This allows you to access the child nodes as properties.

    1 Configuration configuration = 
    2     ConfigurationManager.OpenExeConfiguration(
    3         ConfigurationUserLevel.None);
    4 ProjectConfigurationSection configurationSection = null;
    5 ConfigurationSectionCollection sections = configuration.Sections;
    6 foreach (ConfigurationSection section in sections)
    7 {
    8     if (section is ProjectConfigurationSection)
    9     {
   10         configurationSection = section 
   11             as ProjectConfigurationSection;
   12         break;
   13     }
   14 }
   15 
   16 if (configurationSection != null)
   17 {
   18     Console.WriteLine("Configured Users");
   19     foreach (UserElement user in configurationSection.Users)
   20     {
   21         Console.WriteLine("User {0}: {1} {2}", 
   22             user.Key, 
   23             user.Firstname, 
   24             user.Lastname);    
   25     }
   26     Console.ReadKey(false);
   27 }

Final Words

In this article we looked at the new way of accessing .Net configuration. This new approach offers a more declarative/OO way of accessing configuration.

However, the sheer verbosity of the new programming model is quite tedious. Add to that the lack of proper samples in the documentation; particularly when implementing ConfigurationElementCollections.

In the end, you will have to choose the new approach if you want to future proof your application configuration strategy. But as for me, I’d trade it anytime to get back IConfigurationSectionHandler.

Code for the samples can be found here.

Threading Patterns: The Worker Thread Pattern

2008-12-19T16:22:00Z

As promised, I am beginning this series of posts about threading patterns that I find useful in everyday coding. To start off, I will introduce you guys to the concept of the worker thread pattern.

This article assumes that you know a thing or two about the concept of threading in general.

.Net Threading Basics

Like any discussion about threading and concurrency, I will start off with the obligatory Threading Basics.

For the purpose of illustration we’ll use the scenario of copying files from a given directory. We will be using a class named CopyInfo to encapsulate the details of the copy instruction.

    1 public class CopyInfo
    2 {
    3     private string _source;
    4     private bool _recursive;
    5     private string _destination;
    6 
    7     public CopyInfo(
    8         string source,
    9         string destination,
   10         bool recursive)
   11     {
   12         _source = source;
   13         _destination = destination;
   14         _recursive = recursive;
   15     }
   16 
   17     public string Source
   18     {
   19         get { return _source; }            
   20     }
   21 
   22     public string Destination
   23     {
   24         get { return _destination; }            
   25     }
   26 
   27     public bool Recursive
   28     {
   29         get { return _recursive; }            
   30     }
   31 }

And a function named CopyFiles that copies files from a source directory to a destination directory.

    1 static void CopyFiles(CopyInfo copyInfo)
    2 {
    3     if (!Directory.Exists(copyInfo.Destination))
    4     {
    5         Directory.CreateDirectory(copyInfo.Destination);
    6     }
    7 
    8     Console.WriteLine("CopyFiles from '{0}' to '{1}' {2}...",
    9         copyInfo.Source,
   10         copyInfo.Destination,
   11         copyInfo.Recursive ? "recursive" : "non-recursive");
   12 
   13     foreach (string file in 
   14         Directory.GetFiles(copyInfo.Source))
   15     { 
   16         string destination = Path.Combine(
   17             copyInfo.Destination,
   18             Path.GetFileName(file));
   19         File.Copy(file, destination);
   20     }
   21 
   22     if (copyInfo.Recursive)
   23     {
   24         foreach (string directory in 
   25             Directory.GetDirectories(copyInfo.Source))
   26         { 
   27             string destination = Path.Combine(
   28                 copyInfo.Destination,
   29                 Path.GetFileName(directory) //get the directory name for the path
   30                 );
   31 
   32             CopyFiles(
   33                 new CopyInfo(
   34                     directory,
   35                     destination,
   36                     copyInfo.Recursive));
   37 
   38         }
   39     }
   40 
   41     Console.WriteLine("CopyFiles finished.");            
   42 }

The main class in .Net threading is the Thread class. In order to use it, you need to have a function which matches the signature of either ThreadStart or ParameterizedThreadStart delegate. For your reference they’re declarations are shown below.

public delegate void ThreadStart();
public delegate void ParameterizedThreadStart(object obj);

Then, you take the name of that function and pass it as a parameter to the constructor of the Thread class. This function can be referred to as the ThreadProc. Once started, the Thread class takes the instructions contained in the ThreadProc and executes it in parallel to the calling function. You can start the thread by calling one of the two overloads of the Start method. If you have a function that matches the signature of the ThreadStart delegate, you can call the parameter-less overload of the Start method. And if your function matches the signature of the ParameterizedThreadStart delegate you can call either of the two overloads of the Start method.

Note that it is an error to pass a parameter to the Start method if your function does not accept a parameter.

With this out of the way, below is an example of starting a Thread using a ThreadStart delegate

    1 private static CopyInfo _copyInfo;
    2 static void Main(string[] args)
    3 {
    4     _copyInfo = new CopyInfo(
    5         Environment.GetFolderPath(Environment.SpecialFolder.MyPictures), 
    6         @"c:\pictures", 
    7         true);
    8 
    9     Thread simpleThread = new Thread(CopyFilesProc);
   10     simpleThread.Name = "CopyFiles";
   11     simpleThread.Start();
   12     simpleThread.Join();
   13 }
   14 
   15 static void CopyFilesProc()
   16 {
   17     CopyFiles(_copyInfo);
   18 }

And another example which starts a Thread using a ParameterizedThreadStart delegate

    1 private static CopyInfo _copyInfo;
    2 static void Main(string[] args)
    3 {
    4     _copyInfo = new CopyInfo(
    5         Environment.GetFolderPath(Environment.SpecialFolder.MyPictures), 
    6         @"c:\pictures", 
    7         true);
    8 
    9     Thread parameterizedThread = new Thread(ParameterizedCopyFilesProc);
   10     parameterizedThread.Name = "CopyFiles";
   11     parameterizedThread.Start(_copyInfo);
   12     parameterizedThread.Join();
   13 }
   14 
   15 static void ParameterizedCopyFilesProc(object state)
   16 {
   17     if (state is CopyInfo)
   18     {
   19         CopyFiles(state as CopyInfo);
   20     }
   21 }

The Impedance Mismatch

As you can see, the method which gets executed by the thread is defined outside of it. What's up with that? It doesn’t look natural in an OO kind of sense. And it can be easily mixed up with other parts of the application. Imagine looking at code for a Form object sprinkled with threading and synchronization logic. It doesn’t look too pretty huh?

In the next section, we'll refactor the code above and encapsulate the threading and synchronization logic in its own class in order to provide an object-oriented interface to the Thread class.

Worker Threads

The Worker Thread pattern operates on the concept of Thread-Per-Message. That is, a data/operation pair is executed in the context of its own thread. It effectively speeds up the execution of multiple long running processes when compared to the time it will take them to execute sequentially. It can also be used to move some long running processes in a different thread in order to avoid freezing the ui.

Below is the code for the abstract class WorkerThreadBase. This class exposes different protected constructors which advanced users can use to specify the name, priority and background/foreground behavior of threads. It has one abstract method Work that represents the ThreadProc. Subclasses need to implement this method in order to take advantage of per thread execution. It has two public Methods Start and Stop that does exactly what their name suggests. The Start operation is asynchronous. And the Stop operation blocks until the thread completes execution. There is one Boolean protected property called StopRequested which can be polled inside the ThreadProc method to detect if the user wants to stop the thread.

    1 public abstract class WorkerThreadBase : IDisposable
    2 {
    3     private Thread _workerThread;
    4     private ManualResetEvent _stopping;
    5     private ManualResetEvent _stopped;
    6     private bool _disposed;
    7     private bool _disposing;
    8 
    9     protected WorkerThreadBase()
   10         : this(null, ThreadPriority.Normal)
   11     {
   12     }
   13 
   14     protected WorkerThreadBase(string name)
   15         : this(name, ThreadPriority.Normal)
   16     {
   17     }
   18 
   19     protected WorkerThreadBase(string name,
   20         ThreadPriority priority)
   21         : this(name, priority, false)
   22     {
   23     }
   24 
   25     protected WorkerThreadBase(string name,
   26         ThreadPriority priority,
   27         bool isBackground)
   28     {
   29         _disposing = false;
   30         _disposed = false;
   31         _stopping = new ManualResetEvent(false);
   32         _stopped = new ManualResetEvent(false);
   33 
   34         _workerThread = new Thread(threadProc);
   35         _workerThread.Name = name == null ? GetType().Name : name;
   36         _workerThread.Priority = priority;
   37         _workerThread.IsBackground = isBackground;
   38     }
   39 
   40     protected bool StopRequested
   41     {
   42         get { return _stopping.WaitOne(1, true); }
   43     }
   44 
   45     public void Start()
   46     {
   47         ThrowIfDisposedOrDisposing();
   48         _workerThread.Start();
   49     }
   50 
   51     public void Stop()
   52     {
   53         ThrowIfDisposedOrDisposing();
   54         _stopping.Set();
   55         _stopped.WaitOne();
   56     }
   57 
   58     #region IDisposable Members
   59 
   60     public void Dispose()
   61     {
   62         Dispose(true);
   63     }
   64 
   65     #endregion
   66 
   67     public static void WaitAll(params WorkerThreadBase[] threads)
   68     { 
   69         WaitHandle.WaitAll(
   70             Array.ConvertAll<WorkerThreadBase, WaitHandle>(
   71                 threads,
   72                 delegate(WorkerThreadBase workerThread)
   73                 { return workerThread._stopped; }));
   74     }
   75 
   76     public static void WaitAny(params WorkerThreadBase[] threads)
   77     {
   78         WaitHandle.WaitAny(
   79             Array.ConvertAll<WorkerThreadBase, WaitHandle>(
   80                 threads,
   81                 delegate(WorkerThreadBase workerThread)
   82                 { return workerThread._stopped; }));
   83     }
   84 
   85     protected virtual void Dispose(bool disposing)
   86     {
   87         //do nothing if disposed more than once
   88         if (_disposed)
   89         {
   90             return;
   91         }
   92 
   93         if (disposing)
   94         {
   95             _disposing = disposing;
   96 
   97             //stop the thread;
   98             Stop();
   99 
  100             //make sure the thread joins the main thread
  101             _workerThread.Join(1000);
  102 
  103             //dispose of the waithandles
  104             disposeWaitHandle(_stopping);
  105             disposeWaitHandle(_stopped);
  106 
  107             _disposing = false;
  108             //mark as disposed
  109             _disposed = true;
  110         }
  111     }
  112 
  113     protected void ThrowIfDisposedOrDisposing()
  114     {
  115         if (_disposing)
  116         {
  117             throw new InvalidOperationException(
  118                 Properties.Resources.ERROR_OBJECT_DISPOSING);
  119         }
  120 
  121         if (_disposed)
  122         {
  123             throw new ObjectDisposedException(
  124                 GetType().Name,
  125                 Properties.Resources.ERROR_OBJECT_DISPOSED);
  126         }
  127     }
  128 
  129     protected abstract void Work();
  130 
  131     private void threadProc()
  132     {
  133         Work();
  134         _stopped.Set();
  135     }
  136 
  137     private void disposeWaitHandle(WaitHandle waitHandle)
  138     {
  139         if (waitHandle != null)
  140         {
  141             waitHandle.Close();
  142             waitHandle = null;
  143         }
  144     }
  145 }

For your reference I’ve created a dummy worker thread class to show how to use WorkerThreadBase. Notice how it uses the StopRequested property to check if the user called the Stop method.

    1 public class DummyWorker:WorkerThreadBase
    2 {
    3     protected override void Work()
    4     {
    5         //a relatively long task
    6         System.Threading.Thread.Sleep(10000);
    7 
    8         if (StopRequested)
    9         {
   10             return;
   11         }
   12 
   13         //a relatively long task
   14         System.Threading.Thread.Sleep(5000);
   15 
   16         Console.WriteLine("Dummy worker completed.");
   17     }
   18 }

In order to use it, one has to simply instantiate the class and call the Start/Stop method correspondingly.

DummyWorker dummy = new DummyWorker();
dummy.Start();
//do some stuff in the calling method
 
//wait for the dummy worker to stop
dummy.Stop();

And finally, lets refactor the copy file operation to create a CopyFileWorker. Note how we used the constructor to pass the data that the thread operates on.

    1 public class CopyFileWorker : WorkerThreadBase
    2 {
    3     private static CopyInfo _copyInfo;
    4 
    5     public CopyFileWorker(CopyInfo copyInfo)
    6     {
    7         _copyInfo = copyInfo;
    8     }
    9 
   10     protected override void Work()
   11     {
   12         copyFiles(_copyInfo);
   13     }
   14 
   15     private void copyFiles(CopyInfo copyInfo)
   16     {
   17     //check if the user called Stop
   18     if (StopRequested)
   19     {
   20         Console.WriteLine("User called Stop.");
   21         Console.WriteLine(
   22             "Terminating thread while copying directory '{0}'.", 
   23             copyInfo.Source);
   24         return;
   25     }
   26 
   27         if (!Directory.Exists(copyInfo.Destination))
   28         {
   29             Directory.CreateDirectory(copyInfo.Destination);
   30         }
   31 
   32         Console.WriteLine("CopyFiles from '{0}' to '{1}' {2}...",
   33             copyInfo.Source,
   34             copyInfo.Destination,
   35             copyInfo.Recursive ? "recursive" : "non-recursive");
   36 
   37         foreach (string file in
   38             Directory.GetFiles(copyInfo.Source))
   39         {
   40             string destination = Path.Combine(
   41                 copyInfo.Destination,
   42                 Path.GetFileName(file));
   43             File.Copy(file, destination);
   44         }
   45 
   46         if (copyInfo.Recursive)
   47         {
   48             foreach (string directory in
   49                 Directory.GetDirectories(copyInfo.Source))
   50             {
   51                 string destination = Path.Combine(
   52                     copyInfo.Destination,
   53                     Path.GetFileName(directory) //get the directory name for the path
   54                     );
   55 
   56                 copyFiles(
   57                     new CopyInfo(
   58                         directory,
   59                         destination,
   60                         copyInfo.Recursive));
   61 
   62             }
   63         }
   64 
   65         Console.WriteLine("CopyFiles finished.");
   66     }
   67 
   68 }

To use it in your Main method:

    1 private static CopyInfo _copyInfo;
    2 static void Main(string[] args)
    3 {
    4     _copyInfo = new CopyInfo(
    5         Environment.GetFolderPath(Environment.SpecialFolder.MyPictures),
    6         @"c:\pictures",
    7         true);
    8 
    9     CopyFileWorker copyFileWorker = new CopyFileWorker(_copyInfo);
   10     copyFileWorker.Start();
   11 
   12     //do some stuff in the calling method
   13 
   14     //wait for the copy worker to stop
   15     copyFileWorker.Stop();    
   16 }

You might just be asking what If I want to execute 2 or more parallel operations and wait for them to complete before I can proceed. Well youre in luck. There’s a static WaitAll method on the WorkerThreadBase class that you can use to wait for as many threads as you like. Below is a sample of how to use this.

    1 private static CopyInfo _copyInfo;
    2 static void Main(string[] args)
    3 {
    4     _copyInfo = new CopyInfo(
    5         Environment.GetFolderPath(Environment.SpecialFolder.MyPictures),
    6         @"c:\pictures",
    7         true);
    8 
    9     //start dummy worker
   10     DummyWorker dummyWorker = new DummyWorker();
   11     dummyWorker.Start();            
   12 
   13     //start copy worker
   14     CopyFileWorker copyFileWorker = new CopyFileWorker(_copyInfo);
   15     copyFileWorker.Start();
   16 
   17     //wait for the two threads to finish
   18     WorkerThreadBase.WaitAll(copyFileWorker, dummyWorker);
   19 }

Some Final Notes

The Thread-Per-Message worker thread can be used to easily perform operations in parallel. However, you must be be aware about the overhead of thread creation and clean-up. As such, ensure that the operations you encapsulate in this pattern is limited only to the long running processes.

Source code for the sample provided in this post are available here.

A Year of Lull

2008-12-17T13:13:00Z

It has been a shameful one year of lull from blogging. For which I am mostly apologetic. A lot of things happened. So first a couple of updates.

I recently moved to a new job in Singapore. I now work for the investment banking arm of a large European bank. I have been here for a couple months now learning about capital markets, derivatives, allocations and stuff specific to the banking and finance domain.

For the next couple of entries I will be posting about threading patterns that I find useful in everyday coding.

Cheers!

Closures and Linq in C# 3.0

2007-12-17T06:51:00Z

Keith posted a great article demonstrating Linqs capabilities. There he was asked if his examples where "closures". Let me take this opportunity to explain how closures are used in Linq by clarifying what a closure is and showing how the Linq queries are translated at compile time(from linq queries -> extension method calls -> static methods calls).

Closures

What are closures really? A closure or formally lexical closure is a function that can refer to and alter the values of variables defined in its declaring scope. In C# closures are made possible by using anonymous delegates. Below is a very contrived example demonstrating an occurence of a closure in C#.

int x = 10;
Action<int> addToX = new Action<int>( delegate(int y ) 
    { 
        x = x + y;
    } 
);
addToX(10);
Console.WriteLine(x);

Here the variable x can be accessed and modified inside the anonymous delegate.

Closures in C# are made possible through a compile time translation called Lifting or closure conversion. Try doing an ILDASM on the above sample code to see lifting in action.

Linq Query Translation

For the purpose of illustration, well use Keith's example on how to find a person with age greater than 25.Since it is the only one demonstrating the use of closure in a linq query.

//specify age
int selectedAge = 25;
//get a list of people with age greater than 25
var ageIsGreaterThan25 =
from l
in listOfPerson
where l.BirthDate >= DateTime.Now.AddYears(-selectedAge)
select l;

Linq queries are basically syntactic sugar over extension methods. Meaning during compile time Linq queries are converted into calls to its extension method. The where clause can now be expressed as:

listOfPerson.Where( p => p.BirthDate >= DateTime.Now.AddYears(-selectedAge) )

Then, these calls are converted into actual calls to the corresponding static methods in the class declaring the extension method. Depending on which one is imported via a using directive. So in case of Linq To Objects, it would be Sequence. The call to the Where method now looks like this.

Sequence.Where<Person>( listOfPerson, p => p.BirthDate >= DateTime.Now.AddYears(-selectedAge) )

Putting it All Into Context

The second parameter to Queryable.Where is a lambda expression of type Func<T, bool>. Func<T, bool> is a generic delegate which has a signature of public delegate RT Func<T0, RT>(T0 a0). (I already explained about the Func delegates in a previous article so I wont be talking about them in much detail here)

Hence we can further translate the code to:

Sequence.Where<Person>( listOfPerson, delegate (Person p) { return p.BirthDate >= DateTime.Now.AddYears(-selectedAge); } )

Which brings us to the anonymous delegate call and therefore a closure.

VS 2008 and .NET 3.5 Beta 2 Released

2007-07-26T23:04:00Z

So for those of you aching to try the new partial methods feature this is your chance. You can access it here.

Scott Guthrie has a post about the features and an installation note to prevent your older asp.net ajax applications from having a dependency on the asp.net ajax binaries that come with beta 2 instead of the one you originally installed.

A Conceptual Concurrent Testing Framework for .Net Applications

2007-05-01T13:13:00Z

Technological advancements in micro-processors suggest an imminent paradigm shift in software development. The sudden shift from old architecture to a new breed of multi–core processors dictates the use of concurrency concepts to optimize performance and to get around synchronization abstractions like semaphores, locks, mutexes and monitors; therefore increasing the demand for multi-threaded testing.

In this post ill share too you some bits about my Master's thesis by taking a look at the characteristics of concurrent software; the challenges in concurrent development; then, present a conceptual concurrent testing framework.

What exactly is concurrent software?

Essentially, order and sequence are the keywords that describe concurrency. Thus, by definition, a concurrent program is a condition where two or more processes (or threads) cooperate in performing a particular task. Each process is monolithic, executing a series of instructions to accomplish a goal. The execution can be interleaved, simultaneous or repeating. Using message passing or shared variables to communicate with each other.

Hence concurrent software can be described by the following characteristics:

· Non-Deterministic Execution

When a concurrent program is run twice, each having the same inputs, it is not guaranteed to return the same output on both occasions.

· Sequential Execution

When a concurrent program has to perform tasks that have some form of dependency relationship, the sequential execution model is used.

· Repetitive Execution

When a concurrent program has to perform tasks in a recurring manner, the repetitive execution model is used.

· Simultaneous Execution

When a concurrent program has to perform tasks that are non-repetitive and non-sequential a simultaneous execution model must be employed.

The problem with shared resource

Concurrent programs perform tasks independently. These tasks although independent from each other need to manipulate shared resources or global variable. This is where the problems arise.

The difficulty of finding errors caused by unexpected interleaving of threads in concurrent programs is well known. Problems like deadlock, livelock, starvation and race condition is quite common to concurrent programs.

What we have

Unit testing ensures the integrity and high quality of software being developed. Its effectiveness is so popular that the integration of such in some agile development method like extreme programming is encouraged. However, limitation of this kind of automated test is bounded to sequential programs, and to date, there is no tool currently used for a systematic unit test of concurrent software components.

A Conceptual Concurrent Testing Framework for .Net Applications

Our conceptual concurrent testing framework aims to achieve the aforementioned by providing the following:

.Net attributes for specifying the execution context of test classes and methods
.Net attributes for decorating the classes and methods identified for testing
.Net attributes for decorating test methods to be ignored
.Net attributes for exceptions expected from a test method
A static class for performing assertions inside client code and
A runtime engine for dynamically loading test classes, executing test methods, generating test output.

The Runtime Engine

Like in other popular testing frameworks, .Net attributes and reflection will play a central role this concept. Attributes help developers add behavioral modifiers in structured code while reflection will provide the mechanism for dynamic instantiation, invocation and type discovery.

The Runtime Execution Engine will provide the necessary abstractions for creating application domains, reflecting assemblies for the presence of attributes. Getting the attributes and executing setup, test and teardown methods.

Net attributes allow developers to embed behavioral instructions in structured code. Structured code, as we know it, provides a way of specifying how a piece of code is to be executed by specifying a set of instructions to achieve something. .Net attributes provide a way of programmatically modifying runtime behavior of structured code without adding complexity to it.

Once metadata are inserted in the Portable Executable (PE), other .NET programs may access it using the .NET Reflection API. The Reflection API allows programmatic inspection of assembly metadata, including the classes exposed by the assembly, the methods and the types found therein, the attributes assigned to various objects, and other related information such as referenced assemblies and assembly version numbers.

The Framework API

The Framework API is a set of .Net attributes used to expose behavioral instructions for test code. Such behavioral instructions are for:

Specifying the execution context of test classes and methods;
Decorating the classes and methods identified for testing
Specifying the type of event a method aims to handle
Decorating test methods to be ignored
Flagging exceptions expected of a test method.

The following is a table of attributes and their corresponding behavior implication in test code.

Framework Attributes
Attribute	Applies To	Behavior
TestFixture	Classes	Signals the runtime that decorated class is expected to contain test methods.
TimeOut	Classes, Methods	When used on a class, signals the runtime that all test methods in decorated class must be completed within the specified time-out period. Otherwise it signals failure. When used on a method, it signals the runtime to override existing class level time-out (if applicable) . This allows the developer to detect a deadlock or livelock occured
Test	Methods	Signals the runtime that decorated method is to be executed as a test method.
IsolationContext	Methods	Signals the runtime that method must be executed in the context of a given AppDomain. This allows the developer to choose a logical grouping for test methods therefore allowing one test to have a combination of AsyncRepeat, Repeat and Sequence behaviors.
Setup/FixtureSetup	Methods	Signals the runtime that decorated method is to be executed before executing test methods.
Teardown/FixtureTeardown	Methods	Signals the runtime that decorated method is to be executed after executing test methods.
ExpectedException	Methods	Signals the runtime that decorated method is expected to throw an exception of the specified type. When an exception of the specified type is caught, the runtime halts execution of the method and flags it as a successful run.
AsyncRepeat(int numberOfTimes )	Methods	Signals the runtime that decorated method is to be invoked repeatedly. Each invocation is run asynchronously on a new thread. Cannot be used with methods already decorated with Repeat and Sequence attributes.
Repeat(int numberOfTimes )	Methods	Signals the runtime that decorated method is to be invoked repeatedly. Each invocation runs synchronously on the same thread as the previous one. Cannot be used with methods already decorated with AsyncRepeat and Sequence attributes.
IgnoreAttribute	Methods	Signals the runtime that decorated method is not to be executed (ignored).
SequenceAttribute( int order )	Methods	Signals the runtime that decorated method should be invoked sequentially Cannot be used with methods already decorated with AsyncRepeat and Repeat attributes.

Evaluation

As you can see, this proposed framework supports non-deterministic, simultaneous, sequential and repeated execution of test methods. Non-deterministic and simultaneous execution was achieved by running test methods in an asynchronous fashion. Sequential execution was achieved by specifying Sequence attributes in test code. Sequence attributes forces the execution order of test methods in a developer specified arrangement. Repeated execution was done in two ways. Developers can specify either a AsyncRepeat attribute or a Repeat attribute. AsyncRepeat attribute forces a repeated asynchronous execution of test methods in different threads while Repeat attribute forces the repeated sequential execution of test methods in a single thread. It can detect livelock, deadlock and resource starvation via timeouts. Race conditions can be signaled as an error. It can also be useful in simulating simultaneous execution by multiple users in a deployment scenario.

The next step would be to find out which extensibility model (NUnit, MbUnit) can support this concept then implement it.

A Tribute to the Unsung Heroes of C#’s Functional Revolution

2007-03-29T12:45:00Z

Many of us may not have noticed it but C# 2.0 gave us several really cool delegates and methods aimed at providing a gentle transition to the functional programming concepts introduced in C# 3.0 and Linq. This post aims to take a look at these useful yet under appreciated delegates and functions instrumental in using C#’s functional programming concepts.

Predicate Delegate

In the silent confines of the System namespace lies the Predicate delegate. It has a signature of delegate bool Predicate<T>(T obj). It gives us a way to determine if an object of a given type passes a certain criteria. The .Net framework employs its use in methods that perform a search (read Find*/Exists) on a collection (List<T>) or array (Array<>T).

Let’s take for example the problem Keith is trying to solve for his friend. Here he is looking for a way to look for a Girl/s in a collection that satisfies a given criteria. He utilized a Predicate in finding all girls that match a certain cuteness factor. Below is his solution with the predicate highlighted for your reference.

public class Girls: List<Girl>
{
   public List<Girl> GetGirlByCutenessFactorTheCoolWay(string cutenessFactor)
   {
      return this.FindAll(
         //implement an anonymous delegate that uses a Girl class
         //as its parameter and then checks to see if the the items
         //in our class is equal to that cuteness factor.
         delegate(Girl girl)
         {
            return girl.CutenessFactor.ToString().Equals(cutenessFactor);
         });
   }
}

Action Delegate

Not very far away is the Action delegate. It has a signature of delegate void Action<T>(T obj). It gives us a way to perform an operation/s on an object of a given type. It is used in the generic function ForEach method of a List/Array.

Looking at the above mentioned solution, it’s obvious that the next step would be to look at the result of the said operation say in a console and to a file. For this, it is fairly common that one trained in imperative programming such as myself will be tempted to write code that reads like this (without using the action delegate).

using ( StreamWriter sw = new StreamWriter( @"c:\cuties.txt" ) )
{
    foreach( Girl g1 in girls.GetGirlByCutenessFactorTheCoolWay("5") )
    {
        Console.WriteLine( g1.Name );
        sw.WriteLine( g1.Name );
    }
    sw.Flush();
    sw.Close();
}

While one used to iterators and blocks in Ruby would express it like this with the action delegate highlighted for your reference.

using ( StreamWriter sw = new StreamWriter( @"c:\cuties.txt" ) )
{
    girls.GetGirlByCutenessFactorTheCoolWay("5").ForEach( delegate(Girl g1)
        {
            Console.WriteLine( g1.Name );
            sw.WriteLine( g1.Name );
        }
    );
    sw.Flush();
    sw.Close();
}

Converter Delegate

While if for some reason our heroes decided they only wanted to have a friendly relationship with the girls or if they decide to have all of them at the same time. They can use the ever friendly converter delegate to change everything in one go.

List<Friend> friends = girls.ConvertAll<Friend>( delegate(Girl g ) { return new Friend(g.Name); } );

List<Girlfriend> girlfriends = girls.ConvertAll<Girlfriend>( delegate(Girl g ) { return new Girlfriend(g.Name); } );

Now it’s not pretty obvious what its signature is from looking at the samples. Here it is for your reference delegate TOutput Converter<TIn,TOut> (TIn input ). Some of you may be scratching your heads right now as to where the first generic argument was passed in our sample. That’s type inferencing in action my friend. The first parameter can be inferred from the type of the list or the delegate signature passed.

Comparison Delegate

And last but not least is the comparison delegate. The .Net framework uses it in the Sort methods of the List and Array classes. It has a signature of public delegate int Comparison<T> ( T x, T y ). It accepts two objects of the same type and performs a user specific computation enough to determine which one is greater. It then returns less than 0, 0, or greater than 0 if the first parameter is less than, equal, or greater than other respectively.
Keith used this in his post about sorting using IComparable and anonymous delegates.

girls.Sort( delegate(Girl g1, Girl g2)
   {
      int returnValue = -1;
      if (g2.CutenessFactor > g1.CutenessFactor)
      {
         returnValue = 0;
      }
      else if (g2.CutenessFactor == g1.CutenessFactor)
      {
         //descending
         returnValue = g1.Age.CompareTo(g2.Age);
      }
      return returnValue;
   }
);

Lambda Expressions in C#

2007-03-23T16:26:00Z

C# 3.0 introduces a new feature called lambda expressions. While this is not a revolutionary thing, it opens up a plethora of new possibilities for .Net programming. This post aims to introduce lambda expressions by providing a backgrounder, looking into its syntax and providing examples along the way.

A Little Bit of History

With the release of C# 2.0, came anonymous methods. For those of you who don’t know it, anonymous methods allow developers to write a block of code in places where a delegate is expected. Prior to this, developers are required to explicitly declare methods and attach them to delegates.

To give you a clearer picture, here’s how one will write a very simple add delegate in pre-2.0 C#.

// EXAMPLE 1

delegate int BinaryExpression( int x, int y);
static int addFunction(int x, int y)
{
    return x + y;
}

static void Main(string[] args)
{

    BinaryExpression add = new BinaryExpression( addFunction );

    Console.WriteLine( add(4, 2) );
}

With anonymous methods you can easily bind the body of addFunction to a named delegate

// EXAMPLE 2
delegate int BinaryExpression( int x, int y);
static void Main(string[] args)
{
    BinaryExpression add = delegate(int x,int y)
                           {
                             return x + y;
                           };
    Console.WriteLine( add(4, 2) );
}

Lambda expressions offer an even more clever alternative to this. Just look at the example below.

// EXAMPLE 3
delegate int BinaryExpression( int x, int y);
static void Main(string[] args)
{
BinaryExpression add = (x, y) => x + y;
Console.WriteLine( add(4, 2) );
}

Before Anything Else…

Let’s take a look at the lambda expression syntax:

It follows the form (parameter-list) => expression; where expression can be any C# expression or a block of code. Just like anonymous methods you can use a lambda expression in place of a delegate. Below are some sample lambda expressions and their corresponding delegates.

// assigns to void D(User u)
// Explicitly typed parameter
(User u) => Console.WriteLine(u.Username.ToUpper())

// assigns to bool D(User u)
// Implicitly typed parameter
(u) => u.Username == “User1”

// assigns to int D(int x, int y)
// Explicitly typed parameter
(int x, int y) => x + y

// assigns to int D(int x, int y)
// Implicitly typed parameter
// Uses a block instead of an expression
(x, y) => { return x + y; }

As you see from the above example, lambda expressions provide a little bit of magic in that it can infer the parameter type from the signature of the delegate it is assigned to.

Func’dafied

You might be thinking by now “WTF! I still have to declare those delegates?”. Well, not really. Lucky for us C# 3.0 provides a set of generic delegates at our disposal. Enter the Func delegate type in its different forms.

public delegate RT Func<RT>();
public delegate RT Func<T0, RT>(T0 a0);
public delegate RT Func<T0, T1, RT>(T0 a0, T1 a1);
public delegate RT Func<T0, T1, T2, RT>(T0 a0, T1 a1, T2 a2);
public delegate RT Func<T0, T1, T2, T3, RT>(T0 a0, T1 a1, T2 a2, T3 a3);

The only thing you need to remember here is that the last generic parameter is the return value.

So going back to example 3, we can now write it as.

// EXAMPLE 3.1
static void Main(string[] args)
{
Func<int,int,int> add = (x, y) => x + y;
Console.WriteLine( add(4, 2) );
}

Of course there will be times when this set of Func’s won’t satisfy your need. Like when you need a delegate signature that has more than 3 parameters or takes a ref or out as an argument. This is when you need to manually create a delegate type.

Resources

C# 3.0 Specification - http://download.microsoft.com/download/9/5/0/9503e33e-fde6-4aed-b5d0-ffe749822f1b/csharp%203.0%20specification.doc

Linq May 2006 CTP - http://www.microsoft.com/downloads/details.aspx?familyid=1e902c21-340c-4d13-9f04-70eb5e3dceea&displaylang=en

Linq/C# 3.0 Learning Guide - http://www.theserverside.net/tt/articles/showarticle.tss?id=CsharpLINQLearningGuide2007

whoami

2007-03-22T12:46:00Z

So whoami? My name is Kriztian Jake Sta. Teresa. I am a software developer working for a process control and automation company. I am mostly involved with writing C#/.Net code when i'm at work and i do Java/Eclipse, PHP, Ruby development on my free time. I will be blogging here mostly about random thoughts on software development and about life in general.