How best organize project, versions?

This morning I had a look at my daily news groups and found an interessting question. One guy asked for best practice organizing the project versions within the file structure.

I don’t think that I have the philosopher’s stone, but over the years I found my best practices.

My folder structure looks like:

\{Project name}
  \Build
  \Source
    \v1.x
      \{Apps}
    \v2.x
      \{Apps}
  \Distribute
    \Release {Release Number}
  \Documentation
    \{Documentation sub folders}
  \Test

The build folder contains Nant Scripts and command line tools to make the release builds and also the nightly builds.

As you can see the source folder has sub folders that contains the major releases. The reason why I bundle multiple source versions to 1.x and 2.x is that major changes (like you expect it from a major release change), requires big changes within the code base. That’s why I think it’s better to get rid of the old v1.x stuff and begin with a clear and fresh file structure 2.x.

On the other hand, building a new folder structure for every small release or patch doesn’t make any sense, because it’s much work and cost a lot of storage to copy always a complete folder structure. Therefore I strongly recommend to use a source code management system like SubVersion or CVS. Those tools allows you to manage smaller releases and set marker tags to pin your source for a specific version.

The Distribute folder contains the stable distributions as ZIP and MSI files. Referring to that, every distribution has it’s own subfolder that contains these two files. Last but not least the Test Folder contains a runable nightly build release in order to test the current development.

So I’m interessted in knowing what’s your experience with that issue?

Cheers
Gerhard

---

Link to the news group: How best organize a project, versions?

2000% faster using dynamic method calls

As you all know, the performance of invoking a method using reflection is not best. Or as I would say – it’s beneath contempt.

Due to a performance problem that I had the last days I had to dig into the problem and found a way to improve the method invocation at about 2000%. A MSDN article offered by my collegue Alexander Jung gave me the idea.

The idea behind is to create a generic DynamicMethod at runtime to set and get the value of a property. The dynamic methods will be invoked trough a generic delegate that can be created for that dynamic methods at runtime. The delegate looks like:

public delegate void GenericSetter(object target, object value);
public delegate object GenericGetter(object target);

The target parameter defines the instance of an object of that the property will set. The example code for my performance messure program looks like:


/*
* Create a dynamic method and a delegate for it
*/
PropertyInfo testPropertyInfo = typeof(BusinessEntitiy).GetProperty("Counter");
setter = CreateSetMethod(testPropertyInfo);
getter = CreateGetMethod(testPropertyInfo);

/*
* Now repeat it 1 million times and Set/Get the value
*/
DateTime startTime = DateTime.Now;
for (int x = 0; x < REPEATS; x++)
{
   setter(be, x);
   int check = (int)getter(be);


   Debug.Assert(check == x, "Setter call failed.");
}


The magic behind that code are the methods CreateSetMethod and CreateGetMethod. The implementation of those methods can be downloaded at the end of that posting.

These methods are using the DynamicMethod class and the ILGenerator in order to create a dynamic method at runtime that calls the getter and setter property without reflection. The performance boost is unbelievable. I tested the raw set and get functionality with about 1 million property calls. The standard reflection method invocation took about 4,6 seconds; the dynamic method call took only about 0,22 seconds on my machine.

Click image to enlarge.

If you are interessted in that solution you can download the complete example using the following link:
2000% faster using dynamic method calls

Appendix:


///
/// Creates a dynamic setter for the property
///
private static GenericSetter CreateSetMethod(PropertyInfo propertyInfo)
{
   /*
   * If there's no setter return null
   */
   MethodInfo setMethod = propertyInfo.GetSetMethod();
   if (setMethod == null)
     return null;

   /*
   * Create the dynamic method
   */
   Type[] arguments = new Type[2];
   arguments[0] = arguments[1] = typeof(object);

   DynamicMethod setter = new DynamicMethod(
     String.Concat("_Set", propertyInfo.Name, "_"),
     typeof(void), arguments, propertyInfo.DeclaringType);
   ILGenerator generator = setter.GetILGenerator();
   generator.Emit(OpCodes.Ldarg_0);
   generator.Emit(OpCodes.Castclass, propertyInfo.DeclaringType);
   generator.Emit(OpCodes.Ldarg_1);

   if (propertyInfo.PropertyType.IsClass)
     generator.Emit(OpCodes.Castclass, propertyInfo.PropertyType);
   else
     generator.Emit(OpCodes.Unbox_Any, propertyInfo.PropertyType);

   generator.EmitCall(OpCodes.Callvirt, setMethod, null);
   generator.Emit(OpCodes.Ret);

   /*
   * Create the delegate and return it
   */
   return (GenericSetter)setter.CreateDelegate(typeof(GenericSetter));
}

///
/// Creates a dynamic getter for the property
///
private static GenericGetter CreateGetMethod(PropertyInfo propertyInfo)
{
   /*
   * If there's no getter return null
   */
   MethodInfo getMethod = propertyInfo.GetGetMethod();
   if (getMethod == null)
     return null;

   /*
   * Create the dynamic method
   */
   Type[] arguments = new Type[1];
   arguments[0] = typeof(object);

   DynamicMethod getter = new DynamicMethod(
     String.Concat("_Get", propertyInfo.Name, "_"),
     typeof(object), arguments, propertyInfo.DeclaringType);
   ILGenerator generator = getter.GetILGenerator();
   generator.DeclareLocal(typeof(object));
   generator.Emit(OpCodes.Ldarg_0);
   generator.Emit(OpCodes.Castclass, propertyInfo.DeclaringType);
   generator.EmitCall(OpCodes.Callvirt, getMethod, null);

   if (!propertyInfo.PropertyType.IsClass)
     generator.Emit(OpCodes.Box, propertyInfo.PropertyType);

   generator.Emit(OpCodes.Ret);

   /*
   * Create the delegate and return it
   */
   return (GenericGetter)getter.CreateDelegate(typeof(GenericGetter));
}

New Dictionary: HashList – or How to build an ordered Hashtable

Sometimes the world is quite awful. You need a fast key access to a collection because of the performance of your application, and you need the garantuee that you can read the items in the same sequence like you put it in the list.

This scenario happend to me a few times during my career as a developer. In past I hold the elements twice in a hashtable and in a collection in order to have both – fast key access and the correct ordering. The problem was that this method causes a lot of overhead in managing the consistency of your data.

DotNet 2.0 offers a new collection called “KeyedCollection” which is one big step into the right direction. The problem is that the KeyedCollection ist abstract and therefore must be inherited, furthermore it does not implement the IDictionary interface.

That’s why I wrote a wrapper that is based on the new KeyedCollection and does implement the IDictionary interface. I called it HashList.

Look at this piece of code:

IDictionary contacts = new Dictionary();

contacts.Add("a", contact_a);
contacts.Add("b", contact_b);
contacts.Add("c", contact_c);
Dump(contacts);

contacts.Remove("a");
contacts.Remove("b");
contacts.Remove("c");

contacts.Add("a", contact_a);
contacts.Add("b", contact_b);
contacts.Add("c", contact_c);
Dump(contacts);

a.) contact a
b.) contact b
c.) contact c

The second list is NOT in the right order anymore.
c.) contact c
b.) contact b
a.) contact a

Now the same piece of code using the new HashList (you can download it with the link at the end of the blog).

IDictionary contacts = new HashList();

contacts.Add("a", contact_a);
contacts.Add("b", contact_b);
contacts.Add("c", contact_c);
Dump(contacts);

contacts.Remove("a");
contacts.Remove("b");
contacts.Remove("c");

contacts.Add("a", contact_a);
contacts.Add("b", contact_b);
contacts.Add("c", contact_c);
Dump(contacts);

a.) contact a
b.) contact b
c.) contact c

Now everything is ordered, like we had put them in the list.
a.) contact a
b.) contact b
c.) contact c

That’s my solution for using the KeyedCollection Microsoft offers. Hope you enjoy the hybrid between a Dictionary and a Collection.

The example can be downloaded using the following link:
New Dictionary: HashList – or How to build an ordered Hashtable

PS: Before I get another comment like the first one . It’s not the intention of this example to sort the items by their Key. The intention is to keep the items in the same order like they have been put in the Hashtable.

Best practices: ReaderWriterLock

Yesterday was my first day in office after two weeks of vacation. My collegue Oliver Weichhold told me that he had to fix some piece of my code because of multi threading problems.

I’m dedicate this blog to this because I hadn’t been fully aware of the shattering effect of such a failure.

Imagine you want to cache some data due to performance issues. The cache will be dynamically filled during the run-time of your application. That means that you have to check if a value is already stored within the cache – if not – you have to add it to the cache.

My naive solution was to add a static hashtable to the class that holds the values and to add a method in order to access the cache with a key.

static Hashtable cache = new Hashtable();

static string GetValue (string key)
{
   string result = (string) cache[key];
   if (result != null)
     return result;

   result = "something"; // Calculate result
   cache.Add(key, result); // Add to cache
   return result;
}

The problem is that a hashtable in C# is not thread safe at all, which causes problems when running massive multi threading.
As a solution my collegue added the ReaderWriterLock Pattern.

static Hashtable cache = new Hashtable();
static ReaderWriterLock staticLock = new ReaderWriterLock();

static string GetValue (string key)
{
   staticLock.AcquireReaderLock(Timeout.Infinite);

   try
   {
     /*
      * Query the first time, perhaps a reader lock is enough
      */
     string result = (string) cache[key];
     if (result != null)
       return result;

     LockCookie cookie = staticLock.UpgradeToWriterLock(Timeout.Infinite);
     try
     {

       /*
        * Important to query a second time !!!
        * Maybe a thread before the waiting one has already been added the value into the hash table.
        */
       result = (string) cache[key];
       if (result != null)
         return result;

       result = "something"; // Calculate result
       cache.Add(key, result); // Add to cache
       return result;
     }
     finally
     {
       staticLock.DowngradeFromWriterLock(ref cookie);
     }
   }
   finally
   {
     staticLock.ReleaseReaderLock();
   }
}

I marked the important parts with color red and blue:

Red: The red color indicates the part of the ReaderWriterLock. It’s important that the first time always a reader lock is opened. Only if the value can not be found within the hash table, the static lock object will become a writer lock due to the upgrade method. Using the try/finally construct is the best method to protect the pattern against uncatched exceptions, which would kill our application because of an unreleased reader lock.

Blue:The blue part is (imho) the most important part of the pattern. Imagine you have to threads, one owns the writer lock and the second wait until the first writer thread has been finished. The first thread puts the value into the hash table and releases the lock. If the blue part would miss, the second thread (which is waiting at method UpgradeToWriterLock) would now try to add the value again to the hashtable. That is because he does not know that the value has already been added by the first thread. That’s why it’s unbelievable important two check the existence of the key a second time.

That’s all for now.

The only thing which would be interessting is the amount of performance that get lost using ReaderWriterLock. I think I’m going to profile both solutions when I have time and let you know …

Cheers
Gerhard