Miguel Vitorino

on Enterprise Software Development

I Have an Issue With Heroku Deployment Through Git

Am I the only one who thinks that the fact that Heroku only supports deployment through Git is a big disavantage?

I mean, for non compilable languages like Ruby, JavaScript, Python it shouldn’t matter that much, but for JVM languages (the only compilable ones they support at the moment) I would prefer not to share my whole source code history with a company that really does not need to have it.

I believe that if they want to attract more than pet projects and really make a case for the “enterprise market” this should be taken into consideration. Most of us have stuff in their VCS that most definitely don’t need to be available in the production enviroment. That stuff is our secret sauce. Just because Git makes it easy to push it around should we really do it?

Json.NET Custom DataTable Serializer/Converter

At Blue Bridge we have a product called CoSaCS that processes several hundreds of millions of USD in sales per year in retailers around the world. CoSaCS is definitely not a new product. It is not old either, but because of the fairly large customer base, we have to take some care when evolving the technology behind it.

Several of our customers still run Win 2K on their point-of-sales machines. This prevents us, for example, from upgrading the WinForms client to .NET 3.5. So WCF is out of the question… (on the clients at least, servers run .NET 4.0). This means the hundreds (or could be thousands..) of ASMX webservice methods we’re kind of stuck.

Because we may be in .NET 2.0 on the clients for quite a while, lately we have been evaluating looking at a more lightweight approach to webservices for this scenario. I personally have never been a great fan of either asmx or wcf. SOAP… blargh…. On our new stuff we develop mostly on ASP.NET MVC 3 and use this also for our webservices. It is a simple and common model for both pages and webservices. RESTful when possible, but we’re not purists or ever too excited about these “trends”. I’m not one of those guys that believes that a webservice is immediately more “scalable” just because it you map CRUD to POST/GET/PUT/DELETE. And a lot of those so called “RESTful” webservices out there are really just that… Oh boy… I’m so going to get hammered because of this last statement…

This brings me to Json. I really like Json. Sure we could go for Thrift or Protobuf, but never underestimate the benefits of a human readable format. Also there are a few stats out there that point to gzipped Json being almost as compact as those two formats. And almost is definitely enough for us, considering the other benefits of Json and especially comparing to SOAP - blaaargh…

Being also a long time fan of Json.NET I had to look at its behaviour with DataSets. We have a lot of DataSets on COSACS. I don’t like them very much but they serve their purpose as DTOs very well. I already knew that the SOAP serialization of DataSets was pretty inefficient. Looking at the DataTable serialization on Json.NET didn’t leave that excited also [http://json.codeplex.com/SourceControl/changeset/view/61037#435790] (http://json.codeplex.com/SourceControl/changeset/view/61037#435790) If we have a tabular structure why not take advantage of that?

DataTables serialization in XML and Json.NET treats each row as an entity/object by itself. Result? Every column name is repeated as an element for every value in the row.

DataSet XML Serialization
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

<diffgr:diffgram xmlns:msdata="urn:schemas-microsoft-com:xml-msdata" xmlns:diffgr="urn:schemas-microsoft-com:xml-diffgram-v1">
<NewDataSet>
<Table diffgr:id="Table1" msdata:rowOrder="0">
<CustomerKey>11000</CustomerKey>
<GeographyKey>26</GeographyKey>
<CustomerAlternateKey>AW00011000</CustomerAlternateKey>
<FirstName>Jon</FirstName>
<MiddleName>V</MiddleName>
<LastName>Yang</LastName>
<NameStyle>false</NameStyle>
<BirthDate>1970-04-08T00:00:00+01:00</BirthDate>
<MaritalStatus>M</MaritalStatus>
<Gender>M</Gender>
<EmailAddress>[email protected]</EmailAddress>
<YearlyIncome>90000.0000</YearlyIncome>
<TotalChildren>2</TotalChildren>
<NumberChildrenAtHome>0</NumberChildrenAtHome>
<EnglishEducation>Bachelors</EnglishEducation>

Doing the same thing but in Json didn’t make me very happy. So we wrote this little thing I’m now sharing with you:

JsonDataTableConverter & JsonDataSetConverter
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212

using System;
using System.Data;
using Newtonsoft.Json;

namespace Blue.Cosacs.Shared.Net
{
    public class JsonDataTableConverter : JsonConverter
    {
        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            var table = (DataTable)value;

            writer.WriteStartObject();
            {
                writer.WritePropertyName("Columns");

                // serialize metadata (column names)
                writer.WriteStartArray();

                foreach (DataColumn column in table.Columns)
                {
                    writer.WriteStartObject();
                    writer.WritePropertyName("Name");
                    serializer.Serialize(writer, column.ColumnName);

                    writer.WritePropertyName("Type");
                    serializer.Serialize(writer, column.DataType.Name);
                    writer.WriteEndObject();
                }

                writer.WriteEndArray();

                writer.WritePropertyName("Rows");
                // deserialize data
                writer.WriteStartArray();

                foreach (DataRow row in table.Rows)
                {
                    writer.WriteStartArray();
                    foreach (DataColumn column in row.Table.Columns)
                        serializer.Serialize(writer, row[column]);
                    writer.WriteEndArray();
                }

                writer.WriteEndArray();
            }
            writer.WriteEndObject();
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            DataTable dt;

            if (reader.TokenType == JsonToken.PropertyName)
            {
                dt = new DataTable((string)reader.Value);
                reader.Read();
            }
            else
            {
                dt = new DataTable();
            }

            reader.Read();

            // re-create columns
            if (reader.TokenType == JsonToken.PropertyName &amp;&amp; reader.Value is string &amp;&amp; ((string)reader.Value) == "Columns")
            {
                reader.Read();

                while (reader.TokenType == JsonToken.StartArray)
                {
                    reader.Read();

                    while (reader.TokenType != JsonToken.EndArray)
                    {
                        reader.Read();
                        string columnName = null;
                        Type columnType = null;
                        if (reader.TokenType == JsonToken.PropertyName &amp;&amp; reader.Value is string &amp;&amp; ((string)reader.Value) == "Name")
                        {
                            reader.Read();
                            columnName = reader.Value.ToString();
                            reader.Read();
                        }

                        if (reader.TokenType == JsonToken.PropertyName &amp;&amp; reader.Value is string &amp;&amp; ((string)reader.Value) == "Type")
                        {
                            reader.Read();
                            columnType = GetColumnDataType(reader.Value.ToString());
                            reader.Read();
                        }
                        if (columnName == null || columnType == null)
                            throw new JsonSerializationException("Invalid DataTable column definition.");

                        dt.Columns.Add(new DataColumn(columnName, columnType));
                        reader.Read();
                    }

                    reader.Read();
                }
            }

            if (reader.TokenType == JsonToken.PropertyName &amp;&amp; reader.Value is string &amp;&amp; ((string)reader.Value) == "Rows")
            {
                reader.Read();

                if (reader.TokenType == JsonToken.StartArray)
                {
                    reader.Read();

                    // populate rows
                    while (reader.TokenType == JsonToken.StartArray)
                    {
                        reader.Read();
                        DataRow dr = dt.NewRow();

                        var i = 0;
                        while (reader.TokenType != JsonToken.EndArray)
                        {
                            dr[i++] = reader.Value ?? DBNull.Value;
                            reader.Read();
                        }

                        dr.EndEdit();
                        dt.Rows.Add(dr);

                        reader.Read();
                    }

                    reader.Read(); // EndArray
                }

                reader.Read(); // EndObject
            }

            return dt;
        }

        private static Type GetColumnDataType(string dataType)
        {
            switch (dataType)
            {
                case "Int16":
                    return typeof(System.Int16);
                case "Int32":
                    return typeof(System.Int32);
                case "Int64":
                    return typeof(System.Int64);
                case "String":
                    return typeof(System.String);
                case "DateTime":
                    return typeof(System.DateTime);
                case "Decimal":
                    return typeof(System.Decimal);
                case "Boolean":
                    return typeof(System.Boolean);
                case "Double":
                    return typeof(System.Double);
                case "Single":
                    return typeof(System.Single);
                default:
                    throw new ArgumentException(string.Format("Data set column type '{0}' is not implemented for JSON deserialization.", dataType));
            }
        }

        public override bool CanConvert(Type valueType)
        {
            return (valueType == typeof(DataTable));
        }
    }

    public class JsonDataSetConverter : JsonConverter
    {
        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            var dataSet = (DataSet)value;
            var converter = new JsonDataTableConverter();

            writer.WriteStartObject();

            foreach (DataTable table in dataSet.Tables)
            {
                writer.WritePropertyName(table.TableName);
                converter.WriteJson(writer, table, serializer);
            }

            writer.WriteEndObject();
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            var ds = new DataSet();
            var converter = new JsonDataTableConverter();
            reader.Read();

            while (reader.TokenType == JsonToken.PropertyName)
            {
                var dt = (DataTable)converter.ReadJson(reader, typeof(DataTable), null, serializer);
                ds.Tables.Add(dt);
                reader.Read();
            }

            return ds;
        }

        public override bool CanConvert(Type valueType)
        {
            return (valueType == typeof(DataSet));
        }
    }
}

To use this, all you need to do is:

Using the DataSet and DataTable Json Serializers
1
2
3

var serializer = new Newtonsoft.Json.JsonSerializer();
serializer.Converters.Insert(0, new JsonDataSetConverter());
serializer.Converters.Insert(0, new JsonDataTableConverter());

And that is it. The result is a bit better I believe:

Json Serialization
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

{"Table":
{"Columns":
[
{"Name":"CustomerKey","Type":"Int32"},
{"Name":"GeographyKey","Type":"Int32"},
{"Name":"CustomerAlternateKey","Type":"String"},
{"Name":"Title","Type":"String"},
{"Name":"FirstName","Type":"String"},
{"Name":"MiddleName","Type":"String"},
{"Name":"LastName","Type":"String"},
{"Name":"NameStyle","Type":"Boolean"},
{"Name":"BirthDate","Type":"DateTime"},
{"Name":"MaritalStatus","Type":"String"},
{"Name":"Suffix","Type":"String"},
{"Name":"Gender","Type":"String"},
{"Name":"EmailAddress","Type":"String"},
{"Name":"YearlyIncome","Type":"Decimal"},
{"Name":"TotalChildren","Type":"Byte"},
{"Name":"NumberChildrenAtHome","Type":"Byte"},
{"Name":"EnglishEducation","Type":"String"},
{"Name":"SpanishEducation","Type":"String"},
{"Name":"FrenchEducation","Type":"String"},
{"Name":"EnglishOccupation","Type":"String"},
{"Name":"SpanishOccupation","Type":"String"},
{"Name":"FrenchOccupation","Type":"String"},
{"Name":"HouseOwnerFlag","Type":"String"},
{"Name":"NumberCarsOwned","Type":"Byte"},
{"Name":"AddressLine1","Type":"String"},
{"Name":"AddressLine2","Type":"String"},
{"Name":"Phone","Type":"String"},
{"Name":"DateFirstPurchase","Type":"DateTime"},
{"Name":"CommuteDistance","Type":"String"}
],
"Rows":
[
[11000,26,"AW00011000",null,"Jon","V","Yang",false,"\/Date(8377200000+0100)\/","M",null,"M","[email protected]",90000.0000,2,0,"Bachelors","Licenciatura","Bac + 4","Professional","Profesional","Cadre","1",0,"3761 N. 14th St",null,"1 (11) 500 555-0162","\/Date(1121986800000+0100)\/","1-2 Miles"],
[11001,37,"AW00011001",null,"Eugene","L","Huang",false,"\/Date(-20048400000+0100)\/","S",null,"M","[email protected]",60000.0000,3,3,"Bachelors","Licenciatura","Bac + 4","Professional","Profesional","Cadre","0",1,"2243 W St.",null,"1 (11) 500 555-0110","\/Date(1121641200000+0100)\/","0-1 Miles"],
[11002,31,"AW00011002",null,"Ruben",null,"Torres",false,"\/Date(-12272400000+0100)\/","M",null,"M","[email protected]",60000.0000,3,3,"Bachelors","Licenciatura","Bac + 4","Professional","Profesional","Cadre","1",1,"5844 Linden Land",null,"1 (11) 500 555-0184","\/Date(1120950000000+0100)\/","2-5 Miles"],
[11003,11,"AW00011003",null,"Christy",null,"Zhu",false,"\/Date(66960000000+0000)\/","S",null,"F","[email protected]",70000.0000,0,0,"Bachelors","Licenciatura","Bac + 4","Professional","Profesional","Cadre","0",1,"1825 Village Pl.",null,"1 (11) 500 555-0162","\/Date(1120172400000+0100)\/","5-10 Miles"],
[11004,19,"AW00011004",null,"Elizabeth",null,"Johnson",false,"\/Date(82076400000+0100)\/","S",null,"F","[email protected]",80000.0000,5,5,"Bachelors","Licenciatura","Bac + 4","Professional","Profesional","Cadre","1",4,"7553 Harness Circle",null,"1 (11) 500 555-0131","\/Date(1122332400000+0100)\/","1-2 Miles"]
]}}

With real data samples we are experiencing a reduction of the same DataSets to 20% of their XML/SOAP serialization byte size (our typical DataSets have around 50-200 rows). Yes, that’s right, the result is on average 1 fifth of size, compression aside. Enjoy, and never keep looking for ways to better use the bandwidth you will be saving!

The Customer Is Always Right!

This is a sentence that we are used to hear way too often. Is is also a principle behind which a lot of professionals hide in an attempt to avoid having to justify their actions (or lack or them).

Don’t get me wrong, I do believe the customers are usually right. My point is a very different one. If a customer tells you he wants something done in a particular way, I don’t believe you have to follow those directions religiously.

Then you will hear: “well, the customer is paying you/us to do what he wants, so just do it!”. Not so fast I say. I believe our role as consultants is to try to advise the customer, to alert him of the possible consequences of his decisions. Maybe we have seen that happen with a competitor previously. Maybe there is a much cheaper way to do it and he will get almost the same value with a lot less costs. Or maybe it is our instinct just telling us something smells in that solution. No matter the reason, it is also our job to explain to him our point of view. From my experience, most customers will appreciate your opinion as long as it is provided timely – at least they will learn to, if you do your job properly and keep around long enough.

In the end, if they still aren’t convinced and would rather go with their own idea, just do it (as long as it doesn’t get you in trouble with the law or something like that…).

It all seems very obvious right? I wish it did…