The Story of Bug Hunting in Production

BUGS COME AND GO. DEVELOPERS STAY!

Production bugs mean you cannot debug line by line; even worse that cannot reproduce them. The other days, I got one of them. We know when it happened by a screenshot reported by the client.

After a day hunting the bug, I finally found out the root cause. Once found out, the solution is a piece of cake. I thought the process I had been through is interesting and might be useful for others.

Here we go – the process of hunting a production bug.

Step 1 – Treasure in the Log

The first thing is to get the log. Find the entries at that specific time. Most of the system will display a user-friendly error message to end users. The real error, usually in the form of Exception, is buried in the log.

System.Transactions.TransactionAbortedException: The transaction has aborted. ---> NHibernate.Exceptions.GenericADOException: could not execute batch command.[SQL: SQL not available] ---> System.Data.SqlClient.SqlException: String or binary data would be truncated.
The statement has been terminated.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at System.Data.SqlClient.SqlCommandSet.ExecuteNonQuery()
   at NHibernate.AdoNet.SqlClientBatchingBatcher.DoExecuteBatch(IDbCommand ps)
   --- End of inner exception stack trace ---
   at NHibernate.AdoNet.SqlClientBatchingBatcher.DoExecuteBatch(IDbCommand ps)
   at NHibernate.AdoNet.AbstractBatcher.ExecuteBatchWithTiming(IDbCommand ps)
   at NHibernate.AdoNet.AbstractBatcher.ExecuteBatch()
   at NHibernate.AdoNet.AbstractBatcher.PrepareBatchCommand(CommandType type, SqlString sql, SqlType[] parameterTypes)
   at NHibernate.Persister.Entity.AbstractEntityPersister.Update(Object id, Object[] fields, Object[] oldFields, Object rowId, Boolean[] includeProperty, Int32 j, Object oldVersion, Object obj, SqlCommandInfo sql, ISessionImplementor session)
   at NHibernate.Persister.Entity.AbstractEntityPersister.UpdateOrInsert(Object id, Object[] fields, Object[] oldFields, Object rowId, Boolean[] includeProperty, Int32 j, Object oldVersion, Object obj, SqlCommandInfo sql, ISessionImplementor session)
   at NHibernate.Persister.Entity.AbstractEntityPersister.Update(Object id, Object[] fields, Int32[] dirtyFields, Boolean hasDirtyCollection, Object[] oldFields, Object oldVersion, Object obj, Object rowId, ISessionImplementor session)
   at NHibernate.Action.EntityUpdateAction.Execute()
   at NHibernate.Engine.ActionQueue.Execute(IExecutable executable)
   at NHibernate.Engine.ActionQueue.ExecuteActions(IList list)
   at NHibernate.Engine.ActionQueue.ExecuteActions()
   at NHibernate.Event.Default.DefaultFlushEventListener.OnFlush(FlushEvent event)
   at NHibernate.Impl.SessionImpl.Flush()
   at NHibernate.Transaction.AdoNetWithDistributedTransactionFactory.DistributedTransactionContext.System.Transactions.IEnlistmentNotification.Prepare(PreparingEnlistment preparingEnlistment)

Our system leverages NHibernate for Data Access layer. And the exception tells us that there is a piece of string that does not fit into its column max length. It happens when the system commits a transaction.

The question is which table/column? There are many tables involved in a single transaction. This is a complex system.

In the bug’s context, there is very few information involved. I looked at the code that is responsible for the function. Nothing’s special.

Step 2 – Look for Variants

From the customer, I know that some users had the problem. Some have not. Therefore, logged in user plays an important role in the bug.

When looking at any problem, there are always a number of participants, there are input and output. I took a pencil and paper. I jotted down all the participants (actually the C# object/class, such as User, Workflow, …). Just jot them down. Then I drew relationships between them.

Thinking about both cases success and fail, what are the variant factors? Luckily in my case, I can quickly figure out that was the User. And in our system, we care most about 2 pieces of information: Username and Full Name.

This step is important because it will give us a good picture of what might cause the problem.

Step 3 – Look at Data

Luckily that we can access to the production database. Ran some queries. The data emerged an interesting observation:

If the username is 41 characters, it causes the issue. If the username is shorter (at that time I did not notice how many characters the shorter means). I knew that our system keeps the created/edited information. But those columns have more than 50 characters max. It should not be an issue anyway.

Let’s find all the columns having the limit of 41 characters

SELECT TOP 1000 [TABLE_CATALOG]
      ,[TABLE_NAME]
      ,[COLUMN_NAME]
      ,[DATA_TYPE]
      ,[CHARACTER_MAXIMUM_LENGTH]
  FROM [INFORMATION_SCHEMA].[COLUMNS]
  WHERE CHARACTER_MAXIMUM_LENGTH < 41 AND CHARACTER_MAXIMUM_LENGTH > 0 AND DATA_TYPE='nvarchar'

Hurray, the bad boy could not hide anymore. There I had a very small set of columns which might cause the issue. It did not take much time to check each of them. All I had to do is to check in the code which column accepts Username value.

The fix? Well easy. Just need to extend the column max length.

 

Dealing with production bugs is hard. There is very limited information. And for most developers, if they cannot reproduce, debug the bug, they give up. The first and most important thing is to have a right mindset:

There must be something wrong somewhere in the system.

Fix a production bug? Not a problem!

  1. Have the right mindset. Do not assume.
  2. Gather whatever we can. Use whatever we got. Do not complain.
  3. Take a deep breath.
  4. Apply basic logical thinking. The process I share here is one of them.
  5. Hey, need a bit of luck.

Either you like it or not, bugs happen in production. I suggest you embrace them, have fun with them.

BUGS COME AND GO. DEVELOPERS STAY!

2 Minutes and 5 Seconds Rules

In the internet age, productivity becomes a key factor to our success. One of a big problem we face is the flood of information. There is, just, too much to consume every single second. People do not have much problem accessing knowledge. Knowledge is free. Within some minutes asking google, one can find almost anything they wish to know. Even the word Productivity, you can search for its definition, for how to improve productivity, …

The problem? People ask for advice improving productivity. However, that is it. We do not know how to start.

For some, every day we spend time on

  • Facebook: they feel to have a need to know how the world is going, what their friends are doing, … people all have good reason to explain for that they are doing.
  • Online game: Oh yes, we need to relax.
  • Watch TV: Hey cannot miss that exciting football matches. Oh, there is a romantic movie about love.
  • Yeah, read some local newspapers. Need to keep up to date with some famous stars. Btw, there is a famous couple just broke up.
  • Hang on. There are promotions on these sale websites.
  • …..

And we come to their boss asking for increasing salary. And then we complain how limited time they have, that there is not enough time to read job-related materials, that there is not enough time to finish the work. Finally, we look for ways to improve productivity.

People know that

  • To improve health: at least we must start doing exercise
  • To improve salary: at least we must finish their tasks and deliver more than requested
  • To improve skill: at least we must read materials about that skill, or take classes.
  • To gain more time: at least we must stop doing other things – the stupid things.
  • ….

Productivity is, at its basic, the ability to focus to get things done. There might be other definitions. But I decide to choose the one that I can work with.

Some years ago, I had many of them. I started to change by reading as many books as possible. Some told me that reading books will not help anything. They might be right.

However, reading books give me ideas. Many ideas are so obvious. And because they are so obvious. Not many people care about them. I have applied many of those ideas into my daily life. It works. Because I believe in them. If I was wrong, I lost nothing by the way.

Once again, I have not found enough. I procrastinate. Sometimes when a task comes in, my immediate thought was a denial. My brain finds reasons to not doing it.

I give you some

When I want to write a blog post, my brain gives me

  • No one gonna read your post
  • Hey, it is too easy. Everyone knows.
  • They will make laugh at you – idiot.

When I have to do paper work in government offices, my brain gives me

  • It will cost you lot of time
  • Too crowded
  • Those officers are unprofessional.
  • Hey, today is not the right time.

And I happened to know that that is the way our brain works.

 

I thought what if I combine these ideas and give them a try: 2 Minutes rule from David Allen and 5 Seconds rule from Mel Robbins.

2 Minutes rule

If a task takes less than 2 minutes to complete. You should do it.

5 Seconds rule

Count down 5-4-3-2-1 DO IT

Note: They are not exactly as the original. I made them as I understood them my own way.

My framework: 5-2-DO Rule

Given a task,

If it takes less than 2 minutes

Then count down 5-4-3-2-1 DO IT.

It reads Fine To Do” rule. (I am a programmer)

The main idea is to act, to get things done.

By now, we might wonder “How do I know it takes 2 minutes?” We don’t. It does not really matter the time. The 2-minute window is a concept. It makes we think that those tasks are easy and that we can do it quickly.

How long does 2-minute take? To get a feeling, I suggest you go on a treadmill, set for 60 minutes running, at the 58th minute, you will feel how long a 2-minute take. Or should I suggest you wait for the 2-minute red light in traffic? Feel it?

Writing this post takes me hours. How do I apply the “Fine To Do” rule? I break it down into 4 tasks

  1. Task 1: Decide to write and topic.
  2. Task 2: Open the laptop and go to my blog, click on the New Post button.
  3. Task 3: Write your first sentence.
  4. Task 4: Finish the post

Except the task 4, the first 3 tasks do not take much time, maybe less than 2 minutes. By separating them into a small chunk of easy tasks, it helps me get started, which is the most important thing in Productivity. To do something, you must start first.

By applying that rule, I do not have time for unwanted thoughts. My brain does not have a chance to interrupt with strong reasons to stop me. I know it is going to stop me, therefore, I make a step ahead.

 

Now comes to a final obstacle of the post: The moment, the courage to press the Publish button. Everything so far is my private. I feel safe when writing. But the moment I think of publishing, all the fear jumps in.

Take 2 seconds to press the publish button. 5-4-3-2-1 DO.

Welcome to my rule: 5-2-DO. AKA: Fine To Do.

Book Review: The 5 Seconds Rule by Mel Robbins

I have finished reading the book. I made many notes, highlights from my Kindle. They are all available in My Notes On Goodreads. I am practicing the habit of reviewing my notes right after I finish reading. While reading the notes, I can recall my memories and connect the content in the book. I just realized that I have something in my head. I also know that I want to write them down.

Writing down the thoughts is not an easy task. However, the book has taught me a secret – the courage to do things. You guess it. By counting down 5-4-3-2-1 DO.

I, actually, do not do a book review at all. I chose that title because I have not found any word to replace. Reviewing a book is a big deal. I am not going to do that. Instead, I want to focus my energy on writing what I have learned from the book. I want to write the good part that the book conveys and wants to give to the reader. Others might have different views. They might see the opposite side of the book. I do not look for what is not good. I focus on what is good, on what is in it for me, on what I can take away and apply to my life.

The idea in the book is ridiculously simple. Everyone has 5 seconds to convert from intent to action. By counting down, yes counting down – not up, 5-4-3-2-1 DO.

The moment you have an instinct to act on a goal you must 5-4-3-2-1 and physically move or your brain will stop you.

In my word, move your ass. Nothing happens unless you move your ass. You must move physically.

A normal, but famous, example is getting up early. “Early” means you wake up at the time you want to get up before you go to sleep. For example, many people want to get up at 6:00 AM to do exercise. Every evening, while on the bed, they tell themselves that they will wake up at 6:00 AM in the morning. When the clock alarms, a hidden voice tells you

  1. Hey, you went to bed late last night
  2. Hey, it is cold out there
  3. Hey, com’on it is weekend
  4. Hey, just stay here for 5 minutes
  5. ……

The result? You wake up at 8:00 AM. And the cycle repeats.

The book tells you a simple, but powerful, tip to overcome that habit by counting down: 5-4-3-2-1 UP (use the word that you like most). Right after reaching “1”, out of bed.

I think many people will laugh at that tip and think that it will not work. Sure, it will not work if you think it will not work. Here is the simple fact

  1. If you give it a try, you get 50% success, 50% failure
  2. If you do NOT try, you get 100% failure

 

Another interesting thing I found from the book is the explanation of “procrastination”

For a long time, everyone believed procrastination meant poor time management skills, a lack of willpower, or lack of self-discipline. Boy, were we wrong. Procrastination is not a form of laziness at all. It’s a coping mechanism for stress.

And

Dr. Pychyl has found that the main thing driving procrastination is not avoiding work. It’s avoiding stress. Procrastination is “a subconscious desire to feel good right now” so you can feel a little stress relief.

And that is because of our default behavior of our brain

We are amazing at fooling ourselves into staying exactly where we are.

Th human being is afraid of changes. Our brain is designed to protect us, to keep us safe. It builds a self-defense mechanism.

As I mentioned in Chapter One, change requires you to do things that are uncertain, scary, or new. Your brain, by design, will not let you do such things.

That’s why we love sitting on Sofa, watching a TV show, a movie. The Sofa makes us feel comfortable and safe.

As I read through the book, I thought of many real examples I have been and I have seen. Kind of fun when you reflect on your life. You should try it.

 

Another thing I learned from the book is how people make decisions. It turns out

According to neuroscientist Antonio Damasio, it’s our feelings that decide for us 95% of the time. You feel before you think. You feel before you act. As Damasio puts it, human beings are “feeling machines that think” not “thinking machines that feel.” And that’s how you ultimately make decisions—based on how you feel.

You Make Decisions Based On How You Feel

 

I would suggest people read the book. I have started applying what I have learned in it. If I want to do something or get something done, or whatsoever I want to do, I would choose this

5-4-3-2-1 MOVE-YOUR-ASS!

Code That Embraces Changes

Developers write code. They write code for specific functionalities, with requirements from clients (customers, bosses, leaders,…). When requirements come, they know what to do; know what code to write. In the world of business applications, the implementation is not a big deal. The story might be different if we write code for embedded systems using C/C++.

However, most of the problems come later, when the first draft of the function has done. The functions have deployed to the customers. They test and realize that they have to change here and there. If I have to pick one constant in the software development, I would pick CHANGE. The clients will change their mind. They know better when they actually see the product. You, as a developer, cannot avoid changes and cannot tell clients stop changing. No. It does not work that way.

Agile emerges with the spirit: Embrace changes. A wonderful mindset shift! We have to adapt that fact. However, the biggest question remains: How do you deal with changes?

How do you deal with changes in code?

I know that there are processes to deal with changes. But, ultimately, it all comes down to the code. You have to change the code, most of the time you have to.

I do not have a single answer or solution to that problem. However, I do have some experience in designing code that gives me the courage, the tool to embrace changes.

Here we go. But first, let’s define a simple feature. We have to work on a concrete example.

Requirement

As a business owner, I want to export orders (that are made today) into CVS file so that I can import into Excel and build some charts.

Assumption: There is a shopping system. They take orders from users. The orders are stored in the database. They are all basic stuff if you have ever worked with a business application system.

First Implementation

Without any hesitation, jump right into the VS2017 and implement the feature. Btw, how hard it is to implement such a simple feature, right?

    public class Order
    {
        public string Customer { get; set; }
        public DateTime OrderedAt { get; set; }
        public decimal TotalCost { get; set; }
        public int NumberOfProducts {get;set;}
    }
    class Program
    {
        static void Main(string[] args)
        {
            var ordersFromDatabase = new List<Order>
            {
                new Order{Customer = "Batman", TotalCost = 100000, OrderedAt = DateTime.Today.AddDays(-3)},
                new Order{Customer = "Superman", TotalCost = 1000, OrderedAt = DateTime.Today.AddDays(-2)},
                new Order{Customer = "Spiderman", TotalCost = 100, OrderedAt = DateTime.Today.AddDays(-1)}
            };
            const string headers = "Customer;Total Cost;Ordered At";
            var data = new List<string>{headers};
            foreach (var order in ordersFromDatabase)
            {
                data.Add(string.Join(";", order.Customer, order.TotalCost, order.OrderedAt));
            }
            var fullPathToCsvFile = @"d:\export\orders.csv";
            File.WriteAllLines(fullPathToCsvFile, data, Encoding.Unicode);
        }
    }

The actual implementation in a real project will be different. To demonstrate the point, I simplify the code.

What do we have above?

  1. An order has 3 information: Customer, Total cost, and Ordered At.
  2. There are 3 orders for Batman, Superman, and Spiderman. Assuming that they come from the database. Batman is a rich man so he ordered a lot 😛
  3. Export to CVS with “;” deliminator.
  4. The file is located at “d:\export\orders.csv” with Unicode encoding.

The code works. Mission accomplished.

Changes Come

Here are some changes that a customer might want to make

  1. Add NumberOfProducts into the export
  2. Change the location of exported file or even the encoding.
  3. Hey. I want to see reports of 2 days instead of just today.
  4. ….

Things go on in various ways. Finally, they all come to you, developers, to reflect the changes in code. If the application is just a simple stupid console app in this example, there is not any problem at all. Everything can be changed in a matter of minutes.

But we all know that it is not true in the real life.

Embrace Changes

I would prefer to do some logical analysis before actually writing code. If I start my code first, it will affect my thinking process. Sometimes it helps.

To embrace changes, we, first, must identify the possible changed areas. Isolate and Conquer approach

  1. Identify the possible changed areas
  2. Isolate them. Mean you can conquer them. Because you know where to look at, where to change the code.

Identify

Next and hardest question is

How to identify those areas?

Answer

Think in process and interaction. Capture concepts and patterns

Back to export order example. What does it mean? How do I apply in that example?

What is the process of exporting order?

  1. Get the orders
  2. Build data
  3. Write data into CVS file
Export orders process
Export orders process

Obviously, the process will not change. Because we control how we want to implement it. In other words, it does not depend on the customer. What changes? The 3 boxes: Get orders, Build data and Write to file.

Let’s go through each of them

Get Orders

  • Where to get them?
  • How to get them?
  • Criteria

Build data

  • How many columns do we need?
  • What is the format of currency, datetime, … ? the presentation of the data

Write to file

  • What deliminator to use? “;” or “,”? Maybe directly to Excel file?
  • Where to store the file? in which encoding?
  • What if the file fails to save due to permission?

Let’s say we have identified them. Now, let’s isolate them.

Isolate

I will modify the code as below

    class Program
    {
        static void Main(string[] args)
        {
            var orders = GetOrders();
            var data = BuildData(orders);
            WriteToFile(data);
        }

        private static IList<Order> GetOrders()
        {
            return new List<Order>
            {
                new Order{Customer = "Batman", TotalCost = 100000, OrderedAt = DateTime.Today.AddDays(-3)},
                new Order{Customer = "Superman", TotalCost = 1000, OrderedAt = DateTime.Today.AddDays(-2)},
                new Order{Customer = "Spiderman", TotalCost = 100, OrderedAt = DateTime.Today.AddDays(-1)}
            };
        }

        private static IList<IList<string>> BuildData(IList<Order> orders)
        {
            // Initialize rows with header
            var rows = new List<IList<string>>
            {
                new List<string> {"Customer", "Total Cost", "Ordered At"}
            };
            // Add rows data for each order
            foreach (var order in orders)
            {
                rows.Add(new List<string>
                {
                    order.Customer,
                    order.TotalCost.ToString(CultureInfo.CurrentCulture),
                    order.OrderedAt.ToString(CultureInfo.CurrentCulture)
                });
            }
            return rows;
        }

        private static void WriteToFile(IList<IList<string>> data)
        {
            const string deliminator = ";";
            var fullPathToCsvFile = @"d:\export\orders.csv";
            var csvData = data.Select(a => string.Join(deliminator, a))
                .ToList();
            File.WriteAllLines(fullPathToCsvFile, csvData, Encoding.Unicode);
        }
    }

Each method in the Main class encapsulates a concept, area that we have mentioned. Whether we should move them into separated classes is not that important. But yes, you should move them when in production code.

With that in place, when clients want to change something, you are more confident to embrace them. Or when there are bugs. In the beginning of the post, I mentioned that clients want to add NumberOfProducts into the report. Do you know where to change? Yeah! You got it – change in the BuildData method.

The above refactoring is good to embrace changes. However, we can make it better. I leave it to you if you wish to challenge yourself.

Takeaway

Take the “Embrace changes” seriously with you. Train your mindset into the concept. As a developer, we must be aware of that simple fact. Complaining about them will not solve the problem. You will have to deal with it anyway. Better, you should be in proactive position. Changes come! Not a problem!

When implementing a new feature, write new code, I would suggest that you stop for a moment, draw the process, the interaction in your head (better if you do it on paper) before you actually write code. Once you have done that, the possible changes areas emerge. Capture them immediately.

I have a simple process for you

  1. Think about the process, the interaction in normal language. Not the language of the code.
  2. Identify the possible changes areas.
  3. Capture them immediately. You should write them down in the paper.
  4. Write code.
  5. Design patterns, best practices come last.

I hope you enjoy the reading and use them for your next piece of code.

Have a nice day!