Implement Audit Log for Legacy Code

Given a legacy project, the code has been developed for 6 years, the requirement is to log all the changes in the system. For example, if a user changes his street name, that must be logged so that the administrator can know the old and new values.

The project is a typical codebase where there is domain model, SQL Database using NHibernate framework.

The question is how do we implement that requirement without polluted the domain model, without making breaking changes in the design and functionalities?

Assuming that we have a super simple domain model with User and Address.

    public class User
    {
        public Guid Id { get; set; }
        public string Name { get; set; }
        public ResidentAddress ResidentAddress { get; set; }
        public IList<VisitedAddress> VisitedAddresses { get; set; }
    }

    public class ResidentAddress
    {
        public string HouseNumber { get; set; }
        public string StreetName { get; set; }
    }

    public class VisitedAddress
    {
        public string HouseNumber { get; set; }
        public string StreetName { get; set; }
        public string Country { get; set; }
    }

    public class SuperCoolService
    {
        public void ChangeUserResidentStreet(User user, string streetName)
        {
            user.ResidentAddress.StreetName = streetName;
        }

        public void AddVisitedAddress(User user, VisitedAddress visitedAddress)
        {
            user.VisitedAddresses.Add(visitedAddress);
        }
    }

A user has a residential address and a number of visited addresses. Whenever a user changes his street name, audit log it. Whenever a user visits a new address, audit log it. You got the idea.

Capture Changes

We, unfortunately, cannot redesign the domain model. And we cannot go into every single method and detect if values change. No, that would pollute the domain model and destroy the design. Audit Log is a system concern, not a business domain concern.

It turned out that we can accomplish that goal with the beautiful design of NHibernate. In short, NHibernate allows us to hook in its event pipeline. It allows us to add custom logic for pre and post of Insert/Update/Delete an object. Check more detail from NHibernate document website.

When designing systems, I prefer the explicit approach instead of implicit. I want to have the power of deciding what objects I want to audit, at which properties. Life is so much easier if you are in the control. The same goes for code.

    public interface IAuditable
    {
    }

    [AttributeUsage(AttributeTargets.Property)]
    public class ExplicitAuditAttribute : Attribute
    {
 
    }

    public class User : IAuditable
    {
        public Guid Id { get; set; }
        [ExplicitAudit]
        public string Name { get; set; }
        public ResidentAddress ResidentAddress { get; set; }
        public IList<VisitedAddress> VisitedAddresses { get; set; }
    }

    public class ResidentAddress : IAuditable
    {
        [ExplicitAudit]
        public string HouseNumber { get; set; }
        [ExplicitAudit]
        public string StreetName { get; set; }
    }

    public class VisitedAddress : IAuditable
    {
        [ExplicitAudit]
        public string HouseNumber { get; set; }
        [ExplicitAudit]
        public string StreetName { get; set; }
        [ExplicitAudit]
        public string Country { get; set; }
    }

All need-audit-log classes are decorated with the IAuditable interface. Auditable properties are marked with ExplicitAudit.

    public class NhAuditUpdateDomainHandler : IPostUpdateEventListener
    {
        public void OnPostUpdate(PostUpdateEvent @event)
        {
            var auditable = @event.Entity as IAuditable;
            if (auditable == null)
                return;
            var dirtyFieldsIndex = @event.Persister.FindDirty(@event.State, @event.OldState, @event.Entity, @event.Session);
            if (dirtyFieldsIndex.Length == 0)
                return;
// Custom logic to handle audit logic
        }
    }

Depending on your domain, your audit logic, the detail is yours. By implementing an IPostUpdateEventListener, we can extract all the [ExplicitAudit] properties with their old and new values. Pretty neat design.

Give It Meaning

Soon we hit a problem. We knew the street name was changed. However, we could not know to whom it belongs to. We cannot know the parent of the address. The current design only allows us to know the address of a user, not the other way around.

// Reality
Street name change from "Batman" to "Superman"

// However, we expect this
User: Thai Anh Duc
    Street name change from "Batman" to "Superman"

Think about the situation where you have many changes. The information is useless.

Context. The address is lacking Context. At the time of audit log, it needs to know who it is a part of.

The question becomes how do we give that information for the Address class without introducing a hard reference back to the User class?

Explicit Tell

The goal is to minimize the impact on the existing codebase, to prevent changes in business logic. We cannot support the audit log feature without touching the existing code. I decided to categorize the domain objects into 2 categories: ProvideContext and DependableContext

    public interface IAuditContextProvider
    {
        AuditContext ProvideContext();
    }
	
    public interface IAuditDependableContext
    {
        AuditContext ProvidedContext();
    }
  1. Context Provider: Will tell the infrastructure its own context. It acts like a root. It is the User class in our example.
  2. Dependable Context: Will tell the infrastructure the context it is provided. This allows the infrastructure links the context. It is the Address class in our example.

I must write some code to build the infrastructure to link them. It is infrastructure code, not interfere with the domain code.

There are 2 scenarios we have to deal with: Component and Reference relationship.

Component Relationship

In NHibernate with SQL, a component relationship means 2 objects are saved in the same table. Here is the detail document from NHibernate site.

User_Address_Component
User and home address are merged into one table

Because address cannot stand alone. It cannot be added directly to NHibernate’s session. This gives me a chance to intercept its parent (the user object) when it is attached to NHibernate’s session. I just need to find a way to tell the NHibernate: Hey, whenever you save a user object (via its session), tell the user to populate the audit context to its components.

Thank the beautiful design of NHibernate, this code works beautifully

    public class ProvideAuditContextSaveOrUpdateEventListener : ISaveOrUpdateEventListener
    {
        public void OnSaveOrUpdate(SaveOrUpdateEvent @event)
        {
            var mustProvideContext = @event.Entity as IMustProvideAuditContext;
            if (mustProvideContext != null)
                mustProvideContext.ProvideAuditContext(NHibernateUtil.IsInitialized);
        }
    }

What the code says is that “Hey, if you are a ‘must provide audit context (IMustProvideAuditContext interface)‘, then do it”. When the entity comes to the later phases in the pipeline, it has the audit context provided.

Reference Relationship

However, if the Address object is a reference object (in which it is saved in a separated table), it will not work. Because the referenced objects are inserted first. Take an example, we want to allow a user having many addresses. In this situation, User has its own table. And Address has its own table. There is a UserId column in the Address table.

User has addresses
User has addresses

Take an example, we want to allow a user having many addresses. In this situation, User has its own table. And Address has its own table. When a new address is added to a user, that address is saved first. Then the user is updated.

Because Address knows its user, we can take an advantage by overriding the ProvidedContext method

    public override AuditContext ProvidedContext()
    {
        var context = base.ProvidedContext();
        return context.ContextId == Guid.Empty ? new AuditContext{ContextId = UserId} : context;
    }

If it has a context, then use it. Otherwise, create a new one with UserId

Takeaway

When starting a new project, audit log might not be mentioned, might not be seen as a feature. We, as a developer, should ask customers, managers if we should concern about the Audit Log. No matter the answer, you should be aware of its coming. You should consider Audit Log when designing the code.

Audit Log is an infrastructure concern. Make sure you treat it as is. A very bad approach is to go in every single method call in the domain and register an audit log (or any kind of audit). There are better ways to deal with infrastructure concern from Domain code, depending on the framework you used. In my project, I accomplish that goal by combining

  1. Attribute: By decorated a property with [ExplicitAudit] attribute, I control what properties I want to audit.
  2. NHibernate Events: A well-designed framework will allow you to hook into its pipeline and add your custom logic. I take advantages of NHibernate well-designed events. If you use EF, I am sure you will find many places to hook your custom logic in.

Have you seen the power of OCP (Open-Closed Principle) in NHibernate design? Such a powerful principle.

The real implementation is far more complex than in this post. However, the principles stand. It was really a challenge.

So far so good. It is not a perfect implementation. But I am happy with the result.

Deliver This Feature, Agile Said. Hmm, What About?

Agile is a good methodology to get things done. I buy the philosophies behind it. I think it can be applied in many areas, not just at work, not just at building software. To me, Agile is a tool, a meant to help me get the job done. In software development, I have heard people talking about these beautiful things.

  1. Agile team is self-organized
  2. Deliver working software (features) every sprint.
  3. No upfront design.
  4. Hmm, do we need an Architect? Do we need a Team Leader?
  5. Embrace changes
  6. ….

Many fantasy things. When I first heard about them, I thought

Wow. That’s so cool. Everything looks perfect. Hmm, if everything is perfect, then there is a hidden problem.

Unfortunately, I felt something was missing. Later on (at least up til now), I have found many. Basically, all those things are correct if, and only if, you have a perfect Agile Team. In reality, we all know that human is the root of everything. And having a good team is, well, just a dream. We have to face that truth. Do not try to assume we have a good team at hand. It just does not work that way.

Let’s take a small area from that fantasy: Add a new feature.

The Story

Good morning SWAT Agile Team, in the sprint we will deliver a new cool feature: Delete (soft delete) user. The feature will allow the administrator to delete a user. Users are marked as deleted but not removed from the data storage.

With these facts

  • The system has 10 of features built in.
  • There are features associated with User: Create, Edit, Manage Order, …

The wishes

  • The new feature will not break the existing ones.
  • Reuse the code as much as possible.

The untold facts (which will be discovered when writing user stories for the feature)

  • Some existing features will be impacted. Cannot delete users having orders. Cannot edit a deleted user.
  • Code Quality: Cyclomatic Complexity (CC) must be under control.

If you have been in such a situation, please share how you will handle it.

The Problem

Developers, Testers when joining a so-called agile team, how much have you known about Agile? When a project starts, developers are sold a plan of getting small deliveries, increasements. No upfront design. No architecture. Hey! That is the waterfall, folks, an Agile-men said.

There are many requirement meetings, discussions, … Everything goes well as planned. The team has a clear set of user stories. Estimation is done. Agile is on set.

Good! Ladies and gentlemen, where do we write the first line of code? Serious. Where do we write? Not mention how. To make thing super clear, where/how do you write the first line of production code? Just to opt out the Unit Test code.

A harder question. How do we code in such a way that in the middle of the sprint (or next sprints) the requirements change? How do we embrace the changes?

As the system grows, the harder to add a new feature.

The Solution

Hell No. I do not have a solution. However, I do have some proposals.

Infrastructure and Architecture

No matter what fields we are in, to embrace changes, to stand still, we need support from Infrastructure and Architecture. Without a good foundation, we just build a house on the sand. A single wind can blow it away.

We need an architect. We need a good, stable foundation. If there has not, we must build one.

We need an architect to define system boundaries, to separate the logics into proper concepts, to decide the architectural style, to decide design patterns. A good architect will limit the impact when implementing a new feature or modifying an existing one.

For legacy systems, a good architect can help decouple the dependencies, create boundaries. It is much harder than in the green field projects. But it can be done and can improve over the time. Otherwise, every single change is a risk.

Agile Mindset

People are the core of everything. Members decide the success or failure of the project, of the team. What is their mindset? We bring Agile in to increase the flexibility, to embrace the changes, … None of them will work if members think

  1. I must have a clear requirement to start with.
  2. My job is just coding or testing
  3. Who cares about the design. I just need to code this function. I change only a small part of the code.
  4. Hey, I just add a new method for me. Isolated. No breaking change.

Still the old mindset. I am not saying that mindset is wrong. But it does not fit the new environment either.

I would suggest the first thing we should do is to train our member; to get them adapt with the new mindset; to install the agile mindset into their subconscious mind. It is not easy, not at all.

A few things we can train developers

Architectural Thinking

There is no clear definition of Architect. However, everyone can think architecturally. Every developer, paid for a job, must be able to write code. They should not stop there. Instead, ask these questions

  1. Can I read my code after 3, 6 months? How about a year?
  2. How many ways can break the code?
  3. How easy to change? What if clients want to change XXX?
  4. Is it easy to understand?

What kinds of architectures that support well to embrace changes?

It is a long road. It is worth a try. Oh, wait a minute! Which means we need an architect. Yes, Sir! We need an architect or one having architecture skill in the team.

There is an argument saying that the architect will setup the system, build the framework, make some training sessions, and finally handle over to the team. Good luck! Soon enough, the framework will break. Things will tear apart. Why? Because of the lack of quality of maintainers, the development team.

Coaching

We cannot change someone else mindset. It does not work that way. However, we can show them the right mindset. We can consistently tell them the right mindset. Coaching approach might work. That is you do not train or tell someone what to do. No! They decide what they want to do and how they will do it. We, the team leader (I hope you have one), will

  1. Give some suggestions, show the direction
  2. Encourage
  3. Motivate

Will It Work?

I do not know. But if your current team does not work as you expected, maybe you should try it. There is no silver bullet. Try and Improve!

In discussions, every leader seems to know that people are the most important asset. However, when it comes to daily work, the reality might not the same. How much do we care the most important asset? What have we done to improve that asset?

Agile or waterfall, no matter what methodologies we use, we are a team works on a mission. We want to get that mission done and we want to be proud of what we have done.

Should we need an Architect in the team? Yes. Definitely. Then help everyone think as an architect.

Should we train our people? Yes. Definitely.

To win the war, you need good soldiers!

Happy Sunday! Thank for your reading.

Code Duplication is a Myth

I know that code duplication is not good. We try to avoid code duplication. In a perfect world, you do not have code duplication. What annoys me is that we seem to forget to ask some critical questions to ourselves.

  1. Is it really that bad in all circumstances?
  2. What might be worst if we try to avoid code duplication?

Let’s take a typical web application where you have features to add and edit a user.

With 2 separated models

    public class CreateUserModel
    {
        public string Name { get; set; }
        public string Address { get; set; }
        public DateTime? DateOfBirth { get; set; }
    }

    public class EditUserModel
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public string Address { get; set; }
        public DateTime? DateOfBirth { get; set; }
    }

Or one single User Model?

    public class UserModel
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public string Address { get; set; }
        public DateTime? DateOfBirth { get; set; }
    }

In single user model, if the Id is 0, assume that is creating a new user.

In the real world application, the model is complex. I make this example to demonstrate the point: There are many duplications in 2 models. The question is whether it is good or bad?

Well, it depends … on the context.

Why do we see it duplicated? Mostly because the code looks the same. I, however, want to look at another angle, from the purpose and context angle.

In term of business view, create and edit a user are 2 different features. They operate on the same data object (User). But they are not the same. Let’s assume that we have a web application. There are 2 screens, one for Create and one for Edit.

        public IActionResult CreateUser(CreateUserModel model)
        {
            return View();
        }

        public IActionResult EditUser(EditUserModel model)
        {
            return View();
        }

        // Reuse the same model for 2 different features: Create or Edit
        public IActionResult SaveUser(UserModel model)
        {
            return View();
        }

In the SaveUser action, if the User.Id is 0, use the Create logic, otherwise, use the Edit logic. So we can put aside the logic.

Initially, both Create and Edit screens look the same. They operate on the same piece of data. The single model sounds a right choice. However, I will choose the 2 models. I do not see them duplicated. I do not see benefits of unifying them.

Here are why

  1. Each is an independent feature.
  2. They have different input and output. Sooner or later you will end up some information on a feature not make sense on another.
  3. By separating them, we can have 2 teams (groups) work on 2 features without stepping on each other toes.

Ok then, when will it make sense?

I will consider a duplication when

  1. A piece of code that carries logic. Not just data or POCO.
  2. Independent of context. The refactor code (as a result of killing duplication) might depend on the input parameters. However, keep an eye on the number of parameters and make your instinct decisions base on your experiences.
  3. Easy to understand. There should be a very limit of condition/switching in the implementation.

From my experience, killing duplication is not an easy task. It requires a lot of skills

  1. Skill to ask the right question about the value of the code, the logic of the code, … One should ask themselves serious questions.
  2. Skill to read the code and get a brief of what it does.
  3. Skill to capture just enough to refactor out.
  4. Skill to refactor.

Did I tell you that Naming is one of the hardest tasks? Given that you have a piece of duplicated code, what would you name them?

I fell in love with the slice (with Feature approach) architecture when I first saw it. I saw its value when traditional architecture caused my headache when the code was out of control. Especially, when I have to develop a new feature. The biggest fear when adding new features is the chance of breaking existing features, aka causing regression bugs. With the feature approach, we can limit the risk.

Every approach has its pros and cons. We, developers, have to weigh them and pick the one that fits most.

There are some valuable links from experts.

Jimmy Bogard SOLID Architecture.

Steve Smith ASP.NET Core Feature Architecture.

Application structure: Concepts and Features.

Isolated Feature from Ayende.

Or checkout Jimmy repo here. A wonderful resource to learn how to build a web application with the Slice + MediatR + Feature.

ASP.NET Core MVC – Doors Open, Come In!

Throughout the experiment, I have an empty project. Having an empty project is a good way to learn. It allows me to focus on other things first such as

  1. Build and Release with VSTS
  2. Play around with Docker and Part 2
  3. Fundamental – Configuration.

Ultimately, I want to use ASP.NET Core MVC to build a web application (and other services around it). Time to open the door and step into MVC world; but, as always, with the fundamental first.

ASP.NET Core application is enriched by the use of Middlewares at the developers’ will. MVC is one of the built-in middleware. Plug it in, you have MVC feature. Take it out, well, you are on your own.

MVC Middlewares High Level
MVC Middlewares High Level

Remember, order matters! Let’s take a quick tour with Middleware first.

Middleware

By default, even with an empty project, there are built-in middlewares auto generated. The detail is in the previous post.

Just plug the MVC Middleware in and we are good to go.

Model – View – Controller (MVC)

MVC Model High Level
MVC Model High Level

A request comes in, Controller does 2 things

  1. Build a proper Model.
  2. Decide what view to use.

Remember the keywords here: Build – Model, Decide – View.

The main purpose of a Model is to supply data for the View.

The View takes the data supplied from Model, and render the final HTML for the Response.

Why is that diagram important? Because It came from my understanding of MVC, in my own words (with some borrow from Courses I learned). The point is that I have my own way to express it instead of reading paragraphs of text. Diagrams, images increase my ability to explain.

When you learn something, try to draw diagrams. Try to explain it visually.

Show Me the Code

Let create a home page with some information from an empty project. From a design perspective, what do we need to build an MVC application? Well, I would naively say these

  1. Enable MVC middle
  2. Create a controller to handle the logic
  3. Create a view to display the result

With some fundamentals knowledge about MVC, I created these code and structure

AspNet MVC First Controller
AspNet MVC First Controller

Problems? When I hit F5, an empty page is displayed. I would have expected the Home/Index view displayed.

Old lesson: Nothing works at the first time. If it works, something must have been wrong.

I guess that I missed the routing. Basically, I have to tell the MVC framework how to map a request to a Controller.

Let dig a bit further into routing from MS docs site. There are so many things to learn from that site. However, I just want to get my first simple routing work. I pick what matters most to me.

       public void ConfigureServices(IServiceCollection services)
        {
            // Register the ability to read options in configuration

            services.AddRouting();
            services.AddMvc();
 
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
        {
            loggerFactory.AddConsole();

            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }
            app.UseMvc(routes =>
            {
                routes.MapRoute(
                    "default",
                    "{controller=Home}/{action=Index}/{id?}");
            });
        }

To make the magic work, we must

  1. Enable both Route (AddRouting) and MVC (AddMvc) services. This will register all the necessary services required for routing and MVC
  2. Use MVC with configured routing. Above I defined a default routing.

Visit the home page. It works as I expected. Note

Note: If you create an MVC project from an MVC template, they are all created for you, for free.

Add Logic

The next step is to work with the logic code. So far, I have had a Controller ready to play around, the mighty HomeController.

Display an ugly fact: How far are you to the creator?

Let’s assume a man lives for 80 years maximum. Given a birthday, display how many you have spent and how close you are to the creator.

It is a pretty simple logic. The questions I want to explore here are

  1. Where should I put the logic code?
  2. What kind of project should it be? .NET Standard? .NET Core? or .NET Framework?
  3. How to organize data and flow?

First, let make the functionality work

    public class HomeController : Controller
    {
        public IActionResult Index()
        {
            return View();
        }

        /// <summary>
        /// Serve the GET /home/uglyfact?birthday=dd-MM-yyyy
        /// </summary>
        /// <param name="birthday"></param>
        /// <returns></returns>
        public IActionResult UglyFact(string birthday)
        {
            // Ignore all data validation. Simply assume that the request is correct.
            var dob = DateTime.ParseExact(birthday, "dd-MM-yyyy", CultureInfo.CurrentCulture);
            var meetCreatorOn = dob.AddYears(80);
            var spent = (int)Math.Round(DateTime.Now.Subtract(dob).TotalDays / 7);
            var closed = (int)Math.Round(meetCreatorOn.Subtract(DateTime.Now).TotalDays / 7);
            return View( new UglyFactModel
            {
                SpentWeeks = spent,
                CloseToCreatorWeeks = closed
            });
        }
    }

There is an UglyFact Action which will do some simple calculation. And then it returns an UglyFactModel to the View. Pretty typical MVC stuff. When we take a closer look

When we take a closer look at the UglyFact action implementation, it is a typical piece of business logic. We know that it should not be there. The business logic should not be in the Controller-Action code.

The business logic should bot be in the Controller-Action code.

I decided to go with .NET Standard. Because the business logic code does not depend on the .NET Framework.

namespace Coconut.CoreLogic
{
    public class UglyFactTeller
    {
        public UglyFactModel TellMe(DateTime dob)
        {
            var meetCreatorOn = dob.AddYears(80);
            var spent = (int)Math.Round(DateTime.Now.Subtract(dob).TotalDays / 7);
            var closed = (int)Math.Round(meetCreatorOn.Subtract(DateTime.Now).TotalDays / 7);
            return new UglyFactModel
            {
                SpentWeeks = spent,
                CloseToCreatorWeeks = closed
            };
        }
    }
}

The logic in the Controller-Action has been moved to a separated assembly. And here the nice-clean controller

namespace Coconut.Web.Controllers
{
    public class HomeController : Controller
    {
        private readonly UglyFactTeller _uglyFactTeller;

        public HomeController(UglyFactTeller uglyFactTeller)
        {
            _uglyFactTeller = uglyFactTeller;
        }
        public IActionResult Index()
        {
            return View();
        }

        /// <summary>
        /// Serve the GET /home/uglyfact?birthday=dd-MM-yyyy
        /// </summary>
        /// <param name="birthday"></param>
        /// <returns></returns>
        public IActionResult UglyFact(string birthday)
        {
            // Ignore all data validation. Simply assume that the request is correct.
            var dob = DateTime.ParseExact(birthday, "dd-MM-yyyy", CultureInfo.CurrentCulture);
            return View( _uglyFactTeller.TellMe(dob));
        }
    }
}

To make everything works as before I tell the HomeController how to resolve its dependency (the UglyFactTeller class)

services.AddTransient<UglyFactTeller>();

What’s Next?

At the beginning, I wanted to write more in the post. However, it seems too much for a single post. It is a better idea to stop here and enjoy the “get the job done” feeling.

Next, I will continue with how to make the code structure better (with MediatR (mediator pattern)), explore the unit test.

One of the fun thing when doing from the scratch (or at least with an empty project) is that you think you know them but you do not. You do not know unless you start coding/implementing.

Broke, Stupid and Sick

One of the major key to move on, to get better every day is self-motivation. You cannot hide in your room, lock the door, and wait for someone to knock the door and motivate you. It simply does not work that way.

I am a big fan of Mr. Jim Rohn. In one of his recorded seminar, he said

Being broke is bad; hmm and stupid. But the worst is being broke, stupid and sick.

A man cannot use the same motivation all the time. He needs many, depending on the time and place.

This morning, I woke up at 5:00 am and started my traing as normal. Problem? I wanted to stay in bed. Even when I got to the gym, I did not want to start. Because the weather was so nice, cool wind, fresh air.

I suddenly remembered that quote. I tried to take advantages of it. I used it to give me good reasons to start the training as planned.

I Work Hard

Because I do not want to end up as a broke. I am not sure that I want to be rich. But definitely I do not want to be broke.

I Study A Lot

Because I do not want to be stupid. I do not have to be another Einstein. But definitely not a fool, not a stupid man.

I Train Hard

Because I do not want to be sick. When we are sick, nothing matters. The only wish is getting better, again. Without heath, we are useless.

The 3 Broke, Stupid and Sick – BSS

VS

The 3 Work, Study and Train – WST

By writing them down gives me a clear picture of options. I must choose one of them. Hey I am not stupid. I have chosen the second one WST to keep me away from BSS.
It works for me. It might work for you.

Have a good day!