❮ 2024-08-31

Side effects fun with JOOQ and Lombok

New Project, new things to learn

After years using Scala + Akka for a really long time and never having the opportunity to just use Spring Boot, the time finally came:

New project with a good old, well known software stack. And not even that, for our DAO layer we're using JOOQ which is much nicer as the ancient version of ScaliceJDBC we had to use due to Java 8.

JOOQ provides a DSL to write type safe, database-agnostic queries in Java, plus it offers a lot of utility functionality like aggregating data from multiple rows into a map or list, and mapping ResultSets to Java-Objects.

But the best feature still is the type-safe DSL: If your queries don't get too wild, after Datamodel-Refactorings, JOOQ is able to tell you exactly were old queries will fail and the app won't even run until you fix everything, because everything gets detected at compile-time.

In my previous project we just used plain SQL Strings, finding non functioning queries after a refactoring was only possible at runtime by literally clicking through the app with no safety-net whatsoever, I don't want ever go back to this.

On top of that we also use Lombok, to generate all sorts of boilerplate code via annotations or configs, to get some convenience features that other, more modern languages already have baked in natively. For example:

So in short we use:

And for testing

Together, all libraries bring their own sort of abstractions and "black magic" that might lead to weird interactions if configured or used incorrectly.

And of course I shot myself into the foot, and had a fun 2 day debugging session that taught me a lot of things I did not know before.

Java Developers with a little more experience under their belt might have spotted the error immediately. For me though, after blood, sweat and tears it nevertheless was a cool discovery, and reminder that there is always something new to learn.

Suddenly 75 tests were failing

A normal work day began, I was close finishing a user-story and ran our Integrationtests again, well aware that some tests would fail because there was a medium sized refactoring happening after I rebased my branch on top of the current dev branch ... but 75 failing test were a little bit too much.

One test that was failing

At first I blamed dbrider, because we had our issues with that library and I thought that it could not connect to our database properly, but that seemed to work fine, so I mentally went through my troubleshooting routine, could it be:

It got even more confusing as my colleague ran the tests on his machine and everything passed, same with our CI-Chain, so all mentioned points above (except the last one) are kinda obsolete now, so what is wrong with my machine?

Switching from Liberica to Temurin also didn't change anything (thank god). So I picked one test and started debugging.

FetchInto()

The tests failed because our endpoints returned wrong data, for example, instead of a number it returned null, or instead of a list with 5 entries it returned 0 entries, in some cases some SQL-Subqueries failed with an exception because the code expected data to come back, but the query returned none.

So by stepping deeper I stopped at our DAO methods, that behaved strangely as soon as we used fetchInto()

List<MyBook> myBooks = create.select().from(BOOK).fetchInto(MyBook.class);

fetchInto is a convenience method that instantiates a provided class with the ResultSet from whatever came back from the database.

For Example, if the Book Record would look like this...

public record Book(String title, String author, long pages) {}

...and the select query would return a row with 3 columns.

The JOOQ-ObjectMapper would take these 3 arguments and instantiate the Book-Object by passing all 3 arguments in that order to he constructor.

With that you save a lot of Boilerplate-Code and the code looks a lot cleaner ... but even though the debug log clearly showed that the row held all necessary data, after creating the object some variables remained uninitialized.

I could theoretically fix the issue by abstaining from using fetchInto and go back to map everything by hand, but hat would be a terrible solution, so I dug deeper.

JOOQs Object Mapper

JOOQs Object Mapper is a behemoth of a class with a LOT of code paths, reflection and mutating code, that enabled us to use fetchInto(). I spend a good chunk of my train ride back home just to find promising looking things to slap a breakpoint on.

At some point I reached ImmutablePOJOMapper. Here some kind of lookup happens:

1this.members = new List[fields.length];

2this.methods = new Method[fields.length];

3if (propertyNames.isEmpty()) {

4    ...

5}

6else {

7    fieldLoop:

8    for (int i = 0; i < fields.length; i++) {

9        Field<?> field = fields[i];

10        String name = field.getName();

11        String nameLC = StringUtils.toCamelCaseLC(name);

12        // Annotations are available and present

13        if (useAnnotations) {

14            members[i] = getAnnotatedMembers(configuration, type, name, false);

15            methods[i] = getAnnotatedGetter(configuration, type, name, true);

16        }

17        // No annotations are present

18        else {

19            members[i] = getMatchingMembers(configuration, type, name, false);

20            methods[i] = getMatchingGetter(configuration, type, name, true);

21        }

22        // [#3911] Liberal interpretation of the @ConstructorProperties specs:

23        // We also accept properties that don't have a matching getter or member

24        for (int j = 0; j < propertyNames.size(); j++) {

25            if (name.equals(propertyNames.get(j)) || nameLC.equals(propertyNames.get(j))) {

26                propertyIndexes[i] = j;

27                continue fieldLoop;

28            }

29        }

30    }

31}

Here the mapper tries to find column names that match the member-variables name of the class we try to instantiate.

So for example if you have the column: book_author, the corresponding POJO class needs to also have a member variable with that name either with an underscore or in CamelCase.

If these 2 conditions are not fulfilled the variable gets dropped entirely, the for loop just continues and later at initialization, non-existing values are just nulled: The exact error we got in our test cases.

Some of our queries don't match this naming convention yet, we never took full advantage of the as() method in JOOQ to give a distinct name to a column, so the names were kinda all over the place with names like "coalesce()", or just the column names that don't represent the POJO member-names.

So one mystery solved, but why does that happen now, and why is that not happening at our development branch now?

Debugging the Develop Branch

So I debugged the exact same test case on the develop branch, with the exact breakpoints, profiles and everything.

This time the test passed and NO Breakpoint got hit, so the ObjectMapper behaves differently somewhere, but where?

After some digging I found this code:

1 if (debugCPSettings = !FALSE.equals(configuration.settings().isMapConstructorPropertiesParameterNames())) {

2    for (Constructor<E> constructor : constructors) {

3        ConstructorProperties properties = constructor.getAnnotation(ConstructorProperties.class);

4        if (properties != null) {

5            delegate = new ImmutablePOJOMapper(constructor, constructor.getParameterTypes(), Arrays.asList(properties.value()), true);

6            return;

7        }

8    }

9}

It checks if the ConstructorProperties Annotation is present at our class that we want to instantiate, on the develop branch this Annotation does not seem to be present, but on our feature branch it is.

When this annotation is NOT present the ImmutablePojoMapper gets also instantiated a little later, but instead of doing a lookup onto the classes properties it just assigns the columns to the constructors argument list one after another:

|BOOK_NAME|BOOK_AUTHOR|BOOK_PAGES| -> new Book(getValue("BOOK_NAME"),getValue("BOOK_AUTHOR"),getValue("BOOK_PAGES"))

This logic is also present inside the ImmutablePojoMapper:

1if (propertyNames.isEmpty()) {

2    if (!supportsNesting) {

3        for (int i = 0; i < fields.length; i++)

4            propertyIndexes[i] = i;

5    }

6    else {

7        for (int i = 0; i < fields.length; i++) {

8            Field<?> field = fields[i];

9            String name = field.getName();

10            int separator = name.indexOf(namePathSeparator);

11            propertyIndexes[i] = prefixes().get(separator > -1 ? name.substring(0, separator) : name);

12        }

13    }

14}

That was the old, expected behavior, so how do I get this one back into my feature branch?

Lombok x JOOQ Interaction

Not that much has changed according to my git diff that could lead to such a drastic change in behavior in so many DAO methods, so what could it be?

But I was overlooking something that had a pretty big impact, a small change in Lomboks config file:

1lombok.addLombokGeneratedAnnotation = true

2lombok.anyConstructor.addConstructorProperties=true

Then it hit me, that was a shotgun approach to fix a totally unrelated Jackson deserialization-problem that I had one week prior, and I never realized the severe consequences that this small did to our backend.

In short, we had to deserialize a JSON and Jackson failed with an Exception, and after throwing several fixes at the project some internet blogs or Stackoverflow proposed, that one did indeed fix it, but also introduced that exact new problem.

Reverting the change made all tests pass again instantly.

So what happened?

Java has a ConstructorProperties Annotation gives Deserialisers some extra information about in what order to pass the fields of the serialized object to the constructor of the POJO.

This Lombok-Flag tells the java compiler to auto-generate these properties for every single class within the project.

Little did I know that the JOOQ-ObjectMapper changes behavior when a class has this Annotation attached to it.

Instead of just calling the constructor with the column values ordered as they appear in the ResultSet, it tries to map the column name to a property-name this annotation generated.

Because a lot of column names do not match this naming convention, this algorithm could not match every property to a column, so it left those fields blank and instantiated the Object with leaving everything unmapped null.

Invalid mapping from columns to fields

Closing words

I needed some time to process what just happened, by flipping a switch of a setting in Lombok to mitigate an issue I had with Jackson, I also completely broke JOOQ without noticing because instead of throwing an error or printing a warning, JOOQ just returned incomplete data (with default settings).

I guess that is the dark side of such convenience libraries, you need to have extensive knowledge about what the side effects are, that some flipped switches might introduce in the behavior of code deep down in libraries you use.

Due to the global nature of the Lombok config file this might be an extreme example, but it happened and I learned a lot this way about all the involved libraries.

Takeaways:

TL;DR