Successful modularity depends on your dependency model
By Peter Kriens
What We Depend On
Jigsaw is implementing a module system for Java 8 that deviates from the (modular) spirit of Java in a way that is bad for the language and will cause untold headaches for the Java community in the future. In this article, I will argue that the Jigsaw design does not fit Java because it ignores packages and uses the wrong dependency model by exporting types and importing modules.
If Jigsaw is a module system then we first need the definition of a module:
A module encapsulates content and explicitly shares some of this content with other modules.
An ideal module includes the following kinds of content:
- Private - Private content can be modified without affecting any other modules. Changes in the private content can be treated in local scope; no external party can see the private contents. For example, private fields in a Java class can only affect the owning class and cannot be used by other classes.
- Exported - Exported content is shared with other modules. Changes to exported content affect other modules; it therefore has the burden to evolve in a controlled way as it cannot know its dependents. In Java, the public keyword implies exporting.
- Imported - Imported content comes from another module. The importing module is explicitly depending on the imported content. The importing module has internal assumptions about this content that are verified by the compiler and should be satisfied by the VM during runtime.
An archetypal module can therefore be depicted as:
Modules provide a large number of benefits. However, the most important benefit is that they limit the scope of a change to a software system. In a non-modular world, a change can affect the whole system; in a modular world, the change is limited to the private content and any exported (public) content.
The larger the system, the more benefits this scope reduction provides.
In our definition of a module, the concept of the contents has been deliberately kept vague. The reason for this vagueness is that modularity is a pattern that can be applied at different levels. In the past 60 years we’ve applied this pattern to software constructs multiple times. Though one can argue that a function resembles a module, the clearest example is the Java class.
Classes, modularity and content
If a Java class is the module, then what is the content? For a class, the members (fields and methods) are the contents. Picking the circle as the class symbol and diamond as the member content symbol we can depict the class modules as follows (black is exported (public), grey is private, white is import):
Java also has another module, the package. The Java Language Specification (JLS) compares the package with Modula’s module. It should be clear that the content of the package is the class, symbolized with a circle. So if we use the square symbol for a package we can depict the packages as follows:
From the public discussion, one sometimes gets the idea that JSR 277, JSR 294, OSGi, and Jigsaw (and numerous others) are discussing Java modules. This is wrong as modularity is not a concrete thing but a pattern that is repeatedly applied to the previous module level. The whole discussion is therefore about how we express the next level of modularity in Java.
Where we all agree...
There is one thing all discussions seem to agree on and that is what the next module looks like: something like a JAR. But then the agreement stops. The difference is in what the contents of the JAR should be, how the imports are expressed, and how the public content is exported. However, since we all roughly agree on what the next level looks like, we can at least provide it with a symbol: let’s pick the rounded rectangle. The picture for a Jigsaw module then looks like:
This picture looks quite different from the class and package pictures since Jigsaw modules export types but then they import modules. From a symmetry point of view this is kind of surprising for Java, since both the class and package modules imported and exported their contents.
Symmetry and Proof
Types import/export members, packages import/export types, and it seems natural that the next level, the module, will import/export packages. Instead, Jigsaw exports types (the member of a package) but imports modules (the parent of a package). It basically acts as if packages do not exist. This is clearly seen in the following UML diagram where the Jigsaw design is contrasted with OSGi (closed diamond is containment, open diamond is aggregation):
If we depict the OSGi model it looks like the natural successor in Java:
Symmetry is no proof, but history has shown us that symmetry often leads to the simplest solution. In Java, the VM gives us great leeway to manipulate the visibility of classes and perform all kinds of interesting hacks. Though certain hacks can be very useful (in the short run for some use cases), they rarely outweigh the long-term costs of increased complexity for all of us.
I loved Java in 1996 because it was so much more pure than C++. Unfortunately, they did not go all the way. Java primitive types and arrays might have sounded like a good idea at the time but we’re still paying the price today for these hacks. Look at generics and reflection, for example.
So should we not use packages as the granularity of import/export for modules? Packages are the highest level modules in Java and they are routinely used as cohesive entities. For example, in the JCP almost all JSR’s specify the semantics and syntactic content of one or more packages: JPA (javax.persistence.*), JTA (javax.transaction.*), JDBC (javax.sql), XML parsing (javax.xml.parsing), etc.
Would it not be a grand idea to depend on javax.persistence version 1.1.2 instead of naming some JAR from a random provider? Isn’t this the real thing we really depend on our code? Why do I have to make a choice between artifacts from Apache Geronimo, Oracle, JBoss, or SpringSource to provide me that package when all I really depend on is the well defined javax.transaction package in my code?
There should at least be some reason why Jigsaw deviates so strongly from the current Java architecture. Unfortunately, I’ve never been able to find any that make sense. On the contrary, the deviation causes two acute problems:
- Expressing the imports on modules creates a parallel universe of dependencies.
- Treating the classes as units of exports violates the package modularity.
Your class files embed type imports that fully specify the minimal dependency graph for type-safe code: “Give me your code and I will tell you what you depend on.” Though the class files only contain the type name, a Java type implies a package as part of its name. The relation is full containment.
A package is modular, and its actual contents must be treated as a black box that is not under the control of, or even visible to, outsiders. The layout of a module is a private design choice that is decided by the designer. In a modular system, importing a type implies importing the complete package.
Therefore, it is possible to construct a full dependency graph of packages from your type-safe code. On top of this graph, Jigsaw creates a universe of module dependencies that are parallel to your code dependencies and, hopefully, overlap with the code dependencies.
A Jigsaw module imports another module and uses this information to create a module path. The VM searches the module-path and should find the requested type in one of those modules. For example, a module X looks for type p.T that happens to be in module C. The VM can find this type by searching module A, then module B, and finally finding it in the provider module C, as depicted in the following figure.
For the Java language, there is no containment relation between module C and type p.T. Maintaining the same dependency twice provides ample opportunity for errors because there is no language relation between the two universes so the compiler cannot verify the relationship.
In this article, I argue that a method, a class, and a package must be treated as an atom. Picking and choosing is not allowed. It is all or nothing. Is this also true for a Java module? Well, the JLS neither has rules about Jigsaw modules nor OSGi bundles. So in this case, we can actually allow the packages to move between modules.
We can even allow multiple bundles to export the same package (versioned of course!). Packages are atoms, and have the same access rules for any package regardless of its origin. There might be constraints, for example interpackage dependencies, but they are part of the design of the module and therefore cannot logically break dependants.
Being able to move packages between modules, and even replicating them in different modules, is a powerful concept that can be used to fight the transient dependency problem that causes Maven to Download the Internet.
It is not Maven that downloads the Internet - people download the Internet! This is true, despite the fact that Maven shares the same parallel universe dependency model that makes it easy to inherit too many dependencies in the Jigsaw design. So Maven already suggests that artifacts should be better designed to minimize their transitive dependencies. This will require refactoring existing artifacts.
For example, a module is refactored into two separate modules because it turned out that a single package dragged in a very large library. So let's say we refactor module B into B’ (the new version of B) and a module C that now contains the culprit package.
In Jigsaw, such a refactoring ripples through all our dependants as they express their dependency on module B. The module dependency graph does not express the code dependencies, so we must explain to all our users (if we can find them) what happened and that they must now include module C as well. In the example, we must version module A into A’ by updating its module dependency on modules B’ and C. Since we’ve changed A, we must find all its dependants and they must be updated as well, ad nauseum.
However, when we look inside the modules we see that the actual dependencies are such that module A’ no longer has a dependency on module B’ as the only dependent package y moved to module C. If we could have expressed this dependency on package y then module A would not have been affected by the refactoring of module B at all; the resolver would automatically bind it to module C and not module B’. This is depicted in the following figure:
The parallel universe of module dependencies creates a brittle evolution model. Even minor changes and refactoring of a module ripple through all dependants. It is hard to overestimate the cost of this ripple effect.
The other problem with the Jigsaw asymmetry is that it violates the modularity of the package. Jigsaw imports modules but it exports types. Packages are left out of the design and consequently two different Jigsaw modules can contribute types to the same package.
The primary advantage of a module is that changes to the private parts cannot affect others. However, if two different artifacts can contribute to the same, split package then we no longer have privacy.
For example, a package implements a Complex type and uses real and imaginary float. These are package private fields so that other classes in the same package (a Complex Vector class for example) could directly manipulate the fields for performance reasons. If we treat a package as a module we can change the imaginary and real parts to polar coordinates because we know that all the classes that are potentially impacted by this change are in our package and thus under our control.
However, if somebody else could contribute to our package (for example a Complex Matrix) we can no longer optimize or refactor since we never have knowledge of who depends on us. And even if we do, we can rarely ensure that they adapt to our changes on our schedule so we usually just break them, often in production.
Package dependencies seem so obvious that I am always a bit startled by the sometimes fierce opposition to such a simple idea. So let's take a look at the arguments of the opposition.
The most common objection is that it is too hard. There are simply too many packages to be specified by hand. I agree. Fortunately, we are already specify our imports:
Even better for us, the compiler records every class reference in our class file, and by implication the package. Tools like BND and SpringSource’s Bundlor can pick up these dependencies from the class files and generate the proper metadata for the modules, no need to specify dependencies by hand a second time.
Public packages need to be versioned and there is work involved in that. However, this work is usually less than maintaining a module version. Package versioning is easier to automate and verify because the rate of change in a public shared package is by definition significantly smaller than a module that aggregates many implementation packages as well as API packages. (It always puzzles me what those aggregate version number really mean…)
The Jigsaw model creates a parallel universe for dependencies while package dependencies already exist and do not go away. So how can an additional universe on top of the existing model make things simpler?
The second objection to package dependencies is that finding the artifact that contains that package is too complicated for the tools. The argument is that when the developer specifies some logical name for the artifact we can easily map this name to a URL by concatenating it with the URL of a repository. Therefore, we can get away without any logic for the repository.
Simplicity is important but before simplicity there should be correctness and a broken design is not correct. Maven, for example, supports version ranges and therefore cannot just construct a URL anymore. Version ranges require the same kind of logic as finding the providers of a package. Bolting this logic on a simplistic model is a lot harder than taking this into account from the start.
The third most common objection is that depending on packages creates a rich space of possible sets of artifacts that can provide these packages. Resolving to the optimal solution in this set is an NP-complete problem so it is really, really, really hard. Duh? It is not as if resolving routinely involves hundreds of thousands of packages. Even in very large systems the number of shared packages is in the low thousands and with very few providers.
In a modular µservice-based system, where much less needs to be shared, this number is significantly lower. Many problems can be NP-complete but routinely resolve because the practical day-to-day situations never run in the extremes where NP-complete problems cry wolf.
Both Apache Felix and Equinox can resolve large systems in a very short time. Ever used a navigation system and noticed that they actually come up with a very good solution in a very short time? Their search space is much larger and more complex than ours.
And even if there are too many packages originating with too many artifacts, there is no reason the resolving has to happen at runtime. This is the typical OSGi usage model because it allows us to have dynamicity. However, dynamicity is the superset of static, and static is the easy part. If, for example, production and test require exactly the same solution then systems can be resolved ahead-of-time and not have an impact at runtime.
OSGi started with package dependencies without much thought. It just felt natural. We did not have the insight that I can now share with you in this article. However, I do recall being flabbergasted when we started to work with Eclipse in 2003, and they insisted on express dependencies on modules. We provided them with Require-Bundle (and fragments) as it was clear that we would lose them without this concession.
Though there have been a few cases where Require-Bundle was really useful, it was in general a major painpoint for later OSGi specifications. For example, in a previous OSGi release we wanted to put a composite model in the core, a set of bundles that would look like a single bundle. Package dependencies handled this with elegance and grace. The complications of the interactions with Require-Bundle and fragments added more greyness to my already grey head and we therefore moved it out of the core.
Finally, as the diagrams of the different modules look like an IQ test, why not treat it as such? Of the following forms of modularity, which one does not fit?
There is no doubt in my mind that Jigsaw makes a grave mistake ignoring the elegant existing modularity aspects of the Java language. Breaking the privacy of packages by ignoring packages in the design will unnecessarily create lots of pain with virtually no gain.
Maybe the VM modularity needs split packages for historical reasons, just like James Gosling felt primitives were crucial to support 1990 hardware. Smalltalk had already proved that primitives could effectively be hidden behind an Object Oriented façade and in the same vein OSGi has proved that split packages are not necessary. Ignoring these lessons is unfortunately putting the Java community in peril.
16 Aug 2011