Cool java tools

Power

Dagger
Mapstruct
Autovalue

Safety

Error-prone
Pure4j

Writing your own

Javapoet
Autoservice

Release workflow

This revision control workflow is designed keep you sane.

There are just two types of long lived branches:

• master - This branch lives forever, and all changes are eventually merged to it.
• release branches - These are branched from master for each major release candidate.

Master

New features are implemented against master, by merging feature branches into it. Feature branches should be as short lived as possible, because keeping them up-to-date is expensive. If your feature can't be implemented within a single sprint, consider branching by abstraction.

Release Branches

Release branches are also known as "stabilization branches". Bugs that should be fixed for a release are first merged into the corresponding release branch, and then merged forward to all later releases and then to master.

Change as little code in release branches as possible, because merging to code that has later changed is expensive.

Each published build of a release branch should be tagged so that it can be easily identified and ordered.

Example

In June, we branch master to release/1, and start testing it.

In July, we branch master to release/2, and start testing it.

In August, we find a bug in release/1.

We branch release/1 to bugfix/555, and fix the bug on the bugfix/555 branch.

We merge bugfix/555 to release/1, to release/2, and to master.

Feature development in July and later does not create risk for release/1, and even bugfixes for release/2 do not create risk for release/1.

Tracking Bugfixes

It is straightforward to automatically merge most changes from older releases to newer, and to report regarding changes that have not yet been merged. We want to avoid regressing a bugfix in a later release just because we forgot to merge.

Using the merge history to keep track of what fixes have been applied to releases sometimes requires doing a "trivial merge". Even when there is no change to be made to the later branch, we still have to merge, to inform the revision control software (e.g. git) that the bug is fixed in the later branch.

Cherry picking

If we merge a bugfix into a release branch and only realize afterwards that it should've been fixed in an earlier release, we must cherry pick, and trivial merge as described in the previous section.

Hotfix Branches

It is good to decouple bug fixing from the choice of exactly which build is deployed into production, especially when there are multiple production environments that have different risk profiles. Therefore, the production tag could be different from the tip of the release branch. When this happens and we want to make an urgent fix for that production environment, we don't want to jump to the tip of the release branch. We haven't yet tested all the changes in the release branch, so we hotfix branch from the tag that we have tested.

Example

In September, we deploy the tag 1.2 of release/1 to the London production environment.

In October, we make five bugfixes to release/1 in preparation for its 1.7 New York release.

In November, we discover a critical global bug in release/1.

We branch tag 1.2 to hotfix/1.2-LN, and fix the bug on the hotfix/1.2-LN branch.

We deploy tag 1.2-LN1 of hotfix/1.2-LN to the London production environment.

When we have leisure, we merge hotfix/1.2-LN, to release/1 etc.

Advanced Topics

Gitflow and ProdFlow

The "gitflow" workflow from nvie has two problems: it confusingly changes the meaning of the "master" branch, and it doesn't clearly justify the cost of an additional long lived branch. Both problems could be resolved by tweaking the workflow to support multiple production environments. This is valuable if you actually need to support multiple production environments :). Let's call this improved workflow "prodflow".

Prodflow has release branches and master-as-trunk. Instead of one branch for release history, it uses a branch for each production environment, prefixed with "prod/". So when we deploy release/1 to production in London, we merge it to prod/LN.

An advantage of this approach is that it simplifies hotfixes. Neither gitflow nor prodflow really need hotfix branches; you can just merge bugfixes first into the prod branch and then into release branches and master. We don't need to keep track of the version in production.

Instead of: "

We branch tag 1.2 to hotfix/1.2-LN.
We branch hotfix/1.2-LN to bugfix/777...

" in the example above, that'd be just "We branch prod/LN to bugfix/777...".

Deserializing Merges

When there is a conflict, merging bugfixes from one release branch to the next can be a pain. Developers might not do it quickly, and this blocks merging subsequent bugfixes. We accumulate a backlog of unmerged changes, and it becomes increasingly onerous to work it down, especially with proper code review.

The solution is to "deserialize" merges. When bugfixes are made to different parts of the code, one needn't be blocked by the other. We can convey this to git using "rebase --onto", so that the parent commit of the bugfix branch is a commit that is already present on all the long lived branches. If we didn't rebase, ordinarily the parent commit of the bugfix branch would be the tip of its release branch, and the author of that commit might be procrastinating her merge.

Example

We merge bugfix/888 to release/1 but not to master.

Now we'd like to merge bugfix/999 to release/1 and master without being blocked by bugfix/888.

Before merging to release/1, we run: git rebase --onto origin/master bugfix/999

And it Just Works!

Conclusion

So that's the git release voodoo I learned over the last couple of years. I hope you find it useful!

Posting cljs to get around an unkind firewall.

Extensible Software and Dependency Injection

tl;dr For Dependency Injection of a collection of implementations of an interface, just inject the concrete classes of those implementations.

Supporting change is a big challenge in programming. We want to make the most likely kinds of change quick and easy. How do we do that with Dependency Injection?

So if you have code that does five things that all have the same pattern, you'd like to be able to easily add a sixth. Common examples are programs pulling from multiple sources of data, and programs performing multiple validations. The technical name for this kind of thing is the Open Closed Principle.

If your five things are complicated, their code might organically spread out over your system...

setup();
doWork();
cleanup();

void setup() {
    setupForDataSourceA();
    setupForDataSourceB();
    ...
}

void doWork() {
    workForDataSourceA();
    workForDataSourceB();
    ...
}

...

The idiomatic way to handle this in java is with interfaces:

interface DataSource {
    void setup();
    void doWork();
    void cleanup();
}

So each data source implements the interface, and many parts of the general code loop over a set of instances of the interface.

for (DataSource datasource : datasources) {
    datasource.setup();
}
for (DataSource datasource : datasources) {
    datasource.doWork();
}
...

If you're using Guice for your dependency injection, when you hear the word "set" you might be tempted to use multibindings.

Or you might think, "wouldn't it be cool to add a new datasource without having to change any existing code". You could write a whiz bang code generation framework that automatically wires up implementations, either purely because they implement the interface or when they are additionally annotated.

Keep it simple sweety! Remember the original goal: making software easy to change. When the software is too clever, it becomes harder to change over time, especially as multiple programmers work on it.

There's a simple way to do it in Guice:

@Singleton
class DataSources {
    Set datasources = new HashSet();

    public Set getDataSources() {
        return Colletions.unmodifiableSet(set);
    }

    @Inject void setDataSourceA(DataSourceA datasource) {
        datasources.add(datasource);
    }

    @Inject void setDataSourceB(DataSourceB datasource) {
        datasources.add(datasource);
    }

    ...
}

This works without any additional binding configuration, using Guice's JustInTimeBindings support for eligible constructors.

This wiring is vanilla java, and surprisingly not Guice's embedded domain specific language. These implementations aren't typical injected dependencies, because the design of the application requires that you run lots of them.

Don't be scared by the idea of a concrete class, because the framework code still only ever relies on the interface. The DataSources class above has the same testing burden as Guice modules. If you don't unit test your Guice modules, don't unit these classes either.

In addition to being more simple than multibindings, this pattern is much more explicit because the full configuration is in one place. Multibindings allow multiple Guice modules to add to the one binding.

It's one of the few cases where setter injection is best, though it works fine with field and constructor injection too. Setter injection enables you to add new instances by editing a single location in the composing class. It also allows you to simply subclass in order to compose an additional group of instances.

Test Driven Development (TDD) recommends first writing a failing test.

Continuous Integration recommends committing early and often.

How do you do both and not break the build?

How do you distinguish between test failures associated with regressions and test failures associated with unfinished development?

Using Junit @Rules!

Instead of just disabling a test using the Junit standard @Ignore annotation, you can annotate a new test using @NotImplemented. The result of these tests will be inverted; the automated build's junit test will succeed if and only if the actual test logic fails.

That way, when the application functionality is still incomplete the actual failing test will not break the build. When the functionality is ready, the inverted test will start breaking the build, so that you don't forget to enable it.

And the @NotImplemented annotation provides a machine-checked way to track known issues.

Here's the implementation:

import org.junit.rules.TestRule;

import org.junit.runner.Description;

import org.junit.runners.model.Statement;

import java.lang.annotation.*;




/*

 * To use in a testcase, include a line:

 *

        @Rule

        public NotImplemented.MustFail notImplementedRule = new NotImplemented.MustFail();

 *

 * and annotate individual test methods:

 *

        @NotImplemented @Test

        public void someTest() {

                ...

        }

 *

 */




@Retention(RetentionPolicy.RUNTIME)

@Target(ElementType.METHOD)

public @interface NotImplemented {

        String value() default ""; //to enable grouping tests; doesn't affect tests

        boolean invertTest() default true;




        public static class MustFail implements TestRule {

                @Override

                public Statement apply(final Statement base, Description description) {

                        NotImplemented annotation = description.getAnnotation(NotImplemented.class);

                        if (annotation == null || !annotation.invertTest()) {

                                return base;

                        } else {

                                return new Statement() {

                                        @Override

                                        public void evaluate() throws Throwable {

                                                try {

                                                        base.evaluate();

                                                } catch (AssertionError e) {

                                                        return;

                                                }

                                                throw new AssertionError();

                                        }

                                };

                        }

                }

        }

}

Limited video for kids

Few parents can completely resist employing the video-as-babysitter.

Parents that use linux can easily have their computer play just one movie with the following script:

if [ "$1" ]
then
 totem --fullscreen "$@" &
fi
xtrlock

It starts playing the video(s) you want, but then prevents the kids from playing other videos or engaging in other unsupervised fiddling with the computer.

xtrlock needs to be installed, easily for example from the standard Ubuntu Software Center in Ubuntu 10.04.

You can create a Launcher in the panel for this script and a %F paramter, and drag videos to it from the File Browser.

Google Strategy

Google has three main strategic product categories.

Google has login-based services, like Gmail. These are most like other companies' internet services. They also have the most lock-in effect, though Google expends some effort to reduce it (for example allowing downloading and forwarding away email messages). These services enrich Google search data with user identity data.

Google has internet growth services, like Chrome, Android, and News. In general, companies would like to reduce "substitutes" and increase "complements". Since Google has dominant marketshare of internet search, anything that increases internet use is a complement. Google can run ads on all of these services, and to varying degrees their users are more likely to be Google search users. Developing these services is also a little like Military Keynesianism. It gives Google engineers fun projects to work on, helping Google to get and keep top engineers, who help it to maintain its internet search dominance.

Google has the internet. Since Google is the king of search, it will vigorously defend the internet against closed competing networks like the new social networks Facebook, LinkedIn, and Twitter. At least three distinct Google projects, though they also occupy the second category, attempt to open up the social software world: Social Graph API, Open Social, and Wave. This makes Google "the good guys," because they'll always try to bring the fight to the open web, where they have the competitive advantage. Now, if Google didn't have a challenger, it might just release something into the first category (Orkut for example), but when they do have a competitor, everyone will end up being a lot better off. Wave promises to be a wonderful open technology, though it will put a lot of social software companies out of business.

Temporal and Bitemporal in One Sentence

Temporal database support allow efficiently answering questions like "What was the state of this entry on Monday?", and bitemporal database support, questions like "What was the state of his entry on Monday, if I had asked on Tuesday?"

Distributed Revision Control Yields Centralized Advantages

The new distributed revision control systems (DRCS) have some important advantages for traditional use, apart from the ability to commit without an internet connection.

DRCS make code review much easier. Instead of using an entirely different system for managing the code review process, developers can check all of their un-reviewed changes into a branch, and merge changes as they get reviewed.

DRCS can support graduated continuous integration. In sufficiently large projects, continuous integration can cause frequent build breakage. Instead of having a single, volatile HEAD, subprojects can have their own independent integration branches, which they merge to the superproject integration branch at longer intervals.

DRCS support "lines of development", or working on multiple independent features. With traditional RCS, once a feature is committed it becomes part of the blob, and may not easily be dealt with as a separate identity.

The original motivation for DRCS was empowering decoupled open source contributors, so it's mildly ironic that this functionality should be so valuable for BigCo development as well.

Libertarian OS

What would an operating system look like if it were designed to maximize freedom of development? Could we support a web of code to match the existing web of documents?

It'd need:

• A security framework
• A dependency framework
• No extra baggage
  

For security, programs would be extremely limited by default. They'd have no network, filesystem, or driver access. In order to do anything, they'd use APIs to

• request reading or writing an existing file, selected by the user
• request writing a new file
  
• suggest a url to visit (optionally with post data).

This restricted behavior could safely allow running programs from even the least trusted sources. It's in the spirit of the "One laptop per child" project's revolutionary security model Bitfrost.

For dependencies, programs should specify urls (possibly self-certifying) of code that they depend on. All code could be cached locally or downloaded anew as desired. This is in the spirit of package management approaches without "side effects", like Nix and Zero Install.

Once there's a sufficiently powerful dependency framework, we need not hard code any dependency on "standard software" like programming languages or gui toolkits. Programs could be transported in llvm bytecode.

Implicitly trusted programs could be subject to phishing attacks just as websites currently are, so it would be desirable to have some sort of petnames scheme, as well as a spoof-resistant UI for distinguishing programs.

Dinosaurs

My daughter, 4 years old: Daddy, are dinosaurs kosher?
Me: No.
My daughter: Not even little purple ones?

"Model" in MVC

The "Model" in Model-View-Controller architecture does not refer to the application data model. It refers to the user interface model underlying each widget.

Think of a text box that accepts date values. When the user types "2006-" and then pauses, the application doesn't yet have a date, so it doesn't have anything to put in its data model. The string "2006-" isn't stored somewhere in the view; it's stored in the model that backs the UI.

To integrate your UI and your data model, you've got to use data binding code, like the jgoodies derived data binding code. It would seem that MVC is more about UI framework implementation than interface.

This explains why it's so cumbersome to switch between GUI widgets that would appear to represent the same data, like a select box and a group of radio buttons. It'd actually be against the MVC philosophy to support this, because it'd put artificial constraints on the respective widget models.

Consistent Types

There's a problem with inconsistent downcalling, which I've been hankering after for a bit, and which C# supports using the "new" keyword.

The problem is that extension classes aren't really subtypes of extended classes anymore. They violate the Liskov Substitution Principle.

Interfaces for encapsulation don't have this problem.

(Though it'd be great if all languages had something like c#'s "override" keyword :).)

UPDATE: I was mistaken. Different methods are called on an object in different static "contexts". For example:

((ArtisticCowboy)joe).draw() runs one method, and
((Cowboy)joe).draw() runs another.

This is what C# does, but I don't know if it's wise.

Future Refactoring UI

Refactoring tools have a good case of programmer automorphism. These tools present silly lists of operations like "decompose conditional" and "encapsulate downcast."

Really, automatic refactoring means the computer automatically makes all appropriate changes to accompany your change. It's all about moving and renaming. The tools should get rid of their complicated interfaces, and just support a mode "don't let me introduce any errors."

Simple changes are easy to spot, but moves need to be done with Cut&Paste. For example, cutting from one place and pasting to another should fix up the pasted code to work in the new location. It should also fix up callers to the code at the original location so that they now call to the new location. The user shouldn't have to specify what type of code is moving, or what type of move it's making.

If this could be made fast enough, programmers might want to always work this way. They'd just have to get in the habit of defining before using, and deleting usages before deleting definitions.

Everyone is Like Me, and bad UI

User interfaces suck because programmers think users care about programs.

Users do not care about programs. They think, "Just do it," and, "Get out of my face." Programmers love their programs, so they think everyone else does too. For example, they even write programs that hassle users to make implementation decisions.

In general, every person thinks that everyone is just like himself. Worse, every person thinks that everyone is like himself right now. In the linked example, even the developers won't care about the implementation a year after they've coded.

There is a word for this, though it seems seldom used: automorphism

Interfaces for Encapsulation

The python people haven't succinctly explained why python doesn't need "private", "protected", and "public". In one sentence, python doesn't need visibility modifiers becase...

With sufficiently good refactoring support, encapsulation decisions don't affect code evolution.

Especially according to the XP design principle of collective code ownership, it's easy to change callers of an interface when the interface changes. Though people reading the code will need to understand the changes, that's far more efficient than making artificial barriers to reuse. Encapsulation is powerful; forced encapsulation is onerous.

This is true for a single organization, no matter how large its code base. Once code is "published" however, all bets are off. Code obviously can't be refactored across administrative boundaries. That being the case, we should care about "published" instead of "public".

For encapsulation of published code, separate interfaces are much more useful than visibility keywords. A class with a single public method is better represented by a separate interface containing a single method. When that interface is published, it never changes. In Java for example, the old interface could be kept in a separate jar file, which clients can continue using even after extended versions become available.

More originally, we should allow extending code through an interface. Extending through an interface means that new code will never downcall to a method that isn't in the interface. This is an elegant solution to the fragile base class problem. (Someone recently showed me the cute new c# modifiers "new" and "override". They protect against accidental downcalls, but they don't protect against replacment of methods called by collaborating classes.)

Extending through an interface might also help with downcalling into unrelated methods with mixins and multiple inheritance. For example, to create an "artistic cowboy" with multiple inheritance, we'll inherit from "artist" and "cowboy". Though we want his artistic nature to be dominant, we don't want the cowboy method "drawAndShoot()" to downcall into the artist method "draw()". We can achieve this by extending artist through an interface that excludes "draw()".

DRM and the Little Guy

Maybe published work isn't best treated like property, but publishers will certainly fight to keep it that way. Physical media have been pretty good at proxying ownership of information. If general purpose computers can't do as well, then publishers will just adopt special purpose devices instead. Besides keeping consumption cumbersome and inefficient, this'll give binding power to corporations, but not to individuals.

A general purpose DRM system can have a significant impact on society. If we're going to allow our computers to enter into contracts for us, we need some safeguards to avoid creating an information dystopia. The restrictions that we allow publishers to place on consumers should themselves be restricted. Generally, it shouldn't be possible for publishers to artbitrarily revoke access to their work. The set of digitally managed "rights" should be standard, easy to understand, and when possible, it should not require an ongoing relationship with the publisher.

This task is more difficult from the technology standpoint, and it'll detract from the generality of the full DRM dream, but it could work.

Money and Politics

So long as the only means of engaging in political dialogue is through purchasing expensive television advertising, money will continue by one means or another to dominate American politics.
--Al Gore

Templates for Environments

(The more surprising points are below.) Templates are useful for maintaining different deploy environments. Any large system needs to have a painless way to run in different environments, at least one "production" environment and one non-production environment. Examples of things that change between environments are: machine names, port numbers, database names.

Manually switching code from one environment to another is tedious and error-prone, but not all tools can make it completely automatic in all cases (for example, enabling stored procedures to run against differently named databases). Theoretically, templates are also be useful for unifying environment configuration across languages, but they're really essential for languages that don't support the necessary abstraction.

Even on platforms that have the necessary support, environment management can get gnarly. For example, java has a great configuration mechanism in the form of "properties" and the improved "preferences". If different sets of property files are used, however, it's easy for the non-production property sets to get out-of-date. The solution is to ensure that only a small fraction of properties are environment specific, and put only those in a separate file.

Templates generally break tools for live editing. For example, you can't "alter table" in a test database and easily have that change apply to your template. Or use a gui wizard to maintain templatized xml files for your application servers. We'd like to be able to "round-trip" templates, to be able to automatically generate templates from live files, as well as generate live files from templates. For this to be easy, the templates must meet two requirements:

1. there must be a one-to-one correspondence between template variables and the values in the environment used to generate templates.
2. template values must be unique.

An exampe of #1 is that a test environment must have as many different machine names as there are in production, even if those machine names are just aliases. Otherwise, the test environment will lose information.

An example of #2 is that a machine can't be named "foo" because substituting a template variable like "@FOO_MACHINE@" will also substitute "buffoon" and "foot".

Otherwise, the template generation will have to get complicated.

Google Maps

Google maps should support saving and restoring locations.

It should also support taking the union of the sets of locations created by different API sites.