Let's conversate about the essence of Agile

It was bound to happen for me to have a post on Agile, and now here it is...

Many people have many different opinions on this hyped topic full of
buzz words and passionated feelings. I've spend some years on different kind of Agile projects. 'Different' in the way they adhered to the Agile principles. And this allows me to distill the essence. For me it's not about a complete set of 'rules', although they help if you still need to grow as a developer (see my first of two points).

In my first Agile project, there was a strict following of all the sub methodolies defining Agile (like burn down, TDD, stories, scrum, pairing, the manifesto, ...). The projects after that, were also Agile, but less strict, or less compliant to the rules if you will. They for instance did not adapt all of the principles such as burn down or pair programming.
Each project had valid reasons for their level of compliance. For instance, one of the goals of the strictest project was that half of the developers (junior Java developers) needed to be trained to later perform maintenance on the project, after the senior developers left the project. Pairing serves this need perfectly. The other projects had more senior members, which has an impact on my first essential point.

Colleagues who have never worked according to the methodology often ask me - while barely hiding their scepticism - what 'Agile' exactly is. According to me there are two essential things that all Agile projects have in common:

1. First of all, you need a team of *disciplined* and *courageous* developers. Seriously. Disciplined enough to test and doubt everything, even themselves. Courageous enough to refactor that complicated piece of code or to say 'no' to your boss (refuse to harm the code cfr Uncle Bob).

2. Second of all, the way the development team interacts with the team that determines the requirements. This can be the analists, proxy customers or customers.
I read an excellent Martin Fowler blog post which explains this way short, clear and to the point: Conversational stories.

In the post you can see that Agile, deals with requirements in conversation, sometimes even negociation. The early developer feedback can lead to changing, adding or dropping some requirements.

In the waterfall approach, the analists are super human creatures, who can never fail. They will never be wrong, nor forget the big picture and always
use the proper level of detail. Surely, they must be, since a developer never sees them but needs to implement what's described in their documents.

There are always questions about details that only pop up when you start the implementation. Even if the use case accurately described 90 percent of the cases, how do you handle the other 10%? If there are 3 steps described, what does the software need to do when step 2 fails? And so on.

As explained in the post, conversation is the answer.

Spring Webservices

On my current project I'm using Spring Webservices to interact with external companies. This contract-first framework is easy to use and hides most of the boiler plate code, so you can fully focus on the business part.

Here's a basic example that uses XmlBeans to marshall/unmarshall the XML to and from java objects. (You can also use Jibx, Xstream, Castor, JAXB, ... ).
All you need to do to use the XmlBeans marshaller, is pass it to the super constructor. The AbstractMarshallingPayloadEndpoint is usefull when you only need to process the payload XML. Only the XML is passed in as an argument to the invokeInternal() method.

public class MyEndpoint extends AbstractMarshallingPayloadEndpoint{

private final MyService service;

public MyEndpoint(MyService service, Marshaller xmlBeanMarshaller) {
    super(xmlBeanMarshaller);
    this.service = service;
}

protected Object invokeInternal(Object request) throws Exception {

    XmlBeanRequestDocument requestXml = (XmlBeanRequestDocument) request;
    Object result = service.findById(requestXml.getRequest().getId());

    //marshall the result
    XmlBeanResponseDocument responseXml = XmlBeanResponseDocument.Factory.parse(result);
    return responseXml;
}
}

You can declare your endpoint in a number of endpoint mappers. One of the easiest to use, is the payload endpoint mapper. The mapper is an interceptor that looks at the root element of the payload to route the message to the proper endpoint.

<bean class="org.springframework.ws.server.endpoint.mapping.PayloadRootQNameEndpointMapping">
<property name="mappings">
    <props>
        <prop key="{http://mycompany.com/hr/schemas}MyRequest">myEndpoint</prop>
    </props>
</property>
</bean>

<bean id="myEndpoint" ... /> 

The advantage of this kind of mapping, is that you can host all your webservices on the same url.

Let's say you want access to more than just the payload, meaning you also need to deal with attachments. Here is a simple example giving you an idea on how easy it is to implement your custom endpoint that does this :

public class DetachEndpoint implements MessageEndpoint {

public void invoke(MessageContext messageContext) throws Exception {

    WebServiceMessage requestMessage = messageContext.getRequest();

    if (requestMessage instanceof AbstractSoapMessage) {

        @SuppressWarnings("unchecked")
        Iterator attachments = ((AbstractSoapMessage) requestMessage).getAttachments();

        while (attachments.hasNext()) {
            Attachment attachment = attachments.next();

            ...           //insert business logic here

            //System.out.println("attachment = " + attachment);
            //System.out.println("attachment datahandler = " + attachment.getDataHandler());
            //System.out.println("attachment content-id: " + attachment.getContentId());
            //System.out.println("attachment content type: " + attachment.getContentType());
            //System.out.println("attachment size: " + attachment.getSize());
        }
    } else {
        //throw some sort of exception    
    }
}
}

The messageContext gives you access to the attachments and to the payload XML via requestMessage.getPayloadSource(). This is very basic to show you the principle. A better way to do this, is to have an AbstractMarshallingPayloadEndpoint with a custom interceptor for incoming messages, that does logic similar to my DetachEndpoint.

The class implementing the EndpointInterceptor interface, can be declared in your config like this :

<bean id="myEndPointMapping" class="org.springframework.ws.server.endpoint.mapping.PayloadRootQNameEndpointMapping">
    <property name="mappings">
        <props>
        ...
        </props>
    </property>
    <property name="interceptors">
        <list>
           <ref bean="myInterceptor" />
            ....
        </list>
    </property>

<bean id="myInterceptor" ... />

The interceptor has access to the messageContext via the handleRequest(MessageContext messageContext) method.

Code that glows in the dark

I'm currently enjoying the Pragmatic Programmer (T.Hunt), which reads fluently thanks to the fact that it's build out of tips in the form of small chapters. Currently stranded at tip 15, I discovered a principle that I've already applied on a couple of projects, never realizing that it was a principle. Here's a distilled summary out of the chapter, shortened and mixed with an example:

Summarized book quote:

"There are two ways to fire a machine gun in the dark. You can find out what the target coordinates are. Then determine the environment conditions (wind, temperature, humidity, ...). After that investigate the specs of the cartridges and bullets and their interactions with the model of your gun. You can calculate the bearing and elevation of the barrel by a program. If all your tables of data are correct and the environment doesn't change, your bullets should land pretty close to the target.

Or you could use tracer bullets...

A tracer bullet is loaded on the ammo belt at intervals. When fired in the dark, their phosphorus ignites and leave a trail to what they hit, providing the gunner with instant feedback.

When you're starting a new project or building something that hasn't been built before and you're dealing with multiple layers/tiers/rpc's/libraries/ languages,... there are a lot of unknowns. The classic solution is to specify the system to death, clarifying each unknown. One big calculation of everything up front, then shoot and hope. Pragmatic Programmers tend to use tracer bullets."

Example:

For instance, let's say you have multiple clients on different platforms, calling your new webservice that needs to access some info in the DB, used to shoot an external webservice. Before the last step, giving back the result of this external webservice, it has to be transformed to another format and placed on a queue towars another system.

For each client, you could create simple tests firing a soap request asking about the one row present in the DB. A mock webservice deployed on a local or dev server, always returning the same (validated) result, could act as an external service. Maybe some sort of a persistent mean, like a file could act as a queue to see if the mock answer got transfered in the proper format.

Functionally there are few or no requirements implemented, but you've built a framework that allowed you to find out how the application hangs together as a whole (testing the different libs, marshalling, serialization, db access, soap calls, ...). You can show the users the interactions in practice and provide your development team an architectural skeleton on which to hang code.

That's a negative

The talk at this year's Devoxx I enjoyed the most, was the keynote of Uncle Bob aka Robert C. Martin. The whole talk was brilliant but there was one part I particularly enjoyed. As a reaction on his claim that testing is enormously crucial for a software Craftsman, he was confronted with a question from the audience, stating that ensuring code has a large test coverage, also comes with a lot of hastle maintaining those tests. He then replied with the single coolest answer on the topic ever:

"Ok what's the alternative? We'll just dance in circles around our code claiming that it works then shall we?!?"

It's funny because it's true. To me, it seemed that in the way he replied there was a bitter undertone. As if he had been asked this question far too many times and far too few people listened. To get the proper coding standard, testing is a serious business.

For my first contract I was lucky enough to end up in a fully compliant TDD project. The methodology was taken serious, along with the other eXtreme programming principles such as pair programming.

We swiched keyboards every 15 minutes and always started with the writing of a test. In the beginning this really helped to think about the contract of my class. What is it supposed to do? For me, after a couple of years the test 'first' became less useful since thinking on the interface of your class comes natural after some time. As long as you test your class well, you'll refactor it until it's working and readable.

However, we're not paid for working code, we're paid for bug free (*) code. What does it mean to test your class well? What all developers, including myself, constantly have to keep in mind, is to not forget to also negative test.

On a system level negative testing is referred to as tests trying to break the system, for instance massive load tests. I'm referring to negative testing on a unit level. That's seeing how your code (CUT) responds to different behaviour than expected. The non-standard scenarios if you like.

For instance, let's say a method takes in an account number as a parameter. What will the method do if the account number format is invalid. Or if the format is valid but the account can not be found in the persistent store. Or if it's found but no longer active?

Negative testing might sound logically or easy to do, but two things get in the way; first of all, you need to stay disciplined enough (read: don't be too lazy too implement the additional test cases). All...the...time.

Second of all, it's hard to negative test code that you've written yourself. It is inherently human to find the code you've written two minutes ago, easy and clear code. It does what it needs to do right? When you're pair programming, your pair might have an interesting vision on code you wrote, providing you with a little more feedback. This usually results in additional test cases.

But not everyone is pairing. You might be the only developer on the team. What helps for me is the blackbox testing approach: looking at the interfaces you're dealing with, but not at the implementation.

For instance, on an individual class level, think about how heavy your methods depend on their parameters. If a parameter is optional for a method, how does your code react when it's not there? What if it's there but it contains a functionally wrong value? If a parameter is absolutely crucial and it's not there, does this indicate a bug? If it does, you might want to fail early and add an explicit not null check as your first statement.

On a higher level, if your class depends the output of collaborating classes, is the return value of that class always as expected? What happens when it's not?

Of course integration tests cover a lot of the scenarios at a certain point in time. However, you never know when your piece of code will be called in future or different scenarios.

That's why negative testing is important.

I'm interested in hearing about your experience in negative testing, so comments are welcome.

(*) This is an utopia of course, but we should at least strive towards it.

The birth of a blog

Ah yes, another blog is what the world needs... I've been thinking a lot about the possible usefulness of my own blog and I finally decided to go for it (praise the lord).
There is no point in learning interesting things and keeping them to yourself. By doing this, other people can share their opinions on my ideas and see if they match. Enjoy and by all means, feel free to comment.