Deploying Static and Dynamic Content With Maven Wagon Plugin

One common performance requirement for large websites is that different domain names should be used to serve static and dynamic content. The reason behind this requirement is that usually browsers do not download more than two components in parallel per domain name, because of the HTTP 1.1 specification. Thus, separating static and dynamic content under different hostnames is used to maximize the number of parallel downloads. More information about the performance increase obtained by the usage of this technique is given at Yahoo! User Interface Blog, which contains an entry titled Performance Research, Part 4: Maximizing Parallel Downloads in the Carpoon Line.

Recently I started wondering, how this requirement can be implemented by using Maven. The requirements for the Maven based solution are following:

  1. Static content of the Maven project must be automatically copied to the server used to server static content.
  2. It must be possible to copy CSS files and layout images to different remote directories.
  3. Created war file must be automatically copied to the server used to serve dynamic content.
  4. Files must be copied by using FTP protocol.

When I did some research, I learned that Maven Wagon plugin can be used to fulfill the given requirements.

Required Steps

The steps required to fulfill the given requirements are following:

  • Enabling the FTP support of Maven Wagon Plugin
  • Configuring the used credentials
  • Configuring the Maven Wagon Plugin

Each of these steps are described with more details in following.

Enabling the FTP Support of Maven Wagon Plugin

However, since it does not support FTP protocol out of the box, I first tried to enable FTP support by following the instructions given in the usage page of the plugin's homepage. I soon noticed that this approach does not work, because the extension adding FTP support to Maven Wagon was not compatible with the commons-net-1.4.1.jar. So, instead of following the instructions given in the plugin's homepage, copy following jar files to your $MAVEN_HOME/lib directory:

  • wagon-ftp-beta-6.jar
  • oro-2.0.8.jar
  • commons-net-2.0.jar

Configuring the Used Credentials

Before investigating the configuration added to the pom.xml file, we must configure the the credentials of the used servers to the settings.xml file. I recommend configuring the credentials by using user specific settings file (The default location of this file is ~/.m2/settings.xml). Following example contains own credentials for CSS files, layout images and the war file (Remember to replace the usernames and passwords with correct ones):

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <servers>
    <server>
      <id>css</id>
      <username>cssUser</username>
      <password>cssPassword</password>
      <!-- If you want deploy to active FTP server, uncomment the following block -->
      <!-- 
       <configuration>
         <passiveMode>false</passiveMode>
       </configuration>
       -->
    </server>
    <server>
      <id>images</id>
      <username>imagesUser</username>
      <password>imagesPassword</password>
      <!-- If you want deploy to active FTP server, uncomment the following block -->
      <!-- 
       <configuration>
         <passiveMode>false</passiveMode>
       </configuration>
       -->
    </server>
    <server>
      <id>war</id>
      <username>warUser</username>
      <password>warPassword</password>
      <!-- If you want deploy to active FTP server, uncomment the following block -->
      <!-- 
       <configuration>
         <passiveMode>false</passiveMode>
       </configuration>
       -->
    </server>
  </servers>
</settings>

Configuring the Maven Wagon Plugin

Now that we have fulfilled all the prerequisites for the usage of FTP via the Maven Wagon plugin, we can take a look of the pom file, which makes the magic happen. Since pom.xml is quite large, I will explain it in a smaller pieces, which should make it easier to understand. At first we will concentrate on the plugin declaration of the Maven Wagon plugin:

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>wagon-maven-plugin</artifactId>
  <version>1.0-beta-3</version>
  <!--
    Contains unique execution declarations for CSS files,
    layout images and war file.
   -->
  <executions>
    <!--
      Declaration for copying CSS files via FTP
     -->
    <execution>
      <id>upload-css</id>
      <!--
        Describes the lifecycle phase when the plugin's
        goals are being executed.
       -->
      <phase>install</phase>
      <!--
        Describes the goals are executed in the specified
        lifecycle phase.
       -->
      <goals>
        <goal>upload</goal>
      </goals>
      <configuration>
        <!--
          Describes the local directory, which contents are 
          copied.
         -->
        <fromDir>${css.local.dir}</fromDir>
        <!--
          Describes the included files.
         -->
        <includes>*</includes>
        <!--
          Describes the url of the remote FTP server.
         -->
        <url>${css.remote.url}</url>
        <!--
          Describes the remote directory, where the contents 
          of the local directory are copied.
         -->
        <toDir>${css.remote.dir}</toDir>
        <!--
          Describes the server id, which is used to fetch the
          used credentials from settings.xml.
         -->
        <serverId>${css.remote.server.id}</serverId>
      </configuration>
    </execution>
    <!--
      Declaration for copying images via FTP. See the declaration
      of CSS files for more information about the configuration.
     -->
    <execution>
      <id>upload-images</id>
      <phase>install</phase>
      <goals>
        <goal>upload</goal>
      </goals>
      <configuration>
        <fromDir>${images.local.dir}</fromDir>
        <includes>*</includes>
        <url>${images.remote.url}</url>
        <toDir>${images.remote.dir}</toDir>
        <serverId>${images.remote.server.id}</serverId>
      </configuration>
    </execution>
    <!--
      Declaration for copying war file via FTP. See the 
      declaration of CSS for more information about the
      configuration.
     -->
    <execution>
      <id>upload-war</id>
      <phase>install</phase>
      <goals>
        <goal>upload</goal>
      </goals>
      <configuration>
        <fromDir>${war.local.dir}</fromDir>
        <includes>*</includes>
        <url>${war.remote.url}</url>
        <toDir>${war.remote.dir}</toDir>
        <serverId>${war.remote.server.id}</serverId>
      </configuration>
    </execution>
  <executions>
</plugin>

Cool. We have now configured the Maven Wagon plugin to copy CSS files, images and war file to remote FTP server when install lifecycle phase is executed. Obviously we now have to also configure the Maven War plugin to exclude CSS files and images from the war file. This can be done by using following plugin declaration:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-war-plugin</artifactId>
  <version>2.0.2</version>
  <configuration>
    <!--
      Excludes the contents of the styles and image  directories when copying 
      files to the directory used to create war file. Note that the directories
      are hard coded and you might have to change the configuration
      depending on your project structure.
     -->
    <warSourceExcludes>
      styles/*, images/*
    </warSourceExcludes>
  </configuration>
</plugin>

The last interesting part of the pom file contains the property declarations used in the configuration of Maven Wagon plugin. These declarations are shown in following:

<properties>
  <!-- Settings for CSS files -->
  <css.local.dir>${basedir}/src/main/webapp/styles</css.local.dir>
  <!-- Insert the remote target directory path for CSS files -->
  <css.remote.dir></css.remote.dir>
  <css.remote.server.id>css</css.remote.server.id>
  <!-- Insert the url address of the FTP server for CSS files -->
  <css.remote.url></css.remote.url>

  <!-- Settings for image files -->
  <images.local.dir>${basedir}/src/main/webapp/images</images.local.dir>
  <!-- Insert the remote target directory path for images -->
  <images.remote.dir></images.remote.dir>
  <images.remote.server.id>images</images.remote.server.id>
  <!-- Insert the url address of the FTP server for image files -->
  <images.remote.url></images.remote.url>

  <!-- Settings for War files-->
  <war.local.dir>${basedir}/target</war.local.dir>
  <!-- Insert the remote target directory path for war file -->
  <war.remote.dir></war.remote.dir>
  <war.remote.server.id>war</war.remote.server.id>
  <!-- Insert the url address of FTP server for war file -->
  <war.remote.url></war.remote.url>
</properties>

The deployment process can be triggered by executing command: mvn install in the project's root directory (The directory, which contains the pom.xml file). The outcome is following:

  1. CSS files are copied from the configured local directory to the specified remote directory.
  2. Images are copied from the configured local directory to the specified remote directory.
  3. War file is constructed, and copied from the configured local directory to the specified remote directory.

Final Words

It should now be clear, how the Maven Wagon plugin can be used to copy static and dynamic files to different hosts. However, the current configuration is far from perfect. To be honest, it is not ready for production usage. Different Maven profiles for development, testing and production usage should be added, before I would recommend the usage of this configuration in serious development. The master plan behind this is following:

At this point the only way to run the example project from working war file (styles and images are included in the war file), is to comment out the configuration section of the Maven War plugin declaration. The lack of profiles is also the reason explaining, why the upload goal of the Maven Wagon plugin is triggered by install lifecycle phase. When using different profiles, Maven could be configured in such manner that no files would be copied to remote servers and excluded from the war file, when the development profile is enabled. Also, separating test and production profiles would make it possible to copy files to different remote servers depending on the enabled profile. Of course this would also allow us to determine the url addresses, which are needed to access static content, based on the used profile.

In any case, this improvement is left as an exercise for the reader.

Disclaimer: The example project has been tested with JDK 1.6.0_20, Maven 2.2.0 and OS X 10.6.4. Browse the source code of the example project.

13 comments… add one
  • Jari Timonen Jun 30, 2010 @ 18:02

    Instead of ~/m2/settings.xml you can use "mvn -s settings.xml" so specify own location. This is handy with CI server deploy tasks.

  • Petri Jul 1, 2010 @ 22:57

    Jari, I totally forgoted the existence of that command line option. Nice tip.

  • Antti K Jul 3, 2010 @ 9:55

    Hi,

    Very good! Very good indeed.
    I just thought about this issue regarding our project this week. It's nice that you work during your holiday! :)
    And that -s command line option is a nice thing to know.

  • Petri Jul 3, 2010 @ 12:18

    Antti, since I no dot have got any time to play around with interesting challenges during my time at work, I decided to do it on own time... :)

  • Henri Mar 10, 2011 @ 11:12

    Hi,

    Thank for the article !

    You can add recursive copy with just modify */**. It can be useful if you have a lot of subdirectories.

    • Petri Mar 13, 2011 @ 1:43

      Henri,

      it is nice to hear that this article was useful to you. Also, thanks for your tip concerning the recursive copy function.

  • Petri Dec 18, 2011 @ 16:55

    Moved the example project to GitHub.

  • Domingo Apr 4, 2012 @ 16:34

    As the wagon-ftp set up so PassiveMode=false....
    Thanks

    • Petri Apr 10, 2012 @ 9:37

      Hi Domingo,

      thanks for your comment. I will update the blog entry asap.

  • Don Mendo Sep 24, 2014 @ 16:10

    Hi Petri,
    Thanks a lot for the explanation.
    Just a small consideration:
    It's not the first time I read about HTTP 1.1 standard limiting the number of parallel downloads.
    However, the way I understand HTTP spec, it only says that you SHOULD not have more than two idle keep-alive connections against the same server. That is, it says nothing (IMHO) about the number of concurrent connections, as long as they are active. It only says you shouldn't keep alive more than two of them per site, when not in use.
    Am I wrong? May be browsers are being too conservative?
    Cheers!

    • Petri Sep 24, 2014 @ 16:42

      Hi,

      Thank you for asking this interesting question.

      I actually picked that up from this Yuiblog article which says that:

      The biggest impact on end-user response times is the number of components in the page. Each component requires an extra HTTP request, perhaps not when the cache is full, but definitely when the cache is empty. Knowing that the browser performs HTTP requests in parallel, you may ask why the number of HTTP requests affects response time. Can’t the browser download them all at once?

      The explanation goes back to the HTTP/1.1 spec, which suggests that browsers download two components in parallel per hostname.

      At the time I didn't read the HTTP/1.1 spec but now that you mentioned it, I took a quick look at chapter 8 which says that:

      Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. A proxy SHOULD use up to 2*N connections to another server or proxy, where N is the number of simultaneously active users. These guidelines are intended to improve HTTP response times and avoid congestion.

      In other words, that limit is just a recommendation. Then I did a quick Google search and found this StackOverflow question. I check the value of the network.http.max-connections-per-server config setting (I use Firefox 32.0.2) and noticed that the default setting is 6.

      So I guess the result is that the HTTP 1.1 specification gives a recommendation, and the browsers can either ignore it or follow it.

      I have to admit that I haven't read the whole specification, but if the idea behind this limitation is to improve the response times, this limit probably means active connections. Right?

  • Anonymous Mar 20, 2015 @ 17:37

    Recursive copy does not work for wagon-ftp. Endless loop:
    Reply received: 250 Directory successfully changed.
    Command sent: CWD .
    ..
    Reply received: 250 Directory successfully changed.
    Command sent: CWD .

    • Petri Mar 22, 2015 @ 22:15

      I haven't used the Maven Wagon plugin in four years, and I cannot remember if I had some problems with recursive copy. Have you tried updating the Maven Wagon plugin to the latest version? Also, you might want to create a bug report.

Leave a Reply