Support compressed responses in web services

---

Querying a web service can result to huge responses, e.g. the response of a SOS GetObservation request. To reduce the response size HTTP offers the possibility to compress the response content.

Clients that support compressed responses can indicate this by adding the Accept-Encoding parameter with value gzip to the HTTP header of the request. If the response is compressed, the compression is indicated by the parameter Content-Encoding with value gzip in the HTTP header of the response.

Server

Using servlet container

Servlet containers often have out of the box compression support, please check the documentation.

• Tomcat: https://tomcat.apache.org/tomcat-7.0-doc/config/http.html (search for "compression").

Using libraries or filters

For a general introduction on using a Filter for compression see http://www.onjava.com/pub/a/onjava/2003/11/19/filters.html.

• ehcache GZip Filter: https://www.waltercedric.com/index.php?option=com_content&view=article&id=2092&catid=129:maven-build-system&Itemid=332
• PJL compression filter: http://sourceforge.net/projects/pjl-comp-filter/
• http://stackoverflow.com/questions/4755302/which-compression-is-gzip-the-most-popular-servlet-filter-would-you-suggest

Leveraging HTTP server

If your web service is behind an HTTP server such as nginx or Apache, it might be a good solution to leave the compression to the webserver instead of putting the load on the application or Servlet container.

• nginx gzip module: http://nginx.org/en/docs/http/ngx_http_gzip_module.html
• Apache mode deflate: http://httpd.apache.org/docs/2.2/mod/mod_deflate.html#enable

Standalone implementation in a Java web application

Here is an example for the response content compression support. It checks if the indicator for response compression is set in the HTTP header. If the client supports compression and the response size is greater than 1000000 the service compresses the response and adds the compression indicator to the HTTP header of the response.

Example of a HttpServlet class with doPost method that contains the response compression funtionality

public class SosService extends HttpServlet {
    protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException {

        // process the request
        ServiceResponse serviceResponse = processTheRequest(request);

        OutputStream out = null;
        GZIPOutputStream gzip = null;
        try {
            String contentType = serviceResponse.getContentType();
            response.setContentType(contentType);
            out = response.getOutputStream();
            int contentLength = serviceResponse.getContentLength();

            // compressed response
            // Inversed boolean order check. greaterThan is cheaper
            if (contentLength > 1000000 && checkForClientGZipSupport(request)) {
                response.addHeader("Content-Encoding", "gzip");
                gzip = new GZIPOutputStream(out);
                gzip.write(serviceResponse.getBytes());
                gzip.flush();
                gzip.finish();
            } 
            // uncompressed response
            else {
                response.setContentLength(contentLength);
                out.write(serviceResponse.getBytes());
                out.flush();
            }
        } catch (IOException ioe) {
            String exceptionText = "Error while writing response to ServletResponse!";
            LOGGER.error(exceptionText, ioe);
            throw new ServletException(exceptionText, ioe);
        } finally {
            try {
                if (gzip != null) {
                    gzip.close();
                }
                if (out != null) {
                    out.close();
                }
            } catch (IOException ioe) {
                LOGGER.debug("Error while closing output stream(s)!", ioe);
            }
        }
    }

    // method to check if the HTTP header contains 'Accept-Encoding' parameter with value 'gzip'
    private boolean checkForClientGZipSupport(HttpServletRequest httpServletRequest) {
        String header = httpServletRequest.getHeader("Accept-Encoding");
        if (header != null && !header.isEmpty()) {
            String[] split = header.split(",");
            for (String string : split) {
                if (string.equalsIgnoreCase("gzip")) {
                    return true;
                }
            }
        }
        return false;
    }
}
}

Client

Example Client Implementation (Apache HttpClient 4.x)

import java.io.IOException;
import java.util.Properties;

import org.apache.http.Header;
import org.apache.http.HeaderElement;
import org.apache.http.HttpEntity;
import org.apache.http.HttpException;
import org.apache.http.HttpRequest;
import org.apache.http.HttpRequestInterceptor;
import org.apache.http.HttpResponse;
import org.apache.http.HttpResponseInterceptor;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.entity.GzipDecompressingEntity;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.params.BasicHttpParams;
import org.apache.http.params.CoreConnectionPNames;
import org.apache.http.protocol.HttpContext;

public class HttpClientWithGZIP {

    private static final String GZIP_ENCODING_VALUE = "gzip";
    private static final String ENCODING_HEADER_NAME = "Accept-Encoding";
    private static final String CONNECTION_TIMEOUT_KEY = "CONNECTION_TIMEOUT";
    private static final String USE_GZIP_KEY = "USE_GZIP";

    private DefaultHttpClient httpClient;
    private Properties configuration;

    public HttpClientWithGZIP() {
        //dummy configuration
        this.configuration = new Properties();
        this.configuration.setProperty(USE_GZIP_KEY, "true");
        this.configuration.setProperty(CONNECTION_TIMEOUT_KEY, "5000");

        this.httpClient = new DefaultHttpClient();
        setup();
    }

    public HttpResponse executeRequest(HttpUriRequest request) throws ClientProtocolException, IOException {
        return this.httpClient.execute(request);
    }

    private void setup() {
        if (Boolean.parseBoolean(configuration.getProperty(USE_GZIP_KEY))) {
            addGzipInterceptors(this.httpClient);
        }

        if (this.httpClient.getParams() == null) {
            this.httpClient.setParams(new BasicHttpParams());
        }

        int timeout = Integer.parseInt(configuration.getProperty(CONNECTION_TIMEOUT_KEY));
        this.httpClient.getParams().setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,
                timeout);
        this.httpClient.getParams().setIntParameter(CoreConnectionPNames.SO_TIMEOUT,
                timeout);
    }

    private void addGzipInterceptors(DefaultHttpClient httpclient) {
        httpclient.addRequestInterceptor(new HttpRequestInterceptor() {

            @Override
            public void process(HttpRequest request, HttpContext context)
            throws HttpException, IOException {
                if (!request.containsHeader(ENCODING_HEADER_NAME)) {
                    request.addHeader(ENCODING_HEADER_NAME, GZIP_ENCODING_VALUE);
                }                
            }
        });

        httpclient.addResponseInterceptor(new HttpResponseInterceptor() {
            @Override
            public void process(HttpResponse response, HttpContext context)
            throws HttpException, IOException {
                HttpEntity entity = response.getEntity();
                if (entity != null) {
                    Header header = entity.getContentEncoding();
                    if (header != null) {
                        HeaderElement[] codecs = header.getElements();
                        for (int i = 0; i < codecs.length; i++) {
                            if (codecs[i].getName().equalsIgnoreCase(GZIP_ENCODING_VALUE)) {
                                response.setEntity(new GzipDecompressingEntity(response.getEntity()));
                                return;
                            }
                        }
                    }
                }                
            }
        });
    }

}

Using HttpClient from the OXFramework

The OXF offers easy HttpClient configuration so one is able to decorate configuration as needed. See how it works:

package org.n52.ows.request;

import org.apache.http.HttpEntity;
import org.apache.xmlbeans.XmlObject;
import org.apache.xmlbeans.XmlOptions;
import org.junit.Test;
import org.n52.oxf.util.web.GzipEnabledHttpClient;
import org.n52.oxf.util.web.HttpClient;
import org.n52.oxf.util.web.ProxyAwareHttpClient;
import org.n52.oxf.util.web.SimpleHttpClient;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class HttpClientDecoratorTest {
    
    private static final Logger LOGGER = LoggerFactory.getLogger(HttpClientDecoratorTest.class);
    
    @Test
    public void testGetCapabilities() throws Exception {
        HttpClient proxyAwareClient = new ProxyAwareHttpClient(new SimpleHttpClient());
        HttpClient httpclient = new GzipEnabledHttpClient(proxyAwareClient);

        String baseUri = "http://sensorweb.demo.52north.org/PegelOnlineSOSv2.1/sos";
        GetCapabilitiesParameters parameters = new GetCapabilitiesParameters("SOS", "1.0.0");
        parameters.addAcceptedVersion("2.0.0");
        HttpEntity responseEntity = httpclient.executeGet(baseUri, parameters);
        XmlObject capabilities = XmlObject.Factory.parse(responseEntity.getContent());
        
        LOGGER.debug(capabilities.xmlText(new XmlOptions().setSavePrettyPrint()));
    }
}