Quantcast
Channel: okhttp 3: how to decompress gzip/deflate response manually using Java/Android - Stack Overflow
Viewing all articles
Browse latest Browse all 5

okhttp 3: how to decompress gzip/deflate response manually using Java/Android

$
0
0

I know that okhttp3 library by default it adds the header Accept-Encoding: gzip and decodes the response automatically for us.

The problem I'm dealing with a host that only accepts a header like: Accept-Encoding: gzip, deflate if I don't add the deflate part it fails. Now when I manually add that header to okhttp client, the library doesn't do the decompression anymore for me.

I've tried multiple solutions to take the response and try to manually decompress that but I've always ended up with an exception i.e. java.util.zip.ZipException: Not in GZIP format, here's what I've tried so far:

//decompresser
public static String decompressGZIP(InputStream inputStream) throws IOException
{
    InputStream bodyStream = new GZIPInputStream(inputStream);
    ByteArrayOutputStream outStream = new ByteArrayOutputStream();
    byte[] buffer = new byte[4096];
    int length;
    while ((length = bodyStream.read(buffer)) > 0) 
    {
        outStream.write(buffer, 0, length);
    }

    return new String(outStream.toByteArray());
}


//run scraper
scrape(api, new Callback()
{
    // Something went wrong
    @Override
    public void onFailure(@NonNull Call call, @NonNull IOException e)
    {
    }

    @Override
    public void onResponse(@NonNull Call call, @NonNull Response response) throws IOException
    {
        if (response.isSuccessful())
        {
            try
            {
                InputStream responseBodyBytes = responseBody.byteStream();
                returnedObject = GZIPCompression.decompress(responseBodyBytes);

                if (returnedObject != null)
                {
                    String htmlResponse = returnedObject.toString();
                }
            }
            catch (ProtocolException e){}

            if(response != null) response.close();
        }
    }
});



private Call scrape(Map<?, ?> api, Callback callback)
{
    MediaType JSON = MediaType.parse("application/json; charset=utf-8");
    String method = (String) api.get("method");
    String url = (String) api.get("url");
    Request.Builder requestBuilder = new Request.Builder().url(url);
    RequestBody requestBody;

    requestBuilder.header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0");
    requestBuilder.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
    requestBuilder.header("Accept-Language", "en-US,en;q=0.5");
    requestBuilder.header("Accept-Encoding", "gzip, deflate");
    requestBuilder.header("Connection", "keep-alive");
    requestBuilder.header("Upgrade-Insecure-Requests", "1");
    requestBuilder.header("Cache-Control", "max-age=0");

    Request request = requestBuilder.build();

    Call call = client.newCall(request);
    call.enqueue(callback);

    return call;
}

Just a note, the response headers will always return Content-Encoding: gzip and Transfer-Encoding: chunked

One more thing, I've also tried the solution in this topic and it still fails with D/OkHttp: java.io.IOException: ID1ID2: actual 0x00003c68 != expected 0x00001f8b.

Any help would be appreciated..


Viewing all articles
Browse latest Browse all 5

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>