mitmproxy 0.13 - Response Streaming

# Response Streaming

By using mitmproxy's streaming feature, response contents can be passed to the client incrementally before they have been fully received by the proxy. This is especially useful for large binary files such as videos, where buffering the whole file slows down the client's browser.

By default, mitmproxy will read the entire response, perform any indicated manipulations on it and then send the (possibly modified) response to the client. In some cases this is undesirable and you may wish to "stream" the reponse back to the client. When streaming is enabled, the response is not buffered on the proxy but directly sent back to the client instead.

## On the command-line

Streaming can be enabled on the command line for all response bodies exceeding a certain size. The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes.

command-line | --stream SIZE

## Caveats

When response streaming is enabled, streamed response contents will not be recorded or preserved in any way.

When response streaming is enabled, the response body cannot be modified.

## Customizing Response Streaming

You can also use an inline script to customize exactly which responses are streamed.

Responses that should be tagged for streaming by setting their respective .stream attribute to True:

    def responseheaders(context, flow):
        """
        Enables streaming for all responses.
        """
        flow.response.stream = True

(examples/stream.py)

## Implementation Details

When response streaming is enabled, portions of the code which would have otherwise performed changes on the response body will see an empty response body instead (`libmproxy.protocol.http.CONTENT_MISSING`). Any modifications will be ignored.

Streamed responses are usually sent in chunks of 4096 bytes. If the response is sent with a `Transfer-Encoding: chunked` header, the response will be streamed one chunk at a time.

## Modifying streamed data

If the `.stream` attribute is callable, .stream will work as a hook in chunk data processing.

    """
    This inline script modifies a streamed response.
    If you do not need streaming, see the modify_response_body example.
    Be aware that content replacement isn't trivial:
        - If the transfer encoding isn't chunked, you cannot simply change the content length.
        - If you want to replace all occurences of "foobar", make sure to catch the cases
          where one chunk ends with [...]foo" and the next starts with "bar[...].
    """

    def modify(chunks):
        """
        chunks is a generator that can be used to iterate over all chunks.
        Each chunk is a (prefix, content, suffix) tuple.
        For example, in the case of chunked transfer encoding: ("3\r\n","foo","\r\n")
        """
        for prefix, content, suffix in chunks:
            yield prefix, content.replace("foo", "bar"), suffix

    def responseheaders(context, flow):
        flow.response.stream = modify

(examples/stream_modify.py)

### See Also

- Ignore Domains