Lighttpd on raspberrypi refuses to stream mp4 files

brontosaurusrex · 2020-10-28 20:04:37

So the little machine

    .',;:cc;,'.    .,;::c:,,.    pi@raspberrypi
   ,ooolcloooo:  'oooooccloo:    OS: Raspbian 10 buster
   .looooc;;:ol  :oc;;:ooooo'    Kernel: armv7l Linux 4.19.66-v7+
     ;oooooo:      ,ooooooc.     Uptime: 6d 8h 46m
       .,:;'.       .;:;'.       Packages: 1529
       .... ..'''''. ....        Shell: 325
     .''.   ..'''''.  ..''.      CPU: ARMv7 rev 4 (v7l) @ 4x 1.4GHz
     ..  .....    .....  ..      GPU: BCM2708
    .  .'''''''  .''''''.  .     RAM: 71MiB / 926MiB
  .'' .''''''''  .'''''''. ''.
  '''  '''''''    .''''''  '''
  .'    ........... ...    .'.
    ....    ''''''''.   .''.
    '''''.  ''''''''. .'''''
     '''''.  .'''''. .'''''.
      ..''.     .    .''..
            .'''''''
             ......

running a little lighttpd server with pretty much default settings and dirlist enabled

cat /etc/lighttpd/lighttpd.conf | grep mod 
server.modules = (
        "mod_indexfile",
        "mod_access",
        "mod_alias",
        "mod_redirect",
#server.compat-module-load   = "disable"
server.modules += (
        "mod_compress",
        "mod_dirlisting",
        "mod_staticfile",
server.modules += ("mod_auth", "mod_authn_file")

does play/stream mp4 files when testing inside home network, but refuses to stream to outside users.

Outside Chrome browser will show a broken player window (and that's it) while firefox will preload entire mp4 and then play it.

I'am not using https and the NAT in the router is defined as ext port 80 to internal machine ip port 80 with NAT loopback enabled.

So all in all server is working as expected, minus mp4 streaming. Any clues on how to even start debugging this?

Last edited by brontosaurusrex (2020-10-30 10:13:18)

twoion · 2020-10-28 21:53:36

brontosaurusrex wrote:

So the little machine
    .',;:cc;,'.    .,;::c:,,.    pi@raspberrypi
   ,ooolcloooo:  'oooooccloo:    OS: Raspbian 10 buster
   .looooc;;:ol  :oc;;:ooooo'    Kernel: armv7l Linux 4.19.66-v7+
     ;oooooo:      ,ooooooc.     Uptime: 6d 8h 46m
       .,:;'.       .;:;'.       Packages: 1529
       .... ..'''''. ....        Shell: 325
     .''.   ..'''''.  ..''.      CPU: ARMv7 rev 4 (v7l) @ 4x 1.4GHz
     ..  .....    .....  ..      GPU: BCM2708
    .  .'''''''  .''''''.  .     RAM: 71MiB / 926MiB
  .'' .''''''''  .'''''''. ''.
  '''  '''''''    .''''''  '''
  .'    ........... ...    .'.
    ....    ''''''''.   .''.
    '''''.  ''''''''. .'''''
     '''''.  .'''''. .'''''.
      ..''.     .    .''..
            .'''''''
             ......
running a little lighttpd server with pretty much default settings and dirlist enabled
cat /etc/lighttpd/lighttpd.conf | grep mod 
server.modules = (
        "mod_indexfile",
        "mod_access",
        "mod_alias",
        "mod_redirect",
#server.compat-module-load   = "disable"
server.modules += (
        "mod_compress",
        "mod_dirlisting",
        "mod_staticfile",
server.modules += ("mod_auth", "mod_authn_file")
does play/stream mp4 files when testing inside home network, but refuses to stream to outside users.
Outside Chrome browser will show a broken player window (and that's it) while firefox will preload entire mp4 and then play it.
I'am not using https and the NAT in the router is defined as ext port 80 to internal machine ip port 80 with NAT loopback enabled.
So all in all server is working as expected, minus mp4 streaming. Any clues on how to even start debugging this?

Collect samples from both scenarios using the web browser console F12 in both browsers, then full request log as HAR, then requests from the browsers and response from the server including headers, so 2x2 samples in total. If you can get lighttpd to log full requests that would be fine too, but the developer tools in FF/GC are very easy to use nowadays. Headers like Range: might be interesting. Compare the samples for differences.

Have you tested always with the same file? mp4s can be built so that they can't be played back without having downloaded the entire file, as required metadata might be only at the end of the file. Such files can be converted to streamable, metadata-at-the-head format using tools like qtfaststart and probably ffmpeg too.

ohnonot · 2020-10-29 07:25:38

brontosaurusrex wrote:

does play/stream mp4 files when testing inside home network, but refuses to stream to outside users.

So this is not really limited to mp4 files, but to everything you serve?

brontosaurusrex wrote:

Outside Chrome browser will show a broken player window (and that's it) while firefox will preload entire mp4 and then play it.

Could be a DNS issue? AFAIK this Google Alphabet product uses its own inbuilt DNS server.

brontosaurusrex · 2020-10-29 08:49:07

@twoion, I do not fully understand instructions, but this is internal - working version of header in Chrome

Chrome f12/Network/Headers (Internal network, this is working as expected)

Request URL: __redacted__
Request Method: GET
Status Code: 206 Partial Content
Remote Address: __redacted__:80
Referrer Policy: strict-origin-when-cross-origin
Accept-Ranges: bytes
Content-Length: 34448266
Content-Range: bytes 25034752-59483017/59483018
Content-Type: video/mp4
Date: Thu, 29 Oct 2020 08:44:32 GMT
ETag: "1013408712"
Last-Modified: Fri, 23 Oct 2020 12:54:23 GMT
Server: lighttpd/1.4.53
Accept: */*
Accept-Encoding: identity;q=1, *;q=0
Accept-Language: en-US,en;q=0.9,sl;q=0.8
Authorization: Basic amF6OnNlbQ==
Cache-Control: no-cache
Connection: keep-alive
DNT: 1
Host: __redacted__
Pragma: no-cache
Range: bytes=25034752-
Referer: __redacted__
User-Agent: __redacted__ Chrome/86.0.4240.111 Safari/537.36

Will add the non-working version when on location.

Last edited by brontosaurusrex (2020-10-29 10:07:25)

brontosaurusrex · 2020-10-29 08:55:35

twoion wrote:

Such files can be converted to streamable, metadata-at-the-head format using tools like qtfaststart and probably ffmpeg too.

My ffmpeg cli always includes '-movstart +faststart', but I'am guessing this can't be a problem in this case, since 'streaming' is working fine internally.

edit: File patched using qt-faststart seems to have identical response.

Last edited by brontosaurusrex (2020-10-29 10:07:34)

brontosaurusrex · 2020-10-29 09:06:20

ohnonot wrote:

a DNS issue? AFAIK this Google Alphabet product uses its own inbuilt DNS server.

Using mysubdomain.duckdns.org service, how to check? (using direct ip? test another subdomain service like afraid.org?)

Last edited by brontosaurusrex (2020-10-29 09:20:01)

brontosaurusrex · 2020-10-30 08:53:37

Now observing from a remote location:

Before qt-faststart

Request URL: __redacted__.mp4
Request Method: GET
Status Code: 200 OK
Remote Address: __redacted__
Referrer Policy: strict-origin-when-cross-origin
Accept-Ranges: none
Content-Length: 59483018
Content-Type: video/mp4
Date: Fri, 30 Oct 2020 10:01:51 GMT
ETag: "1013408712"
Last-Modified: Fri, 23 Oct 2020 12:54:23 GMT
Server: lighttpd/1.4.53
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9,sl;q=0.8
Authorization: Basic amF6OnNlbQ==
Cache-Control: max-age=0
Connection: keep-alive
DNT: 1
Host: __redacted__
Referer: __redacted__
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36

After, qt-faststart indeed seem to make a difference, 'streaming' at least 'progressive download' now kinda works (headers not what I expected)

edit: It actually shows two files, 1st

Request URL: __redacted___qtfaststart.mp4
Request Method: GET
Status Code: 416 Requested Range Not Satisfiable
Remote Address: __redacted__:80
Referrer Policy: strict-origin-when-cross-origin
HTTP/1.1 416 Requested Range Not Satisfiable
Content-Length: 0
Connection: close
Accept-Ranges: none
Accept: */*
Accept-Encoding: identity;q=1, *;q=0
Accept-Language: en-US,en;q=0.9,sl;q=0.8
Authorization: Basic amF6OnNlbQ==
Connection: keep-alive
DNT: 1
Host: __redacted__
Range: bytes=11567104-
Referer: __redacted__
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36

and 2nd, this one kinda streams

Request URL: __redacted__qtfaststart.mp4
Request Method: GET
Status Code: 200 OK
Remote Address: __redacted__
Referrer Policy: strict-origin-when-cross-origin
Accept-Ranges: none
Content-Length: 30703449
Content-Type: video/mp4
Date: Fri, 30 Oct 2020 08:50:26 GMT
ETag: "1761460008"
Last-Modified: Thu, 29 Oct 2020 09:39:29 GMT
Server: lighttpd/1.4.53
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9,sl;q=0.8
Authorization: Basic amF6OnNlbQ==
Connection: keep-alive
DNT: 1
Host: __redacted__
Referer: __redacted__
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36

That is remote chrome header ^ for a qt_faststart patched file.

a. I'am guessing happy response should include something like

Accept-Ranges: bytes
Content-Range: bytes 25034752-59483017/59483018

?
b. Is it possible that company firewall somehow mangles those headers?

Last edited by brontosaurusrex (2020-10-30 12:52:14)

#1 2020-10-28 20:04:37

Lighttpd on raspberrypi refuses to stream mp4 files

#2 2020-10-28 21:53:36

Re: Lighttpd on raspberrypi refuses to stream mp4 files

#3 2020-10-29 07:25:38

Re: Lighttpd on raspberrypi refuses to stream mp4 files

#4 2020-10-29 08:49:07

Re: Lighttpd on raspberrypi refuses to stream mp4 files

#5 2020-10-29 08:55:35

Re: Lighttpd on raspberrypi refuses to stream mp4 files

#6 2020-10-29 09:06:20

Re: Lighttpd on raspberrypi refuses to stream mp4 files

#7 2020-10-30 08:53:37

Re: Lighttpd on raspberrypi refuses to stream mp4 files

Board footer