A couple significant issues with mass downloads
Apologies for the long post but this is important and a bit complicated. There are a couple issues here. I assume that this applies to uploads as well but I haven't reall… (read more)
Apologies for the long post but this is important and a bit complicated.
There are a couple issues here. I assume that this applies to uploads as well but I haven't really investigated that. To be fair, probably all browsers suffer from similar problems, but the fix has to start somewhere, and Firefox seems ideal due to its prioritization of UX over advertising etc.
First of all, the serious one: empirically, Firefox just times out and gives up on downloads after a certain period of idleness from the peer, as though I somehow no longer want the file simply because the server is busy. This makes mass downloads impossible in some cases. The browser should never stop retrying until I manually cancel. But I get it: you don't want to hammer the server with endless requests that it can't handle. So what to do?
Exponential backoff is a bad idea because the probability distribution function of server downtime is more accurately modeled as a decaying power law, not a decaying exponential. Therefore quadratic backoff between retries might make more sense: 1, 4, 9, 16, 25, etc. time units between retries. But this is also a bad idea because then retries arrive more or less simultaneously across all N active downloads, which might originate from the same server. So random quadratic backoff would be preferable: a per-file random time delay between 1 and 4 time units, 4 and 9, 9 and 16, etc.
A more sophisticated approach would be to calibrate the decay exponent through training, but the benefit in terms of efficiency would be incremental. I'll leave that to you.
The bottom line is that it's never appropriate for the browser to supplant its own judgment for the user's command. It should never give up on a download even if the network disappears for a while, except maybe in catastrophic circumstances, wherein for example the server says that the file is no longer available.
Users sometimes need to launch a large set of downloads and leave them over the weekend to complete. This should be trivial and routine but it's currently a babysitting job.
Secondly, the UI issue: remaining download times are poorly modeled, even from an algorithmic perspective without any consideration for AI training over time. In the most typical case, a user is downloading from a datacenter with vastly more bandwidth to spare than the user's own connection, notwithstanding that the file is probably coming from a single server with much less bandwidth than the datacenter as a whole. This means that the peer is faster than the user in most cases. Therefore, typically, the download proceeds subject to the user's own bandwidth constraint. Consequently, the user's total bandwidth is divided among all N active downloads.
But the estimated time remaining only incorporates the current average rate of each download. It doesn't model the fact that, once a download finishes, the freed bandwidth will soon be redistributed among the remaining (N-1) downloads, and then (N-2), etc. This can make an enormous difference in the estimated time remaining, to the point that the existing estimates are uninformative. It's not too complicated to model this algorithmically and display much more accurate estimates.
There's also the current download rate as displayed, e.g. 3 MB/s, which is also uninformative because it's often subject to spikes. You're better off displaying the average download rate over the entire history for the file in question, than the rate over the last however-many seconds. In other words, look at the average download rate over the entire download process up to now, not some arbitrarily windowed subset thereof.
A much more informative number is the number of seconds for which the server has been unresponsive. There's no need to display it unless the server has been idle for at least a second. But after that happens, give a warning indicating that you haven't heard from the server for the past however-many seconds. If all N downloads are in this warning mode, then you might post another warning indicating that the network seems to be disconnected from the internet even though the local router might still be reachable. I'm sure you can improve upon this but idle period is useful information.
Please note that there are 2 major complicating factors which should be taken into a account when estimating remaining download time: (1) the N-way stepwise redistributed bandwidth model described above and (2) the empirical fact that, if we look at aggregate download behavior on large scales, it tends to decay (albeit softly) according to a power law throughout the course of the download, as a result of increasing connection management entropy (packet retries wasting time, intervening "smart" routers adding latency instead of helping anything, operating systems distracted with other tasks, cache misses, etc.). (Over very long periods of time, file transfers speed up exponentially due to Moore's Law applied to bandwidth, but this isn't the timescale of most downloads.) I'm sure you can think of yet more sophisticated ways of modeling expected download time, even including the media flushing and garbage collection latency that's the cause of the ubiquitous pause-at-99-percent phenomenon. The point is simply that there's a lot of low-hanging fruit to be harvested with respect to more accurate time estimates.
Everyone does downloads. This issue is worth your time and attention. Thank you for reading this.