On downloading stuff

In the many years of using computers and different operating systems, I can say, without a doubt, that any user-facing program that is meant to download file(s) from the internet sucks. And there are all sorts of such programs:

• Programs that download files you want (wget, curl, IDM and the web browsers we use)
• Download operating system updates (Microsoft’s Windows Update, macOS system updates, linux package managers) or software updates (chrome updater, firefox update service, linux package managers again), or even
• Programs that download and install from the internet (looking at you, Google Chrome Installer).

The Problem

In the early days of downloading stuff, programs like Download Accelerator Plus and IDM were all the rage. Over time, browsers made improvements to their built in download mechanisms and the remaining gaps were filled with extensions like DownThemAll! and many more. But the one thing that has consistently sucked less and has continuously improved to this day are the command-line download tools: the good-old cURL (which is shipped in all of the recent versions of Windows 10, macOS and many popular Linux distros) and wget. And the reason they suck less, is because they provide a way to input the state of the downloaded file, either by reading its state directly from the file (as in case of curl, wget, where no state is stored at all, and it is the user’s responsibility to provide the correct URL) or providing the state from another source (aria2c’s .aria2 files come to mind here). Many times, I’ve observed Google Chrome abruptly terminate my download just because of some slight trouble with the network connection (according to the download status ERR:NETWORK_CONNECTION_CHANGED). Sometimes, it just shits the bed by dropping the speed to 0 bytes/sec, something, which seems to have been going on for years. And since this seems to be a Chromium related thing, that means any browsers that are Chromium-based, such as Opera, Microsoft Edge, Brave etc. have possibly inherited this flaw as well. The Firefox and Safari sides don’t look good either, and this too seems to have been going on for many years.

Can’t the browsers just fix downloading?

This story has 2 actors: the client trying to download a file, and the server serving the file. Measures can be taken up on both sides to ensure that file delivery is smooth. The server side of course, is governed by business considerations most of the time, so reliable file delivery might not always be a high priority, which leaves the client side.

Coming to the question at hand, the answer is that browsers can try to be be more smart (they’re already doing that) and allow customizing download parameters. The problem is that downloading files is a complex affair, and general use tools like Web Browsers generally want to abstract away these complexities from their users. Mostly these problems occur due to the fact that the URL of the file to be downloaded changes, based on various parameters: Sometimes, websites generate a URL based on specific parameters like: the current time (and timezone), the web-browser name/version, the IP address of the computer, the Operating System name/version and any other detail which they might find useful for purposes like analytics, DDoS mitigation and so on. Sometimes, they even check cookies to see if the download was indeed triggered the way they wanted it. Since downloads are always identified by URLs in the web browsers, it implies that web browsers are certainly not suitable for such cases. And such cases, though not common, are increasing. Though at times browsers try to be smart by doing under-the-hood stuff like parsing redirects, but this might not necessarily save you.

So, as a user, what can you do?

Remember the thing I said about browsers being smart? Well, for scenarios when a browser’s smartness falls short, you, the mighty user have to compensate by being smart yourself. Here’s a thumb rule:

• If your download size is a few MBs: browsers are good.
• If not, best to go for command-line tools like cURL and wget, unless command-line tools have definitively failed you in that case, meaning that you should exhaust full knowledge of that tool first. In such cases, go back to browsers, or contact the website operators, if you can.
• Learn more about these tools and the web. If you are reading this post, you already are.

PS: If you are someone who has worked on any of the web browsers, maybe explore the possibility of adding an advanced options menu to a download. And ask the user before removing incomplete terminated downloads. They might still be useful after all.

Adios!