Looks like your server is running out of memory at the 50MB mark.
Why doesn't it happen when the browser directly downloads the file? WP eStore sits between the file and the browser creating a shield (this is how eStore keeps the file location hidden). However this means that eStore has to read the file and serve it to the browser (meaning the file will be temporarily loaded in your server's RAM). It is common to have less RAM available to you on shared hosting.
Possible Workarounds:
-----------------------------
1) You could break the bigger file into multiple parts (e.g. each 40MB) and use the multi part download option of eStore.
2) Try the different download methods of eStore from the "AddOn Settings" section of the plugin. More explanation here:
http://www.tipsandtricks-hq.com/forum/topic/wp-estore-download-methods
3) You could uncheck the "downloadable" checkbox for the products with bigger download (this will force eStore to let the browser handle the download but it means that the customer will be able to find out the real location of the file).
4) You can use the Amazon S3 integration from the "AddOn Settings" of eStore if you have an S3 account.
I prefer option 1, 2 or 4.