Sign in to follow this  

[web] upload progressbar, compression, etc

Recommended Posts

In ASP if i was to use the control provided to upload files i find that a call to Page_Load isn't placed until the file is already archived in memory (or temp dir) somewhere on the server, i can then take that stream and dump it into a file somewhere. This works fine for small files, no problem with that but with files over 100MB this doesn't work so well. Basically there is no feedback to the user about the file uploading, or the browser doing anything. Is there a way to somehow intercept the stream earlier and provide a progress bar that shows the upload progress? Would I have to write a custom control for this, extend from the available and overload some members? Is there a scheme that with the progress bar i could also implement compression, send less data to the server preferably and not compress the file when it is already on the server. Using something like a zip lib is fine (or just making my own simple RLE compressor for it), just need access to the data before it is sent to the server. Thanks in advance

Share this post

Link to post
Share on other sites
My main experience is ASP, but I'm transitioning to .NET... I've found a few links that talk about uploading files in ASP.NET and they all seem to follow the same process:

1) Create Form containing a FileInput Control (remember to runat="server")
2) Create an event handler for the form submit which extracts the data from the System.Web.UI.HtmlControls.HtmlInputFile.PostedFile

[Link] - [Link]

I'm assuming that this is the process you're taking as it makes the most sense (and seems common). If this is the case, it does sound as if the entire file is uploaded before the page is classed as loaded - this makes sense as it's how the ASP model works (in my understanding anyway). What you need is a streamed upload - which I don't think ASP.NET provides without a custom ISAPI filter. I'm most likely wrong about this (due to my limited experience with .NET) and I'm hoping that someone will correct me (with links [grin]).

Having a quick look around has come up with a couple of linksm I've not tried them nore understand how they work - but their descriptions seem to do what you want. A blog I found pointed me to a C# tool that runs on the 1.1 framework called Flesk.Uploader. There's also another that looks promising called ABCUpload - but unfortnately it's compiled and requires registration.

I'm sorry I couldn't help more about this; I'm going to keep looking around as I'm going to need something myself when I get to the filemanager component in evoWeb - so please, if you find any information I'd love to see what you come up with.

Share this post

Link to post
Share on other sites
Yes that is the process that i was using earlier but for large files it wont work for several reasons, including asp thinking it has a deadlock and recycling.

You are correct about the ISAPI filter, in ASP thats the route that i would have had to take but in ASP.NET i can write a module instead. Today beside implementing small features and bug fixes was mainly used for research about how to implement this kind of upload module. Here are some of the links that i came across and found helpful...

ISAPI versus ASP file uploads in web applications

HTTP Handlers and HTTP Modules in ASP.NET

How To Create an ASP.NET HTTP Handler by Using Visual C# .NET

Extending the ASP.NET Runtime with Custom HTTP Modules

Introduction to ASP.NET HTTP Module

MSDN Intercepting Filter

MSDN Implementing Intercepting Filter in ASP.NET Using HTTP Module

Forum post about upload module (just started reading as i was leaving)

MultipartForm Uploader

Basically (the most basic implementation) i will have to implement 2 parts, a streamed reader and a parser for the mimi data. I will also have to solve the problem of putting some data back that i will need, like other form fields. It looks like this can be done by just handling the BeginRequest and EndRequest methods and doing a streamed read on the worker request, parsing that data and throwing the correct sections into files and other data back.

I will also somehow need to be able to work in a progress bar (keep stats for each users uploads) and compression algorithm (probably user choosable, also depending on OS). I'm more worried about the progress part atm.

I'll keep this thread updated about my progress but i doubt i'll be able to post code since this is for work.

Thanks for the help

Share this post

Link to post
Share on other sites
Update, i have my upload module up and running now, the base implementation took 1-2 hours, took quite a bit longer to iron out all the bugs that should never happen but were :p.

I chose a staged/tasked path (kinda like the kernel/task management in the Enginuity articles), everything is in its own task, the reader streams the data in (64K chunks when available), passes that data to the parser which is responsible for where the data goes, either back into the stream or to a file, and then passes that to the writer (if its going to file). Doing it this way i can easily hook up a compression or encryption stage before the writer, made debugging pretty easy too (can just remove the parser stage and stream directly to a file).

Since all file uploads are currently going through the filesystem component (to append date/header info) of the application, i wanted to keep it this way, just modify that component. When I'm pulling out the file from the MIME stream in its place I insert the name of the temporary file where i'm streaming it to (with a header), so when this component gets the posted file i simply check the first bytes of the input stream and compare to my header, if they match then copy the file over from temp and delete the temp file, if they dont match then save the stream to file (as before).

The progress bar, i have that running but it will need more work to keep stats for different downloads (haven't decided what i want to use for the id, a guid, userid, etc). The progress bar is started when the submit/next button is hit on the form and refreshes itself every 1-2 seconds, reading info from the upload module (filename, bytes read, bytes total, speed, etc) and updating the labels and the bar (just a panel).

Most of it is working, i am just concerned about one thing. In a few cases instead of the form handling whatever operation it needs with a postback i have it redirect to a different aspx file that will do it. I'm doing this through javascript (form, action, submit) so i get the form information from the first page. If I was to instead show the progress bar here it would keep running until the files are uploaded and then cause a postback on the first page, not redirecting it to page i need. I'm sure theres a solution to that, probably missed something really simple here :p.

I found another link, full uploader with source, came across it too late to make any use of it tho, might help be of help to you or others. SlickUpload

I think I'm going to have to implement a custom download component too now, same thing with uploading, everything is read into memory first. With huge files this will obviously not work with many users downloading at a time, i dont think there is a simple solution to that (atleast not as simple as the upload).

Well thats all for now, if you need help with implementing an upload module you can PM me, it should be pretty well covered tho in the links i posted above (especially the thread).

Share this post

Link to post
Share on other sites
I like the sound of the upload component - it's a shame you're coding a closed source project [wink]. I like the filtered idea a lot, being able to pass the chunks through a compression/encryption/validation filter sounds very useful.

Thanks very much for the links, they should be useful to anyone else with a similar need.

As for the postbacks, you might want to check out an article I came across. It might be useful for creating and managing a smooth progress bar.

Share this post

Link to post
Share on other sites
Firstly thanks for the article, very interesting

Secondly, huge problem at hand. My upload module does not work on pages that have been generated through an XSLT tranform with the System.Web.UI.XML component. It works perfectly on regular pages, pages with a custom form control, and everything else just not XML tranformed pages. The worst part is that there is no error, absolutely no lead as to where the problem is occuring.

What is happening is my module first reads the preloaded part of the request (GetPreloadedEntityBody) and then stream reads the rest of the body as it is coming in (ReadEntityBody). These are performed on the HttpWorkerRequest that i get from the application (param on begin request). With the XSLT tranformed pages i read the preloaded part fine (always 48KB) but I can't read any more data with 'ReadEntityBody', either returns 0 right away (when running on the server) or blocks for a few minutes and then returns 0 when running on my dev machine.

No error is ever generated, no clue as to what can be wrong. The preloaded part isn't corrupt in any way, arrives perfectly, correct data and everything (fields from the generated page).

I've been pulling my hair out over this today, so if anyone has any ideas, doesn't matter how crazy please let me know. I just dont understand why the preloaded part arrives fine but i can't get the rest (when there is a body, 'HasBody' returns true), if the file is ~25k then it arrives in preloaded and i get it fine.

Is there some microsoft direct support forum or e-mail that i can try?

Thanks a lot in advance

Share this post

Link to post
Share on other sites
I've ha da look around and it seems that a few people have been having the exact same problem on ReadEntityBody. A post over on .NET 246 is hinting that the connection is being lost, but I'm not sure that's your problem by the sounds of it. I also found some code for someone else's handler, but he's having a similar problem to you as well.

So I'm wondering what the difference is between your XSLT generated pages and a normal page? Are you able to post any code or output samples from a 'normal' page and the XSLT generated page? By XSLT page I'm assuming that you've generated a page that contains your form on it, correct? Does the form have anything weird going on in it?

As for support, have you looked over the MSDN ASP.NET support pages and Microsoft Newsgroups?

Share this post

Link to post
Share on other sites
Fixed the problem, really, really, really stupid problem that I should have found right away. *kicks self*

If you are using the worker request then never, ever under any circumstances try accessing anything besides QueryString or contentlength from the app Request object. The worker request and the regular page request do not work together, even calling something like 'totalbytes' will screw it up. It will jack the non preloaded part of the stream even if the worker request has it all, you wont be able to access it from either of them if you use them together.

The regular page request was acting as a fallback for me that i didn't expect to be called, well on XSLT pages the upload id wasn't a string on the url but a hidden field, which i was accessing using the Request.Form method. Well it doesn't support streaming (i knew the hidden field would be the first one) and it doesn't (can't) share data with the worker request.

So the solution now, if the upload id isn't in the query string then my parser will find it and set it. Works like a charm.

I feel really stupid about this, i should have known it, i shouldn't have written the code the way it was, i shouldn't have expected it to work and i shouldn't have spent a day finding it. Well hopefully someone will learn from my stupidity.

Thanks anyway, time to look into implementing a smooth progress bar now :)

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this