This article describes how to implement WCF File Transfer methods hosted in IIS. This includes two techniques Streaming and Chunking Channel.
We needed a scalable file upload facility and after much research found the two best methods Streaming and Chunking Channel for reasoning see the references below. After spending many days and night scouring forums and samples I finally managed to get Streaming and Chunking Channel file transfer services hosted in IIS.
Streaming:
I used the Streamed Upload from Pablo M. Cibraro article Sending Attachments with WCF. I then added two new projects to the solution w3service and w3client.
w3service is a WCF service library project that I then add a simple echo function and a upload function the implements a [OperationContract(IsOneWay=true)] and a [MessageContract()] that allows for a message header and body.
I then added the service.svc file that points to the w3Service.Service methods and a web.config file with my basicHttpBinding. You will need to change the
Now that we have the WCF service hosted in IIS you can navigate to it
WE then use the w3client console application to consume the w3service. Run “svcutil.exe http://localhost/ServiceName/service.svc?wsdl /mc” in the visual studio command prompt and then include the output.config content in the app.config file and import the serive.cs file into the application. Check the app.config file and make sure that the transferMode is set to StreamedRequest and that your endpoint matches. Also make sure the maxReceivedMessageSize is increased.
Make sure that you use the /mc in the svcutil.exe tool, otherwise it will not generate the proxy as you would expect.
That should be that. Run the w3client application and test you file transfer. Hopefully all works out for you.
Chunking Channel:
For this I used the Chunking sample from LarsW.ChunkingChannelEx-0.9. I first customized it to use the same [MessageContract()] functionality that the Streaming sample uses. After that I followed the same steps as above in setting up the w3service and w3client projects with a few distinct differences. With the major differences being in the .config files.
For the w3service config you will need to use bindingExtensions to configure the httpChunkingBinding. You will also need to change the
We now need to consume the new service by using the “svcutil.exe http://localhost/ServiceName/service.svc?wsdl /mc” tool and copy the generated proxy to your w3client application. Make sure that you use the /mc in the svcutil.exe tool, otherwise it will not generate the proxy as you would expect. For Chunking we will also have to make sure that the contract is in the client application. This is done because of the ChannelFactory implementation and configuration.
Conclusion:
After initial testing you can use Fiddler to examine the difference in the way the two samples work. You can also configure both solutions to use SSL.
Code:
Like a picture paints a thousand word so code can make things seem o so clear. Included in my samples are the modified solutions that are mentioned above that I derived my prototypes from.
Download Streamed Sample
Download Chunking Channel Sample
Chunking Channel https
Streamed https
References:
http://msdn.microsoft.com/en-us/library/aa717050.aspx
http://code.msdn.microsoft.com/WCFResources/Release/ProjectReleases.aspx?ReleaseId=1534
http://code.msdn.microsoft.com/WCFResources/Release/ProjectReleases.aspx?ReleaseId=1546
http://kjellsj.blogspot.com/2007/02/wcf-streaming-upload-files-over-http.html
http://weblogs.asp.net/cibrax/archive/2007/08/29/sending-attachments-with-wcf.aspx
Is that possible to use same binding for sending thousands of objects to client ? Iam using an IIS hosted WCF service which needs to send thousands of (atleast 20,000) records back to client.
ReplyDeleteThanks,
Hello. I tried to run w3Service on IIS, but i received this error:
ReplyDeleteThe type 'w3Service.Service', provided as the Service attribute value in the ServiceHost directive could not be found.
Do you know, what is wrong? I run on MS Vista, IIS 7.
Regards,
Jan
Hi Jan,
ReplyDeleteSorry I have only tested it on XP and Server 2003. Have you made the wwwRoot/WCFStreamUpload an application folder in IIS. I would also encourage you to just keep trying, it is tricky to configure.
Google seach on the error: http://www.google.co.za/search?hl=en&q=The+type+%27%27%2C+provided+as+the+Service+attribute+value+in+the+ServiceHost+directive+could+not+be+found&btnG=Search&meta=
Hi Mark.
ReplyDeleteI solved it ;-). I created new WCF service application in existing solution, then copy all stuff releated to chunking channel into web.config file and it's working now. But i have another poblem - timeout exception. I need to send big file (up to 2 GB through HTTP) and i receive timeout. I tried to increase timeout value, but i think this is not solution. I want to remove timeouts, but thay are used on many places. Have you some suggestion/idea how can i solve it right way?
Regards,
Jan
Hi Jan,
ReplyDeleteAre you using the streaming or the chunking? On streaming there is the receiveTimeout="00:10:00" sendTimeout="00:01:00" properties and in the chunking we have the maxRequestLength="2097151" executionTimeout="100" properties that need to be set. The maxRequestLength above is set to just under 2GB.
../mark
Hi,
ReplyDeleteAs per my analysis about the Composite Duplex Binding, HTTPS with CD is not a supported configuration, as Server needs to call back to the client. But as per your comments in your post, you said we can configure this HTTP chunking service to host on IIS over HTTPs, but I could not see any example or direction given with the article or code. Can you please give me some direction about how to configure this to work with HTTPS.
Hi Anonymous,
ReplyDeleteI did these samples some time ago and ran into many compications along the way. I am about 90% sure that I did https tests on both methods and validated the test though Fiddler. I did make a backup of my https implimentations. I hope that this has the anwsers you need.
Some points to look into are mexHttpsBinding, httpsGetEnabled, wsDualHttpBinding.
I'm not going to lie, it will be tricky, but remember if its not had to implement its probably not that secure.
ChunkingChannelEx_https
Hi Mark
ReplyDeleteI've been working on incorporating your sample into my project, but I'm getting some strange errors occurring:
"A null chunk message was received. Null message indicates the end of a session so a null chunk should never be received"
This is the CommunicationException thrown on line 127 of ChunkingDuplexSessionChannel.cs. This problem appears to be intermittent however, like it's thread sync related. Interestingly, I have no problem if I don't close the service proxy after the service call (at least until I reach some sort 10 connection limit with the server).
Any ideas what may be causing this?
Hi Andrew,
ReplyDeleteI’m not sure what the issue could be. This was purely a proof of concept, so I have not used it in a live environment. If you find out please share it.
Thanks,
Mark
What version of IIS are you using for this to work ? I use IIS7 and it dont work.
ReplyDeleteI got this working on IIS6. I have not had a chance to upgrade it to IIS7, If I do i will post my learning here.
ReplyDelete../mark
Ok i tested your project on IIS7 with 2 configuration.
ReplyDelete1- Client and service running on windows server 2008. It work good.
2- Client running on windows xp and service on windows server 2008. It dont work... when i call echo function it run timeout.
Do you know why ?
Not sure but why don't you try and use Fiddler to try and see what is happening with the call.
ReplyDeleteMy only other suggestion would be to look at Proxy's and Firewalls?
../mark
Thank for your answers i got it working on both configuration now.
ReplyDeleteEric
Hi Mark,
ReplyDeleteCan you please send me the https implementations of this sample? I have tried the download that you mentioned in one of the comments above but it doesn't seem to work.
Nick
Hi Nick,
ReplyDeleteI recently got a new laptop so I had to dig around for the https projects and configs.
ChunkingChannelEx_https
StreamedUpload_https
Like I said in the above post security is complicated so stick with it I’m sure you will crack it.
I hope this helps.
../mark
Hello,
ReplyDeleteI am using the chunking channel with basichttp binding. I have set the transfer mode to buffering. When I transferred files of order 27-30 Mb the service crashed. But when I set the Maxrequestlength to the maximum value(2097151) it worked fine. My question is since the chunking channel is going to divide the file into chunks and then send it across, why should I need to worry about setting the maxrequest length to maximum value, since the file is not going to be transferred all at once.
On a different note, I did not set the chunksize to any value. If so what is the default chunk size? Also how would you enable the logging for chunking channel i.e how many chunks were transferred etc?
Hi Sharanya,
ReplyDeleteI am not 100% sure about the inner workings of the httpRuntime Maxrequestlength property works, but my best guess is that it is for an individual request and even though we are using chunking each upload is part of one request therefore subject to the limitations of the httpRunTime properties.
About the default chuck size, it sends it in a byte array of byte[4096] E.G. for a file of 35KB it will be split into about 10 parts, 35KB x 1024 = 35840 bytes then divide by 4096 = 8.75 and the reason it breaks it into 10 parts is because of the xml sent with it.
There is no logging, but you can customise the code.
../mark
Hi Mark,
ReplyDeleteI would like to use a chunking channel with MemoryStream. In this case some not implemented methods in ChunkingWriter and ChunkingReader are called (for example WriteStartAttribut, WriteString, WriteEndAttribut). When I look for a caller of this not-implemented methods there is only "External call" to see in stack trace. Could you please answer some questions:
1. What is the difference in the MemoryStream handling in comparison with FileStream? Could you describe a workflow in ChunkingWriter/ChunkingReader?
2. What do I need to implement these not-implemented methods that will be called by handling of MemoryStream?
Thank
Alisa.
Hi Mark,
ReplyDeleteFirst of all thanks for the excellent post and code samples. While i was configuring my sample app which uses the TcpChunkingBinding, i noticed that the maxBufferedChunks setting i specified in the .config wasn't picked up.
After some debugging i think i found the cause of this issue.
the property MaxBufferedChunks in the ChunkingBindingBase class is declared als following:
public int MaxBufferedChunks { get; set; }
However in the child classes (for example TcpChunkingBinding) the same property is declared.
Now when this property is set in the
OnApplyConfiguration method of the ChunkingConfigurationElementBase class the property of the ChunkingBindingBase is set, but this has no effect for all the child classes since they hide the original property.
declaring the MaxBufferedChunks property as virtual in the ChunkingBindingBase class and override it in the child binding classes solved my problem.
Could you confirm this issue?
Best Regards
Rob
Hi Alisa,
ReplyDeleteSorry for not replying sooner. I’m not going to be able to answer your questions but I will be able to point you in the right direction.
1. It depends on how you plan to use it. If you are going to read a file from a database then it is better to use a memory stream, they should probably be smallish as well, otherwise they will drain the server resources (http://msdn.microsoft.com/en-us/library/system.io.memorystream.asp ). If you wanted to read the file from you disk then use the FileStream (http://msdn.microsoft.com/en-us/library/system.io.filestream.aspx ).
2. You should ask Lars at http://code.msdn.microsoft.com/WCFResources/Release/ProjectReleases.aspx?ReleaseId=1546 he is the person responsible for Chunking sample, I just customised it to work with http and https.
Hopefully you get the answers you need.
../mark
Hi Rob,
ReplyDeleteThanks for your feedback, I have not explored the Chunking sample in that much depth, I just go it working in http/s for my proof of concept. Check out Lars site for an update. http://code.msdn.microsoft.com/WCFResources/Release/ProjectReleases.aspx?ReleaseId=1546
Kind Regards,
Mark
I am having the exact same problem as a previous post
ReplyDelete2- Client running on windows xp and service on windows server 2008. It dont work... when i call echo function it run timeout.
Does anyone know what was done to fix the problem. having a bit of trouble figuring it out.
Thanks
Glenn
Hello mark,
ReplyDeleteHow can you use basichttpBinding with chunking channel?
I want to transfer large files through the WCF service. But I do not want to use the httpchunking binding that comes with the chunking channel because it is duplex and I do not want my client to expose any ports.
So in my config file, I have defined a custom binding configuration under basichttpbinding where i have set the timeouut, MaxArraylength etc to large values so that it can allow large files transfer.
In my client , I use the proxy like this
ChannelFactory factory = new ChannelFactory(ConfigSection);
ICMSExchangeFileTransfer service = factory.CreateChannel(new EndpointAddress(WebServiceURL))
where Configsection points to the basichhtpbinding configuration.
Files upto 150mb transfer fine. But files beyond that gives me an error "Unable to make HTTP request to URL".
So If i do not use the bindings that comes with the chunking channel, is it a problem? I ahve set the Maxrequestlength executiontimeout (under HTTPRunTime) to max values, but still it reaches the limit for 150Mb files.
does this sample buffer at the server? it seems alot of the samples found online seem to still buffer at the server, meaning its not sending the chunk and the writing the chunk at the server. it seems like it is streaming the chunks only to be buffered at the server and then written all at once.
ReplyDeleteHi Anonymous,
ReplyDeleteYes it does buffer but not the entire file that is the main purpose of chunking (that it doesn’t kill your servers resources) If you create a large file to upload you can go to the upload directory and see that that file gets larger as the upload progresses.
Regards,
Mark
I'm trying your streaming project with a 100 mb file and debugging with fiddler. It seems that fiddler is showing the entire 100 mb being sent to the iis server at once. so if I set break points on both endpoint the UploadFile breakpoint will not be hit on the server until the winform client has finished reading the file locally and then sending the entire file all at once. Any ideas?
ReplyDeleteHi,
ReplyDeleteWe are able to transfer files upto 760 mb from WCF service to client (both are in 2 different server) , but when we try to transfer file beyond that, it download upto only 761 mb & get stop further.
Is there are some thing which we need to set on IIS 7 server (both the servers) or in Registry setting.
Hi Anonymous,
ReplyDeleteI am assuming that you are using the channel chunking? Not sure what the problem is, I have I have only tried it up to about 500mb. My only suggestion would be for you to try take the service out of IIS and make it self-hosting and then give it try.
Regards,
Mark
Hello!!
ReplyDeleteExcellent sample. How could I control bandwidth using TcpChunkingBinding?? What parameters should I modify??
I found a solution for bandwidth throttling using this sample:
ReplyDeletehttp://www.codeproject.com/KB/IP/Bandwidth_throttling.aspx
Some tips to implement resuming uploads with chunking?? I think one can read the incomplete file downloaded and knows the number of chunks, and then, one should skip all copied chunks and begin to read the stream from the first chunk after that. My question is how to read the stream from a known chunk??
ReplyDeleteHi, I'm trying to use Chunking channel for communication between a WCF service and gSoap(on linux) on the other end. Could you please provide any inputs on this method and whether it would be possible to do this without having required Chunking channel reader classes on the receiving end which is on Linux?
ReplyDelete