Quantcast

Unable to commit a file of size 4GB to svn repo

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Unable to commit a file of size 4GB to svn repo

sathish Sarvepalli
Hi TortoiseSVN Masters,

Thanks for providing a great tool for windows.

What I did?

I wanted to add a new file into SVN1.5 repository whose size is 4GB. I used TortoiseSVN to add this new file. 

What I wanted to happen?

I expected the file to get committed into the repository and give me a success message.

What actually happened?

I received an error message "413 entity too large".

My environment

Windows XP
Tortoise SVN 1.5.6 32bit

Please let me know if you have any questions.

Thanks,
Sathish



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to commit a file of size 4GB to svn repo

Ben Smith-Mannschott
On Mon, Dec 29, 2008 at 12:30 PM, sathish Sarvepalli <[hidden email]> wrote:
Hi TortoiseSVN Masters,

Thanks for providing a great tool for windows.

What I did?

I wanted to add a new file into SVN1.5 repository whose size is 4GB. I used TortoiseSVN to add this new file. 

What I wanted to happen?

I expected the file to get committed into the repository and give me a success message.

What actually happened?

I received an error message "413 entity too large".

My environment

Windows XP
Tortoise SVN 1.5.6 32bit

Please let me know if you have any questions.

Thanks,
Sathish


* What file system is used to store the repository (FAT32, NTFS, EXT3, ...)? FAT32 has file size limit at 2GB. Barring that, older versions of APR (Apache Portable Runtime), upon which subversion builds have a 32-bit limit on file sizes.
* Are you accessing the repository locally (file:///) or through a server (svn:// , http://, https://)?
* If you try to commit using svn from the command line, does the same error occur?
* Are you sure you really need to be pushing 4 GB(!) files into your repository? Everything committed to the repository is "forever". Even if you later delete it, it will continue to take up space in your repository because it'll continue to be available in older revisions of the repository. That's not a bug. That's version control working as designed, but it is worth being aware of.

--
// Ben Smith-Mannschott
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to commit a file of size 4GB to svn repo

Kurt Pruenner
In reply to this post by sathish Sarvepalli
sathish Sarvepalli wrote:
> I received an error message "413 entity too large".

I think the maximum size of a file is limited by Apache's
LimitXMLRequestBody directive[1] (at least it is for mod_dav), and that
is stored inside Apache as a variable of type size_t, which limits it to
just below 4G.

Maybe setting the limit to 0 (which disables this check) would help?

--
Kurt Bernhard Pruenner --- Haendelstrasse 17 --- 4020 Linz --- Austria
.......It might be written "Mindfuck", but it's spelt "L-A-I-N".......

------------------------------------------------------
http://tortoisesvn.tigris.org/ds/viewMessage.do?dsForumId=4061&dsMessageId=994988

To unsubscribe from this discussion, e-mail: [[hidden email]].
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to commit a file of size 4GB to svn repo

sathish Sarvepalli
In reply to this post by Ben Smith-Mannschott
Hi smith,

Thanks for your reply.

*The SVN server is installed in RHEL 5 (so hopefully it is ext3)

*I'm accessing the server through a server (http://)

*Yes, Im getting the same error from the command line too.

* Yes, I need to push that 4GB file coz I mainly work on files they are too large in size.

Thank you once again for your immediate response. Please let me know if you have any other questions.

Thanks,
Sathish



* What file system is used to store the repository (FAT32, NTFS, EXT3, ...)? FAT32 has file size limit at 2GB. Barring that, older versions of APR (Apache Portable Runtime), upon which subversion builds have a 32-bit limit on file sizes.
* Are you accessing the repository locally (file:///) or through a server (svn:// , http://, https://)?
* If you try to commit using svn from the command line, does the same error occur?
* Are you sure you really need to be pushing 4 GB(!) files into your repository? Everything committed to the repository is "forever". Even if you later delete it, it will continue to take up space in your repository because it'll continue to be available in older revisions of the repository. That's not a bug. That's version control working as designed, but it is worth being aware of.

--
// Ben Smith-Mannschott

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to commit a file of size 4GB to svn repo

sathish Sarvepalli
In reply to this post by Kurt Pruenner
Hi Kurt,

Thank you very much for your reply.

I have already tried setting the LimitXMLRequestBody directive. It didnt work. I heard somewhere that it is something to do with the svn client and not svn server. I also heard it is something to do with *neon* and I have even tried with *serf* but that again didnt work since there is no support for serf in win32. Iam using winxp and Tortoisesvn client for committing the file. I heard from someone that serf is supported unix and linux.

Did anyone ever tried committing this large file ever? Am i missing something here?

Thanks once again for all your efforts.

Thanks,
sathish


On Mon, Dec 29, 2008 at 6:16 PM, Kurt Pruenner <[hidden email]> wrote:
sathish Sarvepalli wrote:
> I received an error message "413 entity too large".

I think the maximum size of a file is limited by Apache's
LimitXMLRequestBody directive[1] (at least it is for mod_dav), and that
is stored inside Apache as a variable of type size_t, which limits it to
just below 4G.

Maybe setting the limit to 0 (which disables this check) would help?

--
Kurt Bernhard Pruenner --- Haendelstrasse 17 --- 4020 Linz --- Austria
.......It might be written "Mindfuck", but it's spelt "L-A-I-N".......

------------------------------------------------------
http://tortoisesvn.tigris.org/ds/viewMessage.do?dsForumId=4061&dsMessageId=994988

To unsubscribe from this discussion, e-mail: [[hidden email]].

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to commit a file of size 4GB to svn repo

Stefan Küng
In reply to this post by sathish Sarvepalli
sathish Sarvepalli wrote:
> Hi smith,
>
> Thanks for your reply.
>
> *The SVN server is installed in RHEL 5 (so hopefully it is ext3)

In that case, you have to make sure that your apr libs are compile with
large file support.

Since this is a server problem (and Linux), I suggest you ask on the
Subversion mailing list for help on this - you'll get much better
answers there.

Stefan

--
       ___
  oo  // \\      "De Chelonian Mobile"
 (_,\/ \_/ \     TortoiseSVN
   \ \_/_\_/>    The coolest Interface to (Sub)Version Control
   /_/   \_\     http://tortoisesvn.net

------------------------------------------------------
http://tortoisesvn.tigris.org/ds/viewMessage.do?dsForumId=4061&dsMessageId=995022

To unsubscribe from this discussion, e-mail: [[hidden email]].

signature.asc (202 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to commit a file of size 4GB to svn repo

sathish Sarvepalli
Stefan,

Thanks for your immediate response.

I was told that it is the client that doesnt support the large file (something to do with neon) not with server since subversion1.5 supports unlimited file size for commit. Am i missing something here please advice.

Thanks,
Sathish.

On Mon, Dec 29, 2008 at 7:04 PM, Stefan Kueng <[hidden email]> wrote:
sathish Sarvepalli wrote:
> Hi smith,
>
> Thanks for your reply.
>
> *The SVN server is installed in RHEL 5 (so hopefully it is ext3)

In that case, you have to make sure that your apr libs are compile with
large file support.

Since this is a server problem (and Linux), I suggest you ask on the
Subversion mailing list for help on this - you'll get much better
answers there.

Stefan

--
      ___
 oo  // \\      "De Chelonian Mobile"
 (_,\/ \_/ \     TortoiseSVN
  \ \_/_\_/>    The coolest Interface to (Sub)Version Control
  /_/   \_\     http://tortoisesvn.net

------------------------------------------------------
http://tortoisesvn.tigris.org/ds/viewMessage.do?dsForumId=4061&dsMessageId=995022

To unsubscribe from this discussion, e-mail: [[hidden email]].

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to commit a file of size 4GB to svn repo

Stefan Küng
sathish Sarvepalli wrote:
> Stefan,
>
> Thanks for your immediate response.
>
> I was told that it is the client that doesnt support the large file
> (something to do with neon) not with server since subversion1.5 supports
> unlimited file size for commit. Am i missing something here please advice.

I don't think that neon is the problem. But you can switch to serf easily:
* Settings dialog->Network. Edit the servers file (click edit button)
* add the line 'http-library = serf'
* save the file

Now TSVN (and all svn clients) will use serf instead of neon.
Serf itself uses the apr library, which on Windows is always compiled
with large file support.

But I still think it's a server problem since 4GB is the maximum size of
a file if large-file support is not compiled into the apr lib. And if
you're using an FSFS repository, then one revision equals one file - and
you're adding a 4GB file in one revision...

Stefan

--
       ___
  oo  // \\      "De Chelonian Mobile"
 (_,\/ \_/ \     TortoiseSVN
   \ \_/_\_/>    The coolest Interface to (Sub)Version Control
   /_/   \_\     http://tortoisesvn.net

------------------------------------------------------
http://tortoisesvn.tigris.org/ds/viewMessage.do?dsForumId=4061&dsMessageId=995035

To unsubscribe from this discussion, e-mail: [[hidden email]].

signature.asc (202 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to commit a file of size 4GB to svn repo

Kevin Radke
Stefan Kueng wrote:

> sathish Sarvepalli wrote:
>> Stefan,
>>
>> Thanks for your immediate response.
>>
>> I was told that it is the client that doesnt support the large file
>> (something to do with neon) not with server since subversion1.5 supports
>> unlimited file size for commit. Am i missing something here please advice.
>
> I don't think that neon is the problem. But you can switch to serf easily:
> * Settings dialog->Network. Edit the servers file (click edit button)
> * add the line 'http-library = serf'
> * save the file
>
> Now TSVN (and all svn clients) will use serf instead of neon.
> Serf itself uses the apr library, which on Windows is always compiled
> with large file support.
>
> But I still think it's a server problem since 4GB is the maximum size of
> a file if large-file support is not compiled into the apr lib. And if
> you're using an FSFS repository, then one revision equals one file - and
> you're adding a 4GB file in one revision...

This is indeed a svn client library problem.  It can easily be seen
by capturing the http client request with a network protocol analyzer.
The request clearly shows a negative size being sent by the client in
the http header.  I reported it to the svn list awhile ago, but no
resolution was found at the time.  (I started debugging it, but quickly
got lost in some unfamiliar svn code.)

We have some 20GB+ sized transactions done over http.  The limitation is
the compressed delta size of one individual file in the transaction.
(At the time I didn't test serf, but if both exhibit the problem, then
it is a problem with the common code that computes the file size to be
placed in the http header.)

FYI, I tested 1.5 versions of TortoiseSVN, command line from windows,
command line from Linux (both 32 and 64 bit redhat) and command line
from Solaris, and all exhibited the same problem to servers running on
Linux, Solaris, and Windows.

I did observe this DOES work correctly using the svn:// protocol, if
that is an option...

Kevin R.

------------------------------------------------------
http://tortoisesvn.tigris.org/ds/viewMessage.do?dsForumId=4061&dsMessageId=995436

To unsubscribe from this discussion, e-mail: [[hidden email]].
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to commit a file of size 4GB to svn repo

sathish Sarvepalli
Kevin, Stefan,

Thanks for your help.

I did try with serf but got the same "Request Entity too large" error. When I read the log the content length seems to hold a negative value as kevin mentioned. It is *Content-Length: -371974493*.

Kevin,

/////I did observe this DOES work correctly using the svn:// protocol, if
that is an option.../////

How do i set the svn protocol with TSVN?

Thank you all once again for your help, time and efforts.

Thanks,
Sathish

On Tue, Dec 30, 2008 at 7:57 AM, Kevin Radke <[hidden email]> wrote:
Stefan Kueng wrote:
> sathish Sarvepalli wrote:
>> Stefan,
>>
>> Thanks for your immediate response.
>>
>> I was told that it is the client that doesnt support the large file
>> (something to do with neon) not with server since subversion1.5 supports
>> unlimited file size for commit. Am i missing something here please advice.
>
> I don't think that neon is the problem. But you can switch to serf easily:
> * Settings dialog->Network. Edit the servers file (click edit button)
> * add the line 'http-library = serf'
> * save the file
>
> Now TSVN (and all svn clients) will use serf instead of neon.
> Serf itself uses the apr library, which on Windows is always compiled
> with large file support.
>
> But I still think it's a server problem since 4GB is the maximum size of
> a file if large-file support is not compiled into the apr lib. And if
> you're using an FSFS repository, then one revision equals one file - and
> you're adding a 4GB file in one revision...

This is indeed a svn client library problem.  It can easily be seen
by capturing the http client request with a network protocol analyzer.
The request clearly shows a negative size being sent by the client in
the http header.  I reported it to the svn list awhile ago, but no
resolution was found at the time.  (I started debugging it, but quickly
got lost in some unfamiliar svn code.)

We have some 20GB+ sized transactions done over http.  The limitation is
the compressed delta size of one individual file in the transaction.
(At the time I didn't test serf, but if both exhibit the problem, then
it is a problem with the common code that computes the file size to be
placed in the http header.)

FYI, I tested 1.5 versions of TortoiseSVN, command line from windows,
command line from Linux (both 32 and 64 bit redhat) and command line
from Solaris, and all exhibited the same problem to servers running on
Linux, Solaris, and Windows.

I did observe this DOES work correctly using the svn:// protocol, if
that is an option...

Kevin R.

------------------------------------------------------
http://tortoisesvn.tigris.org/ds/viewMessage.do?dsForumId=4061&dsMessageId=995436

To unsubscribe from this discussion, e-mail: [[hidden email]].

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to commit a file of size 4GB to svn repo

Ben Smith-Mannschott


On Tue, Dec 30, 2008 at 6:03 AM, sathish Sarvepalli <[hidden email]> wrote:
/////I did observe this DOES work correctly using the svn:// protocol, if
that is an option.../////

How do i set the svn protocol with TSVN?

TSVN supports this protocol out of the box, the real issue is that your server needs to serve the repo using svnserve.  It is possible to serve the same repo with both apache (http) and svnserve, but I think thst would mean maintaining your user list twice.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to commit a file of size 4GB to svn repo

sathish Sarvepalli

Smith,

Thanks for your help.

Is there any fix for this issue? I want to upload files more than 4GB in size.

Thanks to everyone once again for their time, efforts and help.

Thanks,
Sathish


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to commit a file of size 4GB to svn repo

Kurt Pruenner
sathish Sarvepalli wrote:
> Is there any fix for this issue? I want to upload files more than 4GB in
> size.

Then, as it was already stated earlier, you'll have to use svnserve[1]
instead of Apache for the time being...


[1] http://svnbook.red-bean.com/en/1.5/svn.serverconfig.svnserve.html

--
Kurt Bernhard Pruenner --- Haendelstrasse 17 --- 4020 Linz --- Austria
.......It might be written "Mindfuck", but it's spelt "L-A-I-N".......

------------------------------------------------------
http://tortoisesvn.tigris.org/ds/viewMessage.do?dsForumId=4061&dsMessageId=995788

To unsubscribe from this discussion, e-mail: [[hidden email]].
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to commit a file of size 4GB to svn repo

Konstantin Kolinko
In reply to this post by sathish Sarvepalli
2008/12/30 sathish Sarvepalli <[hidden email]>:
>
> Is there any fix for this issue? I want to upload files more than 4GB in
> size.
>

See recent thread entitled "Re: svn commit: r34915 -
trunk/subversion/libsvn_ra_neon"
at dev@ list at subversion.tigris.org

Citing from there:

(1)
> Log:
> Support commit of a new file of size(post-compression) >2G.
>
> Neon has a support of large file requests for quite sometime, Subversion should
> make use of this support if available.

(2)
> Neon doesn't implement its LFS support on Windows :(

It looks like this change is committed to trunk and thus will be in
1.6, but not for
Windows users.

Thus using svn:// protocol instead of HTTP seems the only workaround here, if
switching to serf does not solve the issue as your experience shows.

(also, serf is still beta, though it is expected to replace neon in some future
version of svn, may be 1.7)

Best regards,
Konstantin Kolinko

------------------------------------------------------
http://tortoisesvn.tigris.org/ds/viewMessage.do?dsForumId=4061&dsMessageId=996890

To unsubscribe from this discussion, e-mail: [[hidden email]].
Loading...