summaryrefslogtreecommitdiff
path: root/lib/libfetch/http.c
Commit message (Collapse)AuthorAgeFilesLines
* Document fetchReqHTTP().Mark Johnston2019-08-281-0/+3
| | | | | | | | | | Submitted by: Farhan Khan <khanzf@gmail.com> Reviewed by: 0mp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D18788 Notes: svn path=/head/; revision=351573
* When deciding whether to send the complete URL or just the document part,Dag-Erling Smørgrav2018-11-271-1/+1
| | | | | | | | | | | | we were looking at the original URL rather than the one we were currently processing. This meant that if we were trying to retrieve an HTTP URL but were redirected to an HTTPS URL, and HTTPS proxying was enabled, we would send an invalid request and most likely get garbage back. MFC after: 3 days Notes: svn path=/head/; revision=341072
* A few more cases where strcasecmp() is no longer required.Dag-Erling Smørgrav2018-11-271-4/+4
| | | | | | | MFC after: 1 week Notes: svn path=/head/; revision=341014
* Fix a few (but far from all) style issues.Dag-Erling Smørgrav2018-05-291-24/+29
| | | | | | | MFC after: 3 weeks Notes: svn path=/head/; revision=334319
* Use __VA_ARGS__ to simplify the DEBUG macro.Dag-Erling Smørgrav2018-05-291-27/+23
| | | | | | | MFC after: 3 weeks Notes: svn path=/head/; revision=334317
* Preserve if-modified-since timestamps across redirects.Dag-Erling Smørgrav2018-05-121-0/+1
| | | | | | | | PR: 224426 MFC after: 1 week Notes: svn path=/head/; revision=333571
* lib: further adoption of SPDX licensing ID tags.Pedro F. Giffuni2017-11-261-0/+2
| | | | | | | | | | | | | | | Mainly focus on files that use BSD 2-Clause license, however the tool I was using mis-identified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Notes: svn path=/head/; revision=326219
* r308996 broke IP literals by assuming that a colon could only occur asDag-Erling Smørgrav2017-03-171-13/+4
| | | | | | | | | | | | | | | | | | a separator between host and port, and using strchr() to search for it. Rewrite fetch_resolve() so it handles bracketed literals correctly, and remove similar code elsewhere to avoid passing unbracketed literals to fetch_resolve(). Remove #ifdef INET6 so we still parse IP literals correctly even if we do not have the ability to connect to them. While there, fix an off-by-one error which caused HTTP 400 errors to be misinterpreted as redirects. PR: 217723 MFC after: 1 week Reported by: bapt, bz, cem, ngie Notes: svn path=/head/; revision=315455
* Fix partial requests (used by fetch -r) when the requested file isDag-Erling Smørgrav2017-03-051-1/+1
| | | | | | | | | | | | | | | | | | | | | | | already complete. Since 416 is an error code, any Content-Range header in the response would refer to the error message, not the requested document, so relying on the value of size when we know we got a 416 is wrong. Instead, just verify that offset == 0 and assume that we've reached the end of the document (if offset > 0, we did not request a range, and the server is screwing with us). Note that we cannot distinguish between reaching the end and going past it, but that is a flaw in the protocol, not in the code, so we just have to assume that the caller knows what it's doing. A smart caller would request an offset slightly before what it believes is the end and compare the result to what is already in the file. PR: 212065 Reported by: mandree MFC after: 3 weeks Notes: svn path=/head/; revision=314701
* Fix inverted loop condition which broke multi-line responses to CONNECT.Dag-Erling Smørgrav2016-12-301-1/+1
| | | | | | | | | PR: 194483 Submitted by: Miłosz Kaniewski <milosz.kaniewski@gmail.com> MFC after: 1 week Notes: svn path=/head/; revision=310823
* r169386 (PR 112515) was incomplete: it treated 307 as an error exceptDag-Erling Smørgrav2016-05-311-0/+3
| | | | | | | | | | | | | | | | in verbose mode, and did not handle 308 at all. r241840 (PR 172451) added support for 308, but with the same bug. Correctly handle both by recognizing them as redirects in all places where we check the HTTP result code. PR: 112515 173451 209546 Submitted by: novel@ MFC after: 1 week Notes: svn path=/head/; revision=301027
* Use strlcpy() instead of strncpy() to copy the string returned byDon Lewis2016-05-121-1/+1
| | | | | | | | | | | | | | | setlocale() so that static analyzers know that the string is NUL terminated. This was causing a false positive in Coverity even though the longest string returned by setlocale() is ENCODING_LEN (31) and we are copying into a 64 byte buffer. This change is also a bit of an optimization since we don't need the strncpy() feature of padding the rest of the destination buffer with NUL characters. Reported by: Coverity CID: 974654 Notes: svn path=/head/; revision=299520
* Fix double-free error: r289419 moved all error handling in http_connect()Dag-Erling Smørgrav2016-02-111-1/+0
| | | | | | | | | | | | to the end of the function, but did not remove a fetch_close() call which was made redundant by the one in the shared error-handling code. PR: 206774 Submitted by: Christian Heckendorf <heckendorfc@gmail.com> MFC after: 3 days Notes: svn path=/head/; revision=295536
* As a followup to r292330, standardize on size_t and add a few comments.Dag-Erling Smørgrav2015-12-161-2/+5
| | | | Notes: svn path=/head/; revision=292332
* Reset bufpos to 0 immediately after refilling the buffer. Otherwise, weDag-Erling Smørgrav2015-12-161-3/+2
| | | | | | | | | | | | | risk leaving the connection in an indeterminate state if the server fails to send a chunk delimiter. Depending on the application and on the sizes of the preceding chunks, the result can be anything from missing data to a segfault. With this patch, it will be reported as a protocol error. PR: 204771 MFC after: 1 week Notes: svn path=/head/; revision=292330
* Fix buildworld after r291453, similar to r284346: url->user and url->pwdDimitry Andric2015-11-291-4/+2
| | | | | | | | | | are arrays, so they can never be NULL. Reported by: many Pointy hat to: des Notes: svn path=/head/; revision=291461
* Use .netrc for HTTP sites and proxies, not just FTP.Dag-Erling Smørgrav2015-11-291-0/+8
| | | | | | | | | PR: 193740 Submitted by: TEUBEL György <tgyurci@gmail.com> MFC after: 1 week Notes: svn path=/head/; revision=291453
* Fix two bugs in HTTPS tunnelling:Dag-Erling Smørgrav2015-10-161-4/+32
| | | | | | | | | | | | | - If the proxy returns a non-200 result, set the error code accordingly so the caller / user gets a somewhat meaningful error message. - Consume and discard any HTTP response header following the result line. PR: 194483 Tested by: Fabian Keil <fk@fabiankeil.de> MFC after: 1 week Notes: svn path=/head/; revision=289419
* Remove unused variable to silence clang warning.Marcelo Araujo2015-07-041-4/+1
| | | | | | | | Differential Revision: D2683 Reviewed by: rodrigc, bapt Notes: svn path=/head/; revision=285141
* Fix the following clang 3.7.0 warnings in lib/libfetch/http.c:Dimitry Andric2015-06-131-12/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | lib/libfetch/http.c:1628:26: error: address of array 'purl->user' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion] aparams.user = purl->user ? ~~~~~~^~~~ ~ lib/libfetch/http.c:1630:30: error: address of array 'purl->pwd' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion] aparams.password = purl->pwd? ~~~~~~^~~~ lib/libfetch/http.c:1657:25: error: address of array 'url->user' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion] aparams.user = url->user ? ~~~~~^~~~ ~ lib/libfetch/http.c:1659:29: error: address of array 'url->pwd' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion] aparams.password = url->pwd ? ~~~~~^~~ ~ lib/libfetch/http.c:1669:25: error: address of array 'url->user' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion] aparams.user = url->user ? ~~~~~^~~~ ~ lib/libfetch/http.c:1671:29: error: address of array 'url->pwd' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion] aparams.password = url->pwd ? ~~~~~^~~ ~ Since url->user and url->pwd are arrays, they can never be NULL, so the checks can be removed. Reviewed by: bapt MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D2673 Notes: svn path=/head/; revision=284346
* Add support for arbitrary http requestsBaptiste Daroussin2014-06-051-2/+32
| | | | | | | | | | Submitted by: Alex Hornung <alex@alexhornung.com> Reviewed by: des Obtained from: Dragonfly MFC after: 3 week Notes: svn path=/head/; revision=267133
* Remove unnecessary semicolonsBaptiste Daroussin2014-06-051-4/+4
| | | | | | | | | | | Patch by Sascha Wildner <saw@online.de> for Dragonfly Reviewed by: des Obtained from: Dragonfly MFC after: 1 week Notes: svn path=/head/; revision=267132
* Use NULL instead of 0Baptiste Daroussin2014-06-051-2/+2
| | | | | | | | | | | Patch by Sascha Wildner <saw@online.de> for Dragonfly Reviewed by: des Obtained from: Dragonfly MFC after: 1 week Notes: svn path=/head/; revision=267131
* If HTTP_USER_AGENT is defined but empty, don't send User-Agent at all.Dag-Erling Smørgrav2014-06-051-4/+9
| | | | | | | | | PR: 184507 Submitted by: jbeich@tormail.org (with modifications) MFC after: 1 week Notes: svn path=/head/; revision=267127
* Support Last-Modified behind proxies which return UTC instead of GMT.Bryan Drewery2014-03-111-0/+6
| | | | | | | | | | | | | | | | | | | The standard states that GMT must be used, but that UTC is equivalent. Still parse UTC as otherwise this causes problems for pkg(8). It will refetch the repository every time 'pkg update' or other remote operations are used behind these proxies. RFC2616: "All HTTP date/time stamps MUST be represented in Greenwich Mean Time (GMT), without exception. For the purposes of HTTP, GMT is exactly equal to UTC (Coordinated Universal Time)."" Approved by: bapt (mentor) Reviewed by: des, peter Sponsored by: EMC / Isilon Storage Division MFC after: 1 week Notes: svn path=/head/; revision=263021
* Bump copyright datesDag-Erling Smørgrav2014-01-301-1/+1
| | | | Notes: svn path=/head/; revision=261284
* r261230 broke the cases where the amount of data to be read is notDag-Erling Smørgrav2014-01-291-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | known in advance, or where the caller doesn't care and just keeps reading until it hits EOF. In fetch_read(): the socket is non-blocking, so read() will return 0 on EOF, and -1 (errno == EAGAIN) when the connection is still open but there is no data waiting. In the first case, we should immediately return 0. The EINTR case was also broken, although not in a way that matters. In fetch_writev(): use timersub() and timercmp() as in fetch_read(). In http_fillbuf(): set errno to a sensible value when an invalid chunk header is encountered. In http_readfn(): as in fetch_read(), a zero return from down the stack indicates EOF, not an error. Furthermore, when io->error is EINTR, clear it (but no errno) before returning so the caller can retry after dealing with the interrupt. MFC after: 3 days Notes: svn path=/head/; revision=261263
* Solve http buffering issues and hangs once and for all (hopefully!) byDag-Erling Smørgrav2014-01-281-21/+17
| | | | | | | | | | | | | | | | | | | | simply not trying to return exactly what the caller asked for - just return whatever we got and let the caller be the judge of whether it was enough. If an error occurs or the connection times out after we already received some data, return a short read, under the assumption that the next call will fail or time out before we read anything. As it turns out, none of the code that calls fetch_read() assumes an all-or-nothing result anyway, except for a couple of lines where we read the CR LF at the end of a hunk in HTTP hunked encoding, so the changes outside of fetch_read() and http_readfn() are minimal. While there, replace select(2) with poll(2). MFC after: 3 days Notes: svn path=/head/; revision=261230
* Even though it doesn't really make sense in the context of a CONNECTDag-Erling Smørgrav2013-08-221-0/+2
| | | | | | | | | | | | request, RFC 2616 14.23 mandates the presence of the Host: header in all HTTP 1.1 requests. PR: kern/181445 Submitted by: Kimo <kimor79@yahoo.com> MFC after: 3 days Notes: svn path=/head/; revision=254650
* Include an Accept header in requests.Dag-Erling Smørgrav2013-07-301-1/+7
| | | | | | | | PR: kern/180917 MFC after: 1 week Notes: svn path=/head/; revision=253805
* Implement certificate verification, and many other SSL-relatedDag-Erling Smørgrav2013-07-261-1/+1
| | | | | | | | | | | imrovements; complete details in the PR. PR: kern/175514 Submitted by: Michael Gmelin <freebsd@grem.de> MFC after: 1 week Notes: svn path=/head/; revision=253680
* Use the correct request syntax for proxied (tunneled) HTTPS requests.Dag-Erling Smørgrav2013-07-211-1/+1
| | | | | | | | PR: bin/180666 MFC after: 3 days Notes: svn path=/head/; revision=253514
* Use the CONNECT method to proxy HTTPS connections through HTTP proxies.Dag-Erling Smørgrav2013-04-121-8/+13
| | | | | | | | PR: bin/80176 Submitted by: Yuichiro NAITO <naito.yuichiro@gmail.com> Notes: svn path=/head/; revision=249431
* Fix weird indentation.Dag-Erling Smørgrav2012-11-161-4/+4
| | | | Notes: svn path=/head/; revision=243149
* Implement HTTP 305 redirect handling.Eitan Adler2012-10-221-1/+5
| | | | | | | | | | | PR: 172452 Submitted by: gcooper Reviewed by: des Approved by: cperciva MFC after: 1 week Notes: svn path=/head/; revision=241841
* Don't deny non-temporary redirects if the -A option is set (perEitan Adler2012-10-221-2/+12
| | | | | | | | | | | | | | | the man page) [0] While here add support for draft-reschke-http-status-308-07 PR: 172451 [0] Submitted by: gcooper [0] Reviewed by: des Approved by: cperciva MFC after: 1 week Notes: svn path=/head/; revision=241840
* Be a bit more lenient in the maximum number of redirects allowed.Eitan Adler2012-10-221-1/+1
| | | | | | | | | | | Chrome and Firefox have a limit of 20. IE has a limit of 8. Reviewed by: des Approved by: cperciva MFC after: 3 days Notes: svn path=/head/; revision=241839
* Use libmd if and only if OpenSSL is not available.Dag-Erling Smørgrav2012-09-141-0/+8
| | | | | | | | PR: bin/171402 MFC after: 3 days Notes: svn path=/head/; revision=240496
* Don't reuse credentials if redirected to a different host.Dag-Erling Smørgrav2012-04-301-1/+3
| | | | | | | | Submitted by: Niels Heinen <heinenn@google.com> MFC after: 3 weeks Notes: svn path=/head/; revision=234838
* Fix two issues related to the use of SIGINFO in fetch(1) to displayDag-Erling Smørgrav2012-01-181-5/+12
| | | | | | | | | | | | | | | | progress information. The first is that fetch_read() (used in the HTTP code but not the FTP code) can enter an infinite loop if it has previously been interrupted by a signal. The second is that when it is interrupted, fetch_read() will discard any data it may have read up to that point. Luckily, both bugs are extremely timing-sensitive and therefore difficult to trigger. PR: bin/153240 Submitted by: Mark <markjdb@gmail.com> MFC after: 3 weeks Notes: svn path=/head/; revision=230307
* latin1 -> utf8Dag-Erling Smørgrav2011-10-191-1/+1
| | | | Notes: svn path=/head/; revision=226537
* Update copyright dates and strip my middle name.Dag-Erling Smørgrav2011-09-271-1/+1
| | | | Notes: svn path=/head/; revision=225814
* Increase WARNS to 4.Dag-Erling Smørgrav2011-05-121-1/+1
| | | | Notes: svn path=/head/; revision=221822
* Mechanical whitespace cleanup.Dag-Erling Smørgrav2011-05-121-100/+100
| | | | Notes: svn path=/head/; revision=221821
* Increase WARNS to 3.Dag-Erling Smørgrav2011-05-121-1/+1
| | | | Notes: svn path=/head/; revision=221820
* Fix a couple of embarrassing mistakes in the previous commit.Dag-Erling Smørgrav2010-07-281-2/+2
| | | | | | | Submitted by: Dimitry Andric <dimitry@andric.com> Notes: svn path=/head/; revision=210563
* If the A flag is supplied, http_request() will attempt the request onlyDag-Erling Smørgrav2010-07-011-2/+4
| | | | | | | | | | | | | | once, even if authentication is required, instead of retrying with the proper credentials. Fix this by bumping the countdown if the origin or proxy server requests authentication so that the initial unauthenticated request does not count as an attempt. PR: 148087 Submitted by: Tom Evans <tevans.uk@googlemail.com> MFC after: 2 weeks Notes: svn path=/head/; revision=209632
* Add HTTP digest authentication.Dag-Erling Smørgrav2010-01-191-66/+816
| | | | | | | | Submitted by: Jean-Francois Dockes <jf@dockes.org> Forgotten by: des (repeatedly) Notes: svn path=/head/; revision=202613
* Add support for HTTP 1.1 If-Modified-Since behavior.Murray Stokely2008-12-151-3/+25
| | | | | | | | | | | | | | | | | fetch(1) accepts a new argument -i <file> that if specified will cause the file to be downloaded only if it is more recent than the mtime of <file>. libfetch(3) accepts the mtime in the url structure and a flag to indicate when this behavior is desired. PR: bin/87841 Submitted by: Jukka A. Ukkonen <jau@iki.fi> (partially) Reviewed by: des, ru MFC after: 3 weeks Notes: svn path=/head/; revision=186124
* Don't fail mistakenly with -r when we already have the whole file.Ruslan Ermilov2008-10-241-0/+1
| | | | | | | Reviewed by: des Notes: svn path=/head/; revision=184222
generated by cgit v1.2.3 (git 2.25.1) at 2025-11-07 00:12:28 +0000