| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Enhanced REP MOVSB feature of CPUs starting from Ivy Bridge makes
REP MOVSB the fastest way to copy memory in most of cases. However
Intel Optimization Reference Manual says: "setting the DF to force
REP MOVSB to copy bytes from high towards low addresses will expe-
rience significant performance degradation". Measurements on Intel
Cascade Lake and Alder Lake, same as on AMD Zen3 show that it can
drop throughput to as low as 2.5-3.5GB/s, comparing to ~10-30GB/s
of REP MOVSQ or hand-rolled loop, used for non-ERMS CPUs.
This patch keeps ERMS use for forward ordered memory copies, but
removes it for backward overlapped moves where it does not work.
This is just a cosmetic sync with kernel, since libc does not use
ERMS at this time.
Reviewed by: mjg
MFC after: 2 weeks
(cherry picked from commit f22068d91bf53696ee13a69685e809d35776ec3f)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a tradeoff which saves jumps for smaller sizes while making
the 8-16 range slower (roughly in line with the other cases).
Tested with glibc test suite.
For example size 3 (most common with vfs namecache) (ops/s):
before: 407086026
after: 461391995
The regressed range of 8-16 (with 8 as example):
before: 540850489
after: 461671032
(cherry picked from commit f1be262ec11c1c35e6485f432415b5b52adb505d)
|
| |
|
|
| |
(cherry picked from commit 164c3b81848bc81dc200b12370999474225447a3)
|
| |
|
|
|
|
|
|
|
|
| |
amd64: revamp memcmp
amd64: speed up failing case for memcmp
amd64: sync up libc memcmp with the kernel version (r357208)
amd64: sync up libc memcmp with the kernel version (r357309)
Notes:
svn path=/stable/12/; revision=363505
|
| |
|
|
|
|
|
|
|
|
|
| |
Unify i386 and amd64 getcontextx.c, and use ifuncs while there.
This is yet another attempt of the merge, previously done as r344436 and
reverted in r344463. It is redone since ld was changed to ifunc-capable
linker on i386.
Notes:
svn path=/stable/12/; revision=350352
|
| |
|
|
|
|
|
| |
Fix initial exec TLS mode for dynamically loaded shared objects.
Notes:
svn path=/stable/12/; revision=346155
|
| |
|
|
|
|
|
| |
Add usermode helpers for the Intel userspace protection keys feature.
Notes:
svn path=/stable/12/; revision=344850
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently stable/12 i386 installs old ld.bfd as ld (WITHOUT_LLD_IS_LD),
which prevents ifunc relocations from working properly. If a static
binary is linked, then it looses libc relocations needed.
The stable i386 branch uses ld.lld for the world build (WITH_LLD_BOOTSTRAP),
which is why static binaries from the base do not have the issue.
Reported and diagnosted by: antoine
Notes:
svn path=/stable/12/; revision=344463
|
| |
|
|
|
|
|
| |
Unify i386 and amd64 getcontextx.c, and use ifuncs while there.
Notes:
svn path=/stable/12/; revision=344436
|
| |
|
|
|
|
|
|
|
| |
amd64: tidy up copying backwards in memmove
amd64: remove stale attribution for memmove work
amd64: handle small memmove buffers with overlapping stores
Notes:
svn path=/stable/12/; revision=342240
|
| |
|
|
|
|
|
| |
amd64: align target memmove buffer to 16 bytes before using rep movs
Notes:
svn path=/stable/12/; revision=342238
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
amd64: tidy up memset to have rax set earlier for small sizes
amd64: finish the tail in memset with an overlapping store
amd64: align memset buffers to 16 bytes before using rep stos
amd64: convert libc bzero to a C func to avoid future bloat
amd64: sync up libc memset with the kernel version
amd64: handle small memset buffers with overlapping stores
Fix -DNO_CLEAN amd64 build after r340463
Notes:
svn path=/stable/12/; revision=340684
|
| |
|
|
|
|
|
| |
Convert amd64_get/set_fs/gsbase to ifunc.
Notes:
svn path=/stable/12/; revision=340608
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The function is of limited use and is an almost a direct clone of
memmove/memcpy (with arguments swapped). Introduction of ERMS variants
of string routines would mean avoidable growth of libc.
bcopy will get redefined to a __builtin_memmove later on with this
symbol only left for compatibility.
Reviewed by: kib
Approved by: re (gjb)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17539
Notes:
svn path=/head/; revision=339348
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
bcopy is left alone as it is expected to be converted to a C func.
Due to header mess ALIGN_TEXT is temporarily defined explicitly in memmove.S
Reviewed by: kib
Approved by: re (gjb)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17538
Notes:
svn path=/head/; revision=339347
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
See r339205 for details.
An unused ERMS support is retained in the macro. It will be activated
after ifunc support lands.
Reviewed by: kib
Approved by: re (gjb)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17405
Notes:
svn path=/head/; revision=339206
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a depessimization, see r334537 for an explanation. Routines
remain significantly slower than they have to be.
bzero was removed from the kernel but remains in libc. Macroify to
accommodate differences to memset (no return value, always setting to 0).
The bzero.S file is left in place due to libc build magic which pulls in
a C variant if a matching .S file is missing.
Reviewed by: kib
Approved by: re (gjb)
Differential Revision: https://reviews.freebsd.org/D17355
Notes:
svn path=/head/; revision=339077
|
| |
|
|
|
|
|
|
|
|
| |
The ABI specifies the direction forward on function call, making
the cld instruction redundant.
Approved by: re (kib)
Notes:
svn path=/head/; revision=339011
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Both are significantly slower than hand-coded loops. See r338963 for
kernel commit.
bcmp differs from memcmp by always returning 1 when a difference is
found, as opposed to going for a value bigger or lower than 0
depending on what it is. This means it can do less work. For now the
code is duplicated and modified. This will get deduplicated after
another round of optimization when memcmp will get a longer-term form.
Both tested with the glibc suite. While the suite does not have a test
for bcmp, I created a wrapper routine which verified that values match
(0 vs 0, 1 vs non-zero).
Reviewed by: kib
Approved by: re (gjb)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17336
Notes:
svn path=/head/; revision=338973
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The change resembles what was done in r334537 for kernel routines.
While here take care of i386 variants. Note that primitives remain
suboptimal.
Reviewed by: kib (previous version)
Approved by: re (gjb)
Differential Revision: https://reviews.freebsd.org/D17167
Notes:
svn path=/head/; revision=338713
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, libc.so would initialize its notion of the break address
using _end, a special symbol emitted by the static linker following
the bss section. Compatibility issues between lld and ld.bfd could
cause the wrong definition of _end (libc.so's definition rather than
that of the executable) to be used, breaking the brk()/sbrk()
interface.
Avoid this problem and future interoperability issues by simply not
relying on _end. Instead, modify the break() system call to return
the kernel's view of the current break address, and have libc
initialize its state using an extra syscall upon the first use of the
interface. As a side effect, this appears to fix brk()/sbrk() usage
in executables run with rtld direct exec, since the kernel and libc.so
no longer maintain separate views of the process' break address.
PR: 228574
Reviewed by: kib (previous version)
MFC after: 2 months
Differential Revision: https://reviews.freebsd.org/D15663
Notes:
svn path=/head/; revision=334626
|
| |
|
|
|
|
|
|
| |
SYS_exit has been defined in the repo since 1994 except for a brief
window when SYS_sys_exit was defined in 2000.
Notes:
svn path=/head/; revision=334500
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Originally, on the VAX exect() enable tracing once the new executable
image was loaded. This was possible because tracing was controllable
through user space code by setting the PSL_T flag. The following
instruction is a system call that activated tracing (as all
instructions do) by copying PSL_T to PSL_TP (trace pending). The
first instruction of the new executable image would trigger a trace
fault.
This is not portable to all platforms and the behavior was replaced with
ptrace(PT_TRACE_ME, ...) since FreeBSD forked off of the CSRG repository.
Platforms either incorrectly call execve(), trigger trace faults inside
the original executable, or do contain an implementation of this
function.
The exect() interfaces is deprecated or removed on NetBSD and OpenBSD.
Submitted by: Ali Mashtizadeh <ali@mashtizadeh.com>
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D14989
Notes:
svn path=/head/; revision=332443
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This caching has existed since the CSRG import, but serves no obvious
purpose. Sure, setlogin() is called rarely, but calls to getlogin()
should also be infrequent. The required invalidation was not
implemented on aarch64, arm, mips, amd riscv so updates would never
occur if getlogin() was called before setlogin().
Reported by: Ali Mashtizadeh <ali@mashtizadeh.com>
Reviewed by: kib
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D14965
Notes:
svn path=/head/; revision=332119
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
All of these files are identical (modulo license blocks and VCS IDs) to
the files generated by lib/libc/sys/Makefile.inc and serve no purpose.
Reported by: Ali Mashtizadeh <ali@mashtizadeh.com>
Reviewed by: kib
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D14953
Notes:
svn path=/head/; revision=332048
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
Notes:
svn path=/head/; revision=326193
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Mainly focus on files that use BSD 3-Clause license.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.
Notes:
svn path=/head/; revision=326025
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
instructions, if supported both by CPU and kernel.
Reviewed by: jhb (previous version)
Tested by: pho (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
Differential revision: https://reviews.freebsd.org/D12023
Notes:
svn path=/head/; revision=322763
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
MDSRCS it intended to allow assembly versions of funtions with C
implementations listed in MISRCS. The selection of the correct
machdep_ldis?.c for a given architecture does not follow this pattern
and the file should be added to SRCS directly.
Reviewed by: emaste, imp, jhb
MFC after: 1 week
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D9841
Notes:
svn path=/head/; revision=314557
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Remove .c files which duplicate entries in MISRCS.
- Use the same, less merge conflict prone style in all cases.
- Use MDSRCS for mips (.c and .S files both ended up in SRCS).
- Remove pointless sparc64 Makefile.inc.
- Remove uninformative foreign VCS ID entries.
Reviewed by: emaste, imp, jhb
MFC after: 1 week
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D9841
Notes:
svn path=/head/; revision=314556
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.
Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96
Notes:
svn path=/head/; revision=314436
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The initial value of NOASM is nearly the same in all cases and the
initial value of PSEUDO is the same in all cases so reduce duplication
(and hopefully, future merge conflicts) by machine independent defaults.
Also document the PSEUDO variable.
Reviewed by: jhb, kib
Obtained from: CheriBSD
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D7820
Notes:
svn path=/head/; revision=305630
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Besides removing hand-translation to assembler, this also adds missing
wrappers for arm64 and risc-v.
Reviewed by: emaste, jhb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D7694
Notes:
svn path=/head/; revision=305012
|
| |
|
|
|
|
|
|
|
|
|
| |
In particular, preserve syscall arguments on stack, since callee is
not required to preserve arg-passing registers. Align stack.
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Notes:
svn path=/head/; revision=304931
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since ptrace(2) syscall can return -1 for non-error situations, libc
wrappers set errno to 0 before performing the syscall, as the service
to the caller. On both i386 and amd64, the errno symbol was directly
referenced, which only works correctly in single-threaded process.
Change assembler wrappers for ptrace(2) to get current thread errno
location by calling __error(). Allow __error interposing, as
currently allowed in cerror().
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Notes:
svn path=/head/; revision=304928
|
| |
|
|
|
|
|
|
|
| |
Reported by: brooks
Reviewed by: brooks,jhb
Differential Revision: https://reviews.freebsd.org/D7548
Notes:
svn path=/head/; revision=304395
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Right now, userspace (fast) gettimeofday(2) on x86 only works for
RDTSC. For older machines, like Core2, where RDTSC is not C2/C3
invariant, and which fall to HPET hardware, this means that the call
has both the penalty of the syscall and of the uncached hw behind the
QPI or PCIe connection to the sought bridge. Nothing can me done
against the access latency, but the syscall overhead can be removed.
System already provides mappable /dev/hpetX devices, which gives
straight access to the HPET registers page.
Add yet another algorithm to the x86 'vdso' timehands. Libc is updated
to handle both RDTSC and HPET. For HPET, the index of the hpet device
to mmap is passed from kernel to userspace, index might be changed and
libc invalidates its mapping as needed.
Remove cpu_fill_vdso_timehands() KPI, instead require that
timecounters which can be used from userspace, to provide
tc_fill_vdso_timehands{,32}() methods. Merge i386 and amd64
libc/<arch>/sys/__vdso_gettc.c into one source file in the new
libc/x86/sys location. __vdso_gettc() internal interface is changed
to move timecounter algorithm detection into the MD code.
Measurements show that RDTSC even with the syscall overhead is faster
than userspace HPET access. But still, userspace HPET is three-four
times faster than syscall HPET on several Core2 and SandyBridge
machines.
Tested by: Howard Su <howard0su@gmail.com>
Sponsored by: The FreeBSD Foundation
MFC after: 1 month
Differential revision: https://reviews.freebsd.org/D7473
Notes:
svn path=/head/; revision=304285
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
value.
This eliminates the need for machine dependant assembly wrappers for
pipe(2).
It also make passing an invalid address to pipe(2) return EFAULT rather
than triggering a segfault. Document this behavior (which was already
true for pipe2(2), but undocumented).
Reviewed by: andrew
Approved by: re (gjb)
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D6815
Notes:
svn path=/head/; revision=302092
|
| |
|
|
|
|
|
| |
Mostly on comments.
Notes:
svn path=/head/; revision=298830
|
| |
|
|
|
|
|
|
|
|
|
|
| |
They are not used anywhere else in the base system and are an internal
implementation detail that does not need to be exposed.
Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5728
Notes:
svn path=/head/; revision=297238
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Further to r240152 (i386) and r240178 (amd64), hide the .cerror symbol
so that it is not exported if symbol versioning is not in use. Without
this change WITHOUT_SYMVER libc contains .text relocations for .cerror,
as described in LLVM PR 26813 (http://llvm.org/pr26813).
This is a no-op for the regular build as the symbol version script
already controls .cerror visibility.
PR: 207712
Submitted by: Rafael Espíndola
Reviewed by: jilles, kib
Differential Revision: https://reviews.freebsd.org/D5571
Notes:
svn path=/head/; revision=296474
|
| |
|
|
|
|
|
| |
warnings.
Notes:
svn path=/head/; revision=288019
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
are aliases for the syscall stubs and are plt-interposed, to the
libc-private aliases of internally interposed sigprocmask() etc.
Since e.g. _sigaction is not interposed by libthr, calling signal()
removes thr_sighandler() from the handler slot etc. The result was
breaking signal semantic and rtld locking.
The added __libc_sigprocmask and other symbols are hidden, they are
not exported and cannot be called through PLT. The setjmp/longjmp
functions for x86 were changed to use direct calls, and since
PIC_PROLOGUE only needed for functional PLT indirection on i386, it is
removed as well.
The PowerPC bug of calling the syscall directly in the setjmp/longjmp
implementation is kept as is.
Reported by: Pete French <petefrench@ingresso.co.uk>
Tested by: Michiel Boland <boland37@xs4all.nl>
Reviewed by: jilles (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Notes:
svn path=/head/; revision=287292
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
of the timehands, from the kern_tc.c implementation to vdso. Add
comments giving hints where to look for the algorithm explanation.
To compensate the removal of rmb() in userspace binuptime(), add
explicit lfence instruction before rdtsc. On i386, add usual
complications to detect SSE2 presence; assume that old CPUs which do
not implement SSE2 also execute rdtsc almost in order.
Reviewed by: alc, bde (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Notes:
svn path=/head/; revision=286284
|
| |
|
|
|
|
|
|
|
|
| |
broken - returning 0 from reboot(2) resulted in SIGBUS.
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Notes:
svn path=/head/; revision=285240
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pwrite(2) syscalls are wrapped to provide compatibility with pre-7.x
kernels which required padding before the off_t parameter. The
fcntl(2) contains compatibility code to handle kernels before the
struct flock was changed during the 8.x CURRENT development. The
shims were reasonable to allow easier revert to the older kernel at
that time.
Now, two or three major releases later, shims do not serve any
purpose. Such old kernels cannot handle current libc, so revert the
compatibility code.
Make padded syscalls support conditional under the COMPAT6 config
option. For COMPAT32, the syscalls were under COMPAT6 already.
Remove WITHOUT_SYSCALL_COMPAT build option, which only purpose was to
(partially) disable the removed shims.
Reviewed by: jhb, imp (previous versions)
Discussed with: peter
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Notes:
svn path=/head/; revision=281714
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Per POSIX, siglongjmp() shall be equivalent to longjmp() except that it must
match sigsetjmp() instead of setjmp() and except for the effect on the
signal mask. Therefore, it should preserve the floating point exception
flags.
This was fixed for longjmp() and _longjmp() in r180080 and r180081 for amd64
and i386 respectively.
Notes:
svn path=/head/; revision=267307
|
| |
|
|
| |
Notes:
svn path=/head/; revision=264155
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
if not already defined. This allows building libc from outside of
lib/libc using a reach-over makefile.
A typical use-case is to build a standard ILP32 version and a COMPAT32
version in a single iteration by building the COMPAT32 version using a
reach-over makefile.
Obtained from: Juniper Networks, Inc.
Notes:
svn path=/head/; revision=262722
|
| |
|
|
|
|
|
|
|
| |
CONCAT macros in SYS.h.
Reviewed by: bde, kib
Notes:
svn path=/head/; revision=258447
|