aboutsummaryrefslogtreecommitdiff
path: root/sys/i386/include/atomic.h
Commit message (Collapse)AuthorAgeFilesLines
* Expand generic subword atomic primitivesConrad Meyer2020-03-251-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The goal of this change is to make the atomic_load_acq_{8,16}, atomic_testandset{,_acq}_long, and atomic_testandclear_long primitives available in MI-namespace. The second goal is to get this draft out of my local tree, as anything that requires a full tinderbox is a big burden out of tree. MD specifics can be refined individually afterwards. The generic implementations may not be ideal for your architecture; feel free to implement better versions. If no subword_atomic definitions are needed, the include can be removed from your arch's machine/atomic.h. Generic definitions are guarded by defined macros of the same name. To avoid picking up conflicting generic definitions, some macro defines are added to various MD machine/atomic.h to register an existing implementation. Include _atomic_subword.h in arm and arm64 machine/atomic.h. For some odd reason, KCSAN only generates some versions of primitives. Generate the _acq variants of atomic_load.*_8, atomic_load.*_16, and atomic_testandset.*_long. There are other questionably disabled primitives, but I didn't run into them, so I left them alone. KCSAN is only built for amd64 in tinderbox for now. Add atomic_subword implementations of atomic_load_acq_{8,16} implemented using masking and atomic_load_acq_32. Add generic atomic_subword implementations of atomic_testandset_long(), atomic_testandclear_long(), and atomic_testandset_acq_long(), using atomic_fcmpset_long() and atomic_fcmpset_acq_long(). On x86, add atomic_testandset_acq_long as an alias for atomic_testandset_long. Reviewed by: kevans, rlibby (previous versions both) Differential Revision: https://reviews.freebsd.org/D22963 Notes: svn path=/head/; revision=359311
* i386: hide more of atomic 64-bit definitions under _KERNELAndriy Gapon2019-10-081-0/+2
| | | | | | | | | | | | At the moment i386 does not provide 64-bit atomic operations in userland. Exposing some atomic_*_64 defines can cause unnecessary confusion. Discussed with: kib MFC after: 2 weeks Notes: svn path=/head/; revision=353301
* i386: Implement atomic_load_64(9) and atomic_store_64(9).Konstantin Belousov2019-08-181-0/+2
| | | | | | | | Sponsored by: The FreeBSD Foundation MFC after: 1 week Notes: svn path=/head/; revision=351190
* Implement missing atomic_fcmpset_XXX() support for i386.Hans Petter Selasky2018-08-091-0/+51
| | | | | | | | | | This also fixes i386 build after r337527. MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=337529
* Make sure kernel modules built by default are portable between UP andHans Petter Selasky2018-07-061-2/+2
| | | | | | | | | | | | SMP systems by extending defined(SMP) to include defined(KLD_MODULE). This is a regression issue after r335873 . Discussed with: mmacy@ Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=336025
* inline atomics and allow tied modules to inline locksMatt Macy2018-07-021-1/+1
| | | | | | | | | | | | | | - inline atomics in modules on i386 and amd64 (they were always inline on other arches) - allow modules to opt in to inlining locks by specifying MODULE_TIED=1 in the makefile Reviewed by: kib Sponsored by: Limelight Networks Differential Revision: https://reviews.freebsd.org/D16079 Notes: svn path=/head/; revision=335873
* Implement atomic_add_64() and atomic_subtract_64() for the i386 target.Hans Petter Selasky2018-05-291-0/+36
| | | | | | | | | | | | While at it add missing _acq_ and _rel_ variants for 64-bit atomic operations under i386. Reviewed by: kib @ MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=334320
* Add atomic_load(9) and atomic_store(9) operations.Konstantin Belousov2017-12-191-0/+2
| | | | | | | | | | | | | | | | | | | | | | | They provide relaxed-ordered atomic access semantic. Due to the FreeBSD memory model, the operations are syntaxical wrappers around the volatile accesses. The volatile qualifier is used to ensure that the access not optimized out and in turn depends on the volatile semantic as implemented by supported compilers. The motivation for adding the operation is to help people coming from other systems or knowing the C11/C++ standards where atomics have special type and require use of the special access operations. It is still the case that FreeBSD requires plain load and stores of aligned integer types to be atomic. Suggested by: jhb Reviewed by: alc, jhb Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D13534 Notes: svn path=/head/; revision=326971
* sys/i386: further adoption of SPDX licensing ID tags.Pedro F. Giffuni2017-11-271-0/+2
| | | | | | | | | | | | | | | Mainly focus on files that use BSD 2-Clause license, however the tool I was using misidentified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Notes: svn path=/head/; revision=326260
* Implement atomic_fetchadd_64() for i386. This function is needed by theHans Petter Selasky2017-11-241-0/+12
| | | | | | | | | | | atomic64 header file in the LinuxKPI for i386. Reviewed by: kib MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=326161
* - Remove 'struct vmmeter' from 'struct pcpu', leaving only global vmmeterGleb Smirnoff2017-04-171-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | in place. To do per-cpu stats, convert all fields that previously were maintained in the vmmeters that sit in pcpus to counter(9). - Since some vmmeter stats may be touched at very early stages of boot, before we have set up UMA and we can do counter_u64_alloc(), provide an early counter mechanism: o Leave one spare uint64_t in struct pcpu, named pc_early_dummy_counter. o Point counter(9) fields of vmmeter to pcpu[0].pc_early_dummy_counter, so that at early stages of boot, before counters are allocated we already point to a counter that can be safely written to. o For sparc64 that required a whole dummy pcpu[MAXCPU] array. Further related changes: - Don't include vmmeter.h into pcpu.h. - vm.stats.vm.v_swappgsout and vm.stats.vm.v_swappgsin changed to 64-bit, to match kernel representation. - struct vmmeter hidden under _KERNEL, and only vmstat(1) is an exclusion. This is based on benno@'s 4-year old patch: https://lists.freebsd.org/pipermail/freebsd-arch/2013-July/014471.html Reviewed by: kib, gallatin, marius, lidl Differential Revision: https://reviews.freebsd.org/D10156 Notes: svn path=/head/; revision=317061
* Adjust the constraint for "src" in atomic_(f)cmpset_8.Mark Johnston2017-03-271-6/+6
| | | | | | | | | | | "r" is not sufficient to prevent the use of invalid byte-width registers with at least gcc. Reported and reviewed by: bde X-MFC-With: r315718 Notes: svn path=/head/; revision=316031
* Add support for 8- and 16-bit atomic_(f)cmpset to x86.Mark Johnston2017-03-221-37/+74
| | | | | | | | | Reviewed by: kib MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D10068 Notes: svn path=/head/; revision=315718
* For i386, remove config options CPU_DISABLE_CMPXCHG, CPU_DISABLE_SSEKonstantin Belousov2017-02-031-29/+0
| | | | | | | | | | | | | | | | | | | | and device npx. This means that FPU is always initialized and handled when available, and SSE+ register file and exception are handled when available. This makes the kernel FPU code much easier to maintain by the cost of slight bloat for CPUs older than 25 years. CPU_DISABLE_CMPXCHG outlived its usefulness, see the removed comment explaining the original purpose. Suggested by and discussed with: bde Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 3 weeks Notes: svn path=/head/; revision=313154
* i386: fixup fcmpsetMateusz Guzik2017-02-021-2/+2
| | | | | | | | | | | | An incorrect output specifier was used which worked with clang by accident, but breaks with the in-tree gcc version. While here plug a whitespace nit. Reported by: bde Notes: svn path=/head/; revision=313080
* i386: add atomic_fcmpsetMateusz Guzik2017-01-301-0/+34
| | | | | | | Tested by: pho Notes: svn path=/head/; revision=312975
* atomic: Add testandclear on i386/amd64Sepherosa Ziehau2016-05-161-0/+26
| | | | | | | | | Reviewed by: kib Sponsored by: Microsoft OSTC Differential Revision: https://reviews.freebsd.org/D6381 Notes: svn path=/head/; revision=299912
* Improve comments.Konstantin Belousov2015-07-301-3/+3
| | | | | | | | Submitted by: bde MFC after: 2 weeks Notes: svn path=/head/; revision=286078
* Use private cache line for the locked nop in *mb() on i386.Konstantin Belousov2015-07-301-32/+39
| | | | | | | | | | | Suggested by: alc Reviewed by: alc, bde Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Notes: svn path=/head/; revision=286051
* MFamd64 r285934: Remove store/load (= full) barrier from the i386Konstantin Belousov2015-07-291-17/+7
| | | | | | | | | | | | | atomic_load_acq_*(). Noted by: alc (long time ago) Reviewed by: alc, bde Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Notes: svn path=/head/; revision=286050
* Add the atomic_thread_fence() family of functions with intent toKonstantin Belousov2015-07-081-0/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | provide a semantic defined by the C11 fences with corresponding memory_order. atomic_thread_fence_acq() gives r | r, w, where r and w are read and write accesses, and | denotes the fence itself. atomic_thread_fence_rel() is r, w | w. atomic_thread_fence_acq_rel() is the combination of the acquire and release in single operation. Note that reads after the acq+rel fence could be made visible before writes preceeding the fence. atomic_thread_fence_seq_cst() orders all accesses before/after the fence, and the fence itself is globally ordered against other sequentially consistent atomic operations. Reviewed by: alc Discussed with: bde Sponsored by: The FreeBSD Foundation MFC after: 3 weeks Notes: svn path=/head/; revision=285283
* Add a comment about too strong semantic of atomic_load_acq() on x86.Konstantin Belousov2015-06-291-0/+9
| | | | | | | | Submitted by: bde MFC after: 2 weeks Notes: svn path=/head/; revision=284922
* Remove unneeded data dependency, currently imposed byKonstantin Belousov2015-06-281-49/+74
| | | | | | | | | | | | | | | | | | | | | | | | | | atomic_load_acq(9), on it source, for x86. Right now, atomic_load_acq() on x86 is sequentially consistent with other atomics, code ensures this by doing store/load barrier by performing locked nop on the source. Provide separate primitive __storeload_barrier(), which is implemented as the locked nop done on a cpu-private variable, and put __storeload_barrier() before load, to keep seq_cst semantic but avoid introducing false dependency on the no-modification of the source for its later use. Note that seq_cst property of x86 atomic_load_acq() is not documented and not carried by atomics implementations on other architectures, although some kernel code relies on the behaviour. This commit does not intend to change this. Reviewed by: alc Discussed with: bde Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Notes: svn path=/head/; revision=284901
* Implement atomic_cmpset_64() and atomic_swap_64() for i386.Jung-uk Kim2013-08-211-0/+113
| | | | Notes: svn path=/head/; revision=254620
* Reimplement atomic_load_acq_64() and atomic_store_rel_64() for i386. TheseJung-uk Kim2013-08-211-82/+110
| | | | | | | | | | functions are now real functions rather than function pointers. Supposedly, it is faster for modern processors. Suggested by: bde Notes: svn path=/head/; revision=254619
* Remove empty lines before return statements for style consistency.Jung-uk Kim2013-08-211-6/+0
| | | | Notes: svn path=/head/; revision=254618
* Implement atomic_swap() and atomic_testandset().Jung-uk Kim2013-08-211-19/+43
| | | | | | | Reviewed by: arch, bde, jilles, kib Notes: svn path=/head/; revision=254617
* - Remove the "a" constraint from main output operand for atomic_cmpset().Jung-uk Kim2013-08-211-5/+5
| | | | | | | - Use "+" modifier for the "expect" because it is also an output (unused). Notes: svn path=/head/; revision=254614
* Use '+' modifier for a memory operand that is both an input and an output.Jung-uk Kim2013-08-211-29/+21
| | | | | | | | | It was actually done in r86301 but reverted in r150182 because GCC 3.x was not able to handle it for a memory operand. Apparently, this problem was fixed in GCC 4.1+ and several contrib sources already rely on this feature. Notes: svn path=/head/; revision=254612
* Remove bogus labels. No functional change.Jung-uk Kim2013-08-211-1/+0
| | | | Notes: svn path=/head/; revision=254611
* Use consistent style. No functional change.Jung-uk Kim2013-08-211-10/+10
| | | | Notes: svn path=/head/; revision=254610
* Add an unified macro to deny ability from the compiler to reorderAttilio Rao2012-10-091-2/+2
| | | | | | | | | | | | | instruction loads/stores at its will. The macro __compiler_membar() is currently supported for both gcc and clang, but kernel compilation will fail otherwise. Reviewed by: bde, kib Discussed with: dim, theraven MFC after: 2 weeks Notes: svn path=/head/; revision=241374
* Use plain store for atomic_store_rel on x86, instead of implicitlyKonstantin Belousov2012-06-021-39/+39
| | | | | | | | | | | | locked xchg instruction. IA32 memory model guarantees that store has release semantic, since stores cannot pass loads or stores. Reviewed by: bde, jhb Tested by: pho MFC after: 2 weeks Notes: svn path=/head/; revision=236456
* Implement atomic_load_acq_64(9) and atomic_store_rel_64(9) for i386. TheseJung-uk Kim2011-04-061-0/+86
| | | | | | | | | | | | | | functions are implemented with CMPXCHG8B instruction where it is available, i. e., all Pentium-class and later processors. Note this instruction is also used for atomic_store_rel_64() because a simple XCHG-like instruction for 64-bit memory access does not exist, unfortunately. If the processor lacks the instruction, i. e., 80486-class CPUs, two 32-bit load/store are performed with interrupt temporarily disabled, assuming it does not support SMP. Although this assumption may be little naive, it is true in reality. This implementation is inspired by Linux. Notes: svn path=/head/; revision=220404
* Inform a compiler which asm statements in the x86 implementation ofKonstantin Belousov2010-12-181-6/+7
| | | | | | | | | | atomics change eflags. Reviewed by: jhb MFC after: 2 weeks Notes: svn path=/head/; revision=216524
* Rename an argument from "exp" to "expect" since the former makes FlexeLintPoul-Henning Kamp2010-05-201-8/+8
| | | | | | | | | uneasy, in case anybody think it might be exp(3) in libm. This also makes it consistent with other archs. Notes: svn path=/head/; revision=208332
* atomic_cmpset_barr_* was added in order to cope with compilers willing toAttilio Rao2009-10-091-64/+48
| | | | | | | | | | | | | | | | | | | | specify their own version of atomic_cmpset_* which could have been different than the membar version. Right now, however, FreeBSD is bound mostly to GCC-like compilers and it is desired to add new support and compat shim mostly when there is a real necessity, in order to avoid too much compatibility bloats. In this optic, bring back atomic_cmpset_{acq, rel}_* to be the same as atomic_cmpset_* and unwind the atomic_cmpset_barr_* introduction. Requested by: jhb Reviewed by: jhb Tested by: Giovanni Trematerra <giovanni dot trematerra at gmail dot com> Notes: svn path=/head/; revision=197910
* - All the functions in atomic.h needs to be in "physical" form (likeAttilio Rao2009-10-061-45/+50
| | | | | | | | | | | | | | | not defined through macros or similar) in order to be later compiled in the kernel and offer this way the support for modules (and compatibility among the UP case and SMP case). Fix this for the newly introduced atomic_cmpset_barr_* cases by defining and specifying a template. Note that the new DEFINE_CMPSET_GEN() template save more typing on amd64 than the current code. [1] - Fix the style for memory barriers on amd64. [1] Reported by: Paul B. Mahol <onemda at gmail dot com> Notes: svn path=/head/; revision=197824
* Per their definition, atomic instructions used in conjuction withAttilio Rao2009-10-061-43/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | memory barriers should also ensure that the compiler doesn't reorder paths where they are used. GCC, however, does that aggressively, even in presence of volatile operands. The most reliable way GCC offers for avoid instructions reordering is clobbering "memory" even if that is theoretically an heavy-weight operation, flushing the content of all the registers and forcing reload of them (We could rely, however, on gcc DTRT by just understanding the purpose as this is a well-known pattern for many modern operating-systems). Not all our memory barriers, right now, clobber memory for GCC-like compilers. The most notable cases are IA32 and amd64 where the memory barrier are treacted the same as normal atomic instructions. Fix this by offering the possibility to implement atomic instructions with memory barriers separately from the normal version and implement the GCC-like specific one using memory clobbering. Thanks to Chris Lattner (@apple) for his discussion on llvm specifics. Reported by: jhb Reviewed by: jhb Tested by: rdivacky, Giovanni Trematerra <giovanni dot trematerra at gmail dot com> Notes: svn path=/head/; revision=197803
* Restore memory clobber, to cause mb on the compiler level too.Konstantin Belousov2008-12-061-3/+3
| | | | | | | | | Use more sane formatting of the assembler. Pointed out by: bde Notes: svn path=/head/; revision=185720
* Unconditionally use locked addition of zero to tip of the stack forKonstantin Belousov2008-12-051-14/+3
| | | | | | | | | | | | | | memory barriers on i386. It works as a serialization instruction on all IA32 CPUs. Alternative solution of using {s,l,}fence requires run-time checking of the presense of the corresponding SSE or SSE2 extensions, and possible boot-time patching of the kernel text. Suggested by: many Notes: svn path=/head/; revision=185651
* - bump __FreeBSD version to reflect added buf_ring, memory barriers,Kip Macy2008-11-221-0/+15
| | | | | | | | | | | | | | | | | | | | and ifnet functions - add memory barriers to <machine/atomic.h> - update drivers to only conditionally define their own - add lockless producer / consumer ring buffer - remove ring buffer implementation from cxgb and update its callers - add if_transmit(struct ifnet *ifp, struct mbuf *m) to ifnet to allow drivers to efficiently manage multiple hardware queues (i.e. not serialize all packets through one ifq) - expose if_qflush to allow drivers to flush any driver managed queues This work was supported by Bitgravity Inc. and Chelsio Inc. Notes: svn path=/head/; revision=185162
* Implement atomic_fetchadd_long() for all architectures and document it.Pawel Jakub Dawidek2008-03-161-0/+7
| | | | | | | Reviewed by: attilio, jhb, jeff, kris (as a part of the uidinfo_waitfree.patch) Notes: svn path=/head/; revision=177276
* Fix oops in previous commit.Bruce Evans2006-12-291-1/+1
| | | | Notes: svn path=/head/; revision=165636
* Fixed some style bugs (mainly assorted errors in comments, and inconsistentBruce Evans2006-12-291-26/+26
| | | | | | | spelling of `result'). Notes: svn path=/head/; revision=165635
* Fixed some style bugs (whitespace only).Bruce Evans2006-12-291-31/+34
| | | | Notes: svn path=/head/; revision=165633
* Try harder to garbage-collect the "LOCORE" (really asm) version ofBruce Evans2006-12-291-5/+5
| | | | | | | | | | | | | MPLOCKED. The cleaning in rev.1.25 was supposed to have been undone by rev.1.26, but 1.26 could never have actually affected asm files since atomic.h is full of C declarations so including it in asm files would just give syntax errors. The asm MPLOCKED is even less needed than when misplaced definitions of it were first removed, and is now unused in any asm file in the src tree except in anachronismns in sys/i386/i386/support.s. Notes: svn path=/head/; revision=165630
* Avoid an instruction in atomic_cmpset_{int_long)() in most cases.Bruce Evans2006-12-271-11/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | These functions are used a lot for mutexes, so this reduces the text size of an average kernel by about 0.75%. This wasn't intended to be a significant optimization, but it somehow increased the maximum number of packets per second that can be transmitted by my bge hardware from 320000 to 460000 (this benchmark is CPU-bound and remarkably sensitive to changes in the text section). Details: we would prefer to leave the result of the cmpxchg in %al, but cannot tell gcc that it is there, so we have to convert it to an integer register. We converted to %al, then to %[re]ax, but the latter step is usually wasted since gcc usually only wants the condition code and can recover it from %al just as easily as from %[re]ax. Let gcc promote %al in the few cases where this is needed. Nearby style fixes; - let gcc manage the load of `res', and don't abuse `res' for a copy of `exp' - don't echo `res's name in comments - consistently spell the condition code as 'e' after comparison for equality - don't hard-code %al anywhere except in constraints - for the version that doesn't use cmpxchg, there is no requirement to use %al anywhere, so don't hard-code it in the constraints either. Style non-fix: - for the versions that use cmpxchg, keep using "a" (was %[re]ax, now %al) for the main output operand, although this is not required. The input and output operands that use the "a" constraint are now decoupled, and this makes things clearer except for the reason that the output register is hard-coded. It is now just a hack to tell gcc that the input "a" has been clobbered without increasing the number of operands. Notes: svn path=/head/; revision=165572
* Use wrapper macros for atomic pointer operations in order to perform theDag-Erling Smørgrav2006-03-281-18/+36
| | | | | | | correct casts. This should probably be merged to other architectures. Notes: svn path=/head/; revision=157212
* Add a new atomic_fetchadd() primitive that atomically adds a value to aJohn Baldwin2005-09-271-0/+21
| | | | | | | | | | | | variable and returns the previous value of the variable. Tested on: i386, alpha, sparc64, arm (cognet) Reviewed by: arch@ Submitted by: cognet (arm) MFC after: 1 week Notes: svn path=/head/; revision=150627
generated by cgit v1.3 (git 2.53.0) at 2026-04-16 15:02:01 +0000