summaryrefslogtreecommitdiff
path: root/sys/arm/include/atomic.h
Commit message (Collapse)AuthorAgeFilesLines
* Remove now unused armv4 and not-INTRNG files.Michal Meloun2020-11-281-4/+0
| | | | Notes: svn path=/head/; revision=368126
* Expand generic subword atomic primitivesConrad Meyer2020-03-251-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The goal of this change is to make the atomic_load_acq_{8,16}, atomic_testandset{,_acq}_long, and atomic_testandclear_long primitives available in MI-namespace. The second goal is to get this draft out of my local tree, as anything that requires a full tinderbox is a big burden out of tree. MD specifics can be refined individually afterwards. The generic implementations may not be ideal for your architecture; feel free to implement better versions. If no subword_atomic definitions are needed, the include can be removed from your arch's machine/atomic.h. Generic definitions are guarded by defined macros of the same name. To avoid picking up conflicting generic definitions, some macro defines are added to various MD machine/atomic.h to register an existing implementation. Include _atomic_subword.h in arm and arm64 machine/atomic.h. For some odd reason, KCSAN only generates some versions of primitives. Generate the _acq variants of atomic_load.*_8, atomic_load.*_16, and atomic_testandset.*_long. There are other questionably disabled primitives, but I didn't run into them, so I left them alone. KCSAN is only built for amd64 in tinderbox for now. Add atomic_subword implementations of atomic_load_acq_{8,16} implemented using masking and atomic_load_acq_32. Add generic atomic_subword implementations of atomic_testandset_long(), atomic_testandclear_long(), and atomic_testandset_acq_long(), using atomic_fcmpset_long() and atomic_fcmpset_acq_long(). On x86, add atomic_testandset_acq_long as an alias for atomic_testandset_long. Reviewed by: kevans, rlibby (previous versions both) Differential Revision: https://reviews.freebsd.org/D22963 Notes: svn path=/head/; revision=359311
* Implement atomic_swap_xxx() for all platforms.Hans Petter Selasky2018-12-101-0/+7
| | | | | | | | | | Differential Revision: https://reviews.freebsd.org/D18450 Reviewed by: kib@ MFC after: 3 days Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=341787
* Remove arm-specific implementations of atomic_load/store_xxx() now thatIan Lepore2017-12-201-27/+0
| | | | | | | they are provided by sys/atomic_common.h. Notes: svn path=/head/; revision=327047
* Add atomic_load(9) and atomic_store(9) operations.Konstantin Belousov2017-12-191-0/+2
| | | | | | | | | | | | | | | | | | | | | | | They provide relaxed-ordered atomic access semantic. Due to the FreeBSD memory model, the operations are syntaxical wrappers around the volatile accesses. The volatile qualifier is used to ensure that the access not optimized out and in turn depends on the volatile semantic as implemented by supported compilers. The motivation for adding the operation is to help people coming from other systems or knowing the C11/C++ standards where atomics have special type and require use of the special access operations. It is still the case that FreeBSD requires plain load and stores of aligned integer types to be atomic. Suggested by: jhb Reviewed by: alc, jhb Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D13534 Notes: svn path=/head/; revision=326971
* sys/arm: further adoption of SPDX licensing ID tags.Pedro F. Giffuni2017-11-271-0/+2
| | | | | | | | | | | | | | | Mainly focus on files that use BSD 2-Clause license, however the tool I was using misidentified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Notes: svn path=/head/; revision=326258
* Stop including sys/types.h from arm's machine/atomic.h, fix the placesIan Lepore2017-02-111-1/+0
| | | | | | | | where atomic.h was being included without ensuring that types.h (via param.h) was included first, as required by atomic(9). Notes: svn path=/head/; revision=313573
* Implement atomic_fcmpset_* for arm and arm64.Olivier Houchard2017-01-281-0/+6
| | | | Notes: svn path=/head/; revision=312925
* Remove arm's cpuconf.h, and references to it, after moving a few lines fromIan Lepore2017-01-161-2/+0
| | | | | | | | | | | | | | | | it into pmap-v4.h where they are used. Other than those few lines of support for different MMU types, nothing in cpuconf.h has been used in our code for quite a while. The file existed to set up a variety of symbols to describe the architecture. Over the past few years we have converted all of our source to use the new architecture symbols standardized by ARM Inc, and predefined by both clang and gcc. PR: 216104 Notes: svn path=/head/; revision=312292
* Add some missing atomic_*_ptr #defines for arm.Mark Johnston2017-01-041-1/+8
| | | | | | | MFC after: 1 week Notes: svn path=/head/; revision=311204
* Include machine/acle-compat.h in cdefs.h on arm if the compiler doesn'tIan Lepore2016-05-251-2/+0
| | | | | | | | | | | | | | | | | | | | | | | | | have ACLE support built in. The ACLE (ARM C Language Extensions) defines a set of standardized symbols which indicate the architecture version and features available. ACLE support is built in to modern compilers (both clang and gcc), but absent from gcc prior to 4.4. ARM (the company) provides the acle-compat.h header file to define the right symbols for older versions of gcc. Basically, acle-compat.h does for arm about the same thing cdefs.h does for freebsd: defines standardized macros that work no matter which compiler you use. If ARM hadn't provided this file we would have ended up with a big #ifdef __arm__ section in cdefs.h with our own compatibility shims. Remove #include <machine/acle-compat.h> from the zillion other places (an ever-growing list) that it appears. Since style(9) requires sys/types.h or sys/param.h early in the include list, and both of those lead to including cdefs.h, only a couple special cases still need to include acle-compat.h directly. Loves it: imp Notes: svn path=/head/; revision=300694
* ARM: Implement atomic_swap_int(9). It's used in DRM2 code.Michal Meloun2015-11-281-0/+1
| | | | | | | Approved by: kib (mentor) Notes: svn path=/head/; revision=291426
* FreeBSD does not support SMP on ARMv5. Since processor is alwaysKonstantin Belousov2015-10-021-28/+0
| | | | | | | | | | | | | | | | | | | self-consistent, there is no need in anything but compiler barrier in the implementation of atomic_thread_fence_*() on ARMv5. Split implementation of fences for ARMv4/5 and ARMv6; the former use compiler barriers, the later also perform hardware barriers. An issue which is fixed by the change is the faults from the CP15 coprocessor accesses in the user mode. This was uncovered by the pthread_once() changes in r287556. Reported by: Mattia Rossi <mattia.rossi.mailinglists@gmail.com> Discussed with: alc, cognet, jhb Sponsored by: The FreeBSD Foundation MFC after: 1 week Notes: svn path=/head/; revision=288491
* Split out the arm and armv6 parts of atomic.h to new files. While here useAndrew Turner2015-07-161-1026/+5
| | | | | | | | | __ARM_ARCH to determine which revision of the architecture is applicable. Sponsored by: ABT Systems Ltd Notes: svn path=/head/; revision=285631
* Add the atomic_thread_fence() family of functions with intent toKonstantin Belousov2015-07-081-0/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | provide a semantic defined by the C11 fences with corresponding memory_order. atomic_thread_fence_acq() gives r | r, w, where r and w are read and write accesses, and | denotes the fence itself. atomic_thread_fence_rel() is r, w | w. atomic_thread_fence_acq_rel() is the combination of the acquire and release in single operation. Note that reads after the acq+rel fence could be made visible before writes preceeding the fence. atomic_thread_fence_seq_cst() orders all accesses before/after the fence, and the fence itself is globally ordered against other sequentially consistent atomic operations. Reviewed by: alc Discussed with: bde Sponsored by: The FreeBSD Foundation MFC after: 3 weeks Notes: svn path=/head/; revision=285283
* List both registers to use in the 64-bit atomic instructions. We will needAndrew Turner2015-05-111-19/+19
| | | | | | | these to build for Thumb-2. Notes: svn path=/head/; revision=282776
* Revert r279338. The casts are apparently bogus, despite the fact thatIan Lepore2015-03-021-17/+7
| | | | | | | they've been working in i386 (where this change came from). Notes: svn path=/head/; revision=279543
* Add casting to make atomic ops work for pointers. (Apparently nobody hasIan Lepore2015-02-261-7/+17
| | | | | | | | | ever done atomic ops on pointers before now on arm). Submitted by: Svatopluk Kraus <onwahe@gmail.com> Notes: svn path=/head/; revision=279338
* Correct a comment which was exactly backwards from reality.Ian Lepore2015-02-211-2/+2
| | | | Notes: svn path=/head/; revision=279114
* Unify interrupts bit definition and usage. While here remove PSR_C_bit.Andrew Turner2014-09-101-7/+3
| | | | | | | | | Submitted by: Svatopluk Kraus <onwahe at gmail.com>, Michal Meloun <meloun at miracle.cz> Differential Revision: https://reviews.freebsd.org/D754 Notes: svn path=/head/; revision=271398
* Rename new to newval in inline asm code, to avoid clashes with C++ new.Ian Lepore2014-09-091-5/+5
| | | | | | | | Also rename cmp to cmpval just to keep the asm variable names similar to the C variable names. Notes: svn path=/head/; revision=271310
* When arm 64-bit atomic ops are available, define ARM_HAVE_ATOMIC64. UseIan Lepore2014-08-021-0/+4
| | | | | | | | that symbol (which will be correct in both kernel and userland contexts) rather than just __arm__ to decide whether to use a local implementation. Notes: svn path=/head/; revision=269414
* Add 64-bit atomic ops for armv4, only for kernel code, mostly so that weIan Lepore2014-08-011-0/+69
| | | | | | | | | | don't need any #ifdef stuff to use atomic_load/store_64() elsewhere in the kernel. For armv4 the atomics are trivial to implement for kernel code (just disable interrupts), less so for user mode, so this only has the kernel mode implementations for now. Notes: svn path=/head/; revision=269405
* Add 64-bit atomic ops for armv6. The only safe way to access a 64-bitIan Lepore2014-08-011-0/+248
| | | | | | | | | | | | | value shared across multiple cores is with atomic_load_64() and atomic_store_64(), because the normal 64-bit load/store instructions are not atomic on 32-bit arm. Luckily the ldrexd/strexd instructions that are atomic are fairly cheap on armv6. Because it's fairly simple to do, this implements all the ops for 64-bit, not just load/store. Reviewed by: andrew, cognet Notes: svn path=/head/; revision=269403
* Make the hardware memory and instruction barrier functions work on armv4Ian Lepore2014-05-111-3/+3
| | | | | | | and armv5 as well. Notes: svn path=/head/; revision=265861
* Update all arm code that manipulates the PSR registers to use modern syntax.Ian Lepore2014-02-021-2/+2
| | | | | | | | | | | | | | | It turns out the version of gas we're using interprets the old '_all' mask as 'fc' instead of 'fsxc'. That is, "all" doesn't really mean "all". This was the cause of the "wrong-endian register restore" bug that's been causing problems with some cortex-a9 chips. The 'endian' bit in the spsr register would never get changed (it falls into the 'x' mask group) and the first return-from-exception would fail if the chip had powered on with garbage in the spsr register that included the big-endian bit. It's unknown why this affected only certain cortex-a9 chips. Notes: svn path=/head/; revision=261393
* Fix an itt instruction. We need to execute both the mov and b instructionsAndrew Turner2013-10-261-1/+1
| | | | | | | when building for Thumb. Notes: svn path=/head/; revision=257189
* Start adding support to build bits of our code using the Thumb-2Andrew Turner2013-07-201-9/+26
| | | | | | | | | | | | | | | | | | instruction set. Thumb-2 requires an if-then instruction to implement conditional codes. When building for ARM mode the it-then instructions do not generate any assembled instruction as per the ARMv7-A Architecture Reference Manual, and are safe to use. While this allows the atomic instructions to be built, it doesn't mean we fully support Thumb code. It works in small tests, but is still known to fail in a large number of places. While here add a check for the armv6t2 architecture. Notes: svn path=/head/; revision=253489
* Don't define rel/acq variants of some atomic operations as the regularOlivier Houchard2013-01-151-13/+12
| | | | | | | version for armv6. Notes: svn path=/head/; revision=245475
* Implement barriers for AMRv6 and ARMv7Oleksandr Tymoshenko2013-01-071-3/+19
| | | | | | | | Submitted by: Daisuke Aoyama <aoyama at peach.ne.jp> Reviewed by: ian, cognet Notes: svn path=/head/; revision=245135
* Fix the clobber list on the atomic operators that do comparisons. WithoutAndrew Turner2012-10-011-15/+17
| | | | | | | | | this some compilers will place a cmp instruction before the atomic operation and expect to be able to use the result afterwards. By adding "cc" to the list of used registers we tell the compiler to not do this. Notes: svn path=/head/; revision=241080
* Merging projects/armv6, part 1Oleksandr Tymoshenko2012-08-151-59/+455
| | | | | | | | | | | | Cummulative patch of changes that are not vendor-specific: - ARMv6 and ARMv7 architecture support - ARM SMP support - VFP/Neon support - ARM Generic Interrupt Controller driver - Simplification of startup code for all platforms Notes: svn path=/head/; revision=239268
* Revert committal of local change accidentally swept up in r238329.Warner Losh2012-07-101-3/+1
| | | | Notes: svn path=/head/; revision=238347
* Remove some unused variables/externs that have been copied too many times...Warner Losh2012-07-101-1/+3
| | | | Notes: svn path=/head/; revision=238329
* trim trailing whitespaceWarner Losh2012-06-131-1/+0
| | | | Notes: svn path=/head/; revision=236992
* Fix the userland, RAS, version of atomic_fetchadd_32 :Olivier Houchard2009-03-311-9/+10
| | | | | | | | | | return the correct value, and do not store the wrong one in the supplied pointer. Submitted by: Mark Tinguely <tinguely casselton net> Notes: svn path=/head/; revision=190603
* force atomic_cmpset_ptr types to match atomic_cmpset_32;Sam Leffler2009-02-031-1/+2
| | | | | | | | | | this matches what powerpc does Submitted by: stass MFC after: 2 weeks Notes: svn path=/head/; revision=188085
* - bump __FreeBSD version to reflect added buf_ring, memory barriers,Kip Macy2008-11-221-0/+4
| | | | | | | | | | | | | | | | | | | | and ifnet functions - add memory barriers to <machine/atomic.h> - update drivers to only conditionally define their own - add lockless producer / consumer ring buffer - remove ring buffer implementation from cxgb and update its callers - add if_transmit(struct ifnet *ifp, struct mbuf *m) to ifnet to allow drivers to efficiently manage multiple hardware queues (i.e. not serialize all packets through one ifq) - expose if_qflush to allow drivers to flush any driver managed queues This work was supported by Bitgravity Inc. and Chelsio Inc. Notes: svn path=/head/; revision=185162
* Improve ARM_TP_ADDRESS and RAS area.Rafal Jaworowski2008-02-051-54/+29
| | | | | | | | | | | | | | | | | De-hardcode usage of ARM_TP_ADDRESS and RAS local storage, and move this special purpose page to a more convenient place i.e. after the vectors high page, more towards the end of address space. Previous location (0xe000_0000) caused grief if KVA was to go beyond the default limit. Note that ARM world rebuilding is required after this change since the location of ARM_TP_ADDRESS is shared between kernel and userland. Submitted by: Grzegorz Bernacki (gjb AT semihalf dot com) Reviewed by: imp Approved by: cognet (mentor) Notes: svn path=/head/; revision=175982
* Close a race.Olivier Houchard2007-12-021-18/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The RAS implementation would set the end address, then the start address. These were used by the kernel to restart a RAS sequence if it was interrupted. When the thread switching code ran, it would check these values and adjust the PC and clear them if it did. However, there's a small flaw in this scheme. Thread T1, sets the end address and gets preempted. Thread T2 runs and also does a RAS operation. This resets end to zero. Thread T1 now runs again and sets start and then begins the RAS sequence, but is preempted before the RAS sequence executes its last instruction. The kernel code that would ordinarily restart the RAS sequence doesn't because the PC isn't between start and 0, so the PC isn't set to the start of the sequence. So when T1 is resumed again, it is at the wrong location for RAS to produce the correct results. This causes the wrong results for the atomic sequence. The window for the first race is 3 instructions. The window for the second race is 5-10 instructions depending on the atomic operation. This makes this failure fairly rare and hard to reproduce. Mutexs are implemented in libthr using atomic operations. When the above race would occur, a lock could get stuck locked, causing many downstream problems, as you might expect. Also, make sure to reset the start and end address when doing a syscall, or a malicious process could set them before doing a syscall. Reviewed by: imp, ups (thanks guys) Pointy hat to: cognet MFC After: 3 days Notes: svn path=/head/; revision=174170
* In atomic_fetchadd_32(), do not blindly increase the value of %3.Olivier Houchard2007-11-271-2/+3
| | | | | | | | | | | It should just contain the value we want to add, as if we're interrupted between the add and the str, we will restart from the beginning. Just use a register we can scratch instead. MFC After: 1 week Notes: svn path=/head/; revision=173999
* MFp4: Add missing atomic functionsBernd Walter2007-01-051-34/+64
| | | | | | | Based on a patch by: des Notes: svn path=/head/; revision=165786
* Add atomic_cmpset_acq_32.Olivier Houchard2006-11-071-0/+1
| | | | Notes: svn path=/head/; revision=164059
* Add definitions for atomic_subtract_rel_32, atomic_add_rel_32 andOlivier Houchard2006-05-151-0/+3
| | | | | | | atomic_load_acq_32, needed for hwpmc. Notes: svn path=/head/; revision=158593
* Disable/enable fiqs as well as irqs.Olivier Houchard2006-04-131-1/+1
| | | | Notes: svn path=/head/; revision=157725
* Use memory clobbers, to be on the safe side.Olivier Houchard2006-02-061-6/+11
| | | | | | | Suggested by: jhb Notes: svn path=/head/; revision=155391
* Backout rev 1.12. It would have been a good thing, if gcc was smart enoughOlivier Houchard2006-02-051-24/+18
| | | | | | | not to generate bad code. Notes: svn path=/head/; revision=155355
* A #define is not enough, we need to cast from u_long * to uint32_t *.Olivier Houchard2005-12-091-1/+8
| | | | Notes: svn path=/head/; revision=153276
* Define atomic_whatever_longOlivier Houchard2005-12-091-0/+4
| | | | Notes: svn path=/head/; revision=153275
* Whitespace.John Baldwin2005-10-141-1/+1
| | | | Notes: svn path=/head/; revision=151340
generated by cgit v1.2.3 (git 2.25.1) at 2025-10-16 21:55:32 +0000