- Architecture/platform information for compiler writers -
- -
-
-
-
-
-
-
-
-Note: This document is a work-in-progress. Additions and clarifications - are welcome.
-Hardware
- - -
-
-
-
-
-
-ARM
- -
-
-
-
--
-
- ARM documentation -(Processor -Cores) -
- ABI -
Itanium (ia64)
- -
-
-
-
-
-MIPS
- -
-
-
-
-
-PowerPC
- -
-
-
-
-
-
-IBM - Official manuals and docs
- -
-
-
-
-
--
-
- PowerPC
-Architecture Book
-
-
-
- Book I: PowerPC - User Instruction Set Architecture -
- Book II: PowerPC - Virtual Environment Architecture -
- Book III: PowerPC - Operating Environment Architecture -
- - PowerPC -Compiler Writer's Guide -
- PowerPC -Processor Manuals -
- Intro to -PowerPC architecture -
- IBM AIX/5L for POWER Assembly reference -
Other documents, collections, notes
- - - -SPARC
- -
-
-
-
-
-
-
-X86
- -
-
-
-
-
-
-AMD - Official manuals and docs
- -
-
-
-
-
-Intel - Official manuals and docs
- - - - -Other x86-specific information
- - - -Other relevant lists
- -
-
-
-
-
-
-ABI
- - - - - -Miscellaneous resources
- - --
-
- Executable -File Format library -
- GCC prefetch project -page has a good survey of the prefetching capabilities of a variety of modern -processors. -
- -
- - LLVM Compiler Infrastructure
- Last modified: $Date: 2012-04-19 22:20:34 +0200 (Thu, 19 Apr 2012) $ - - - - diff --git a/docs/CompilerWriterInfo.rst b/docs/CompilerWriterInfo.rst new file mode 100644 index 000000000000..e41f5f9eecea --- /dev/null +++ b/docs/CompilerWriterInfo.rst @@ -0,0 +1,118 @@ +.. _compiler_writer_info: + +======================================================== +Architecture & Platform Information for Compiler Writers +======================================================== + +.. contents:: + :local: + +.. note:: + + This document is a work-in-progress. Additions and clarifications are + welcome. + + Compiled by `Misha Brukman
Debugging JIT-ed Code With GDB
--
-
- Background -
- GDB Version -
- Debugging MCJIT-ed code -
- Example -
-
-
Background
- -
-
-
-
-
-Without special runtime support, debugging dynamically generated code with -GDB (as well as most debuggers) can be quite painful. Debuggers generally read -debug information from the object file of the code, but for JITed code, there is -no such file to look for. -
- -In order to communicate the necessary debug info to GDB, an interface for -registering JITed code with debuggers has been designed and implemented for -GDB and LLVM MCJIT. At a high level, whenever MCJIT generates new machine code, -it does so in an in-memory object file that contains the debug information in -DWARF format. MCJIT then adds this in-memory object file to a global list of -dynamically generated object files and calls a special function -(__jit_debug_register_code) marked noinline that GDB knows about. When -GDB attaches to a process, it puts a breakpoint in this function and loads all -of the object files in the global list. When MCJIT calls the registration -function, GDB catches the breakpoint signal, loads the new object file from -the inferior's memory, and resumes the execution. In this way, GDB can get the -necessary debug information. -
-GDB Version
- - -In order to debug code JIT-ed by LLVM, you need GDB 7.0 or newer, which is -available on most modern distributions of Linux. The version of GDB that Apple -ships with Xcode has been frozen at 6.3 for a while. LLDB may be a better -option for debugging JIT-ed code on Mac OS X. -
- - - -Debugging MCJIT-ed code
- -
-
-
-
-
-
-The emerging MCJIT component of LLVM allows full debugging of JIT-ed code with -GDB. This is due to MCJIT's ability to use the MC emitter to provide full -DWARF debugging information to GDB.
- -Note that lli has to be passed the -use-mcjit flag to JIT the code -with MCJIT instead of the old JIT.
- -Example
- -
-
-
-Consider the following C code (with line numbers added to make the example -easier to follow):
- -
-1 int compute_factorial(int n)
-2 {
-3 if (n <= 1)
-4 return 1;
-5
-6 int f = n;
-7 while (--n > 1)
-8 f *= n;
-9 return f;
-10 }
-11
-12
-13 int main(int argc, char** argv)
-14 {
-15 if (argc < 2)
-16 return -1;
-17 char firstletter = argv[1][0];
-18 int result = compute_factorial(firstletter - '0');
-19
-20 // Returned result is clipped at 255...
-21 return result;
-22 }
-
-
-Here is a sample command line session that shows how to build and run this -code via lli inside GDB: -
- --$ $BINPATH/clang -cc1 -O0 -g -emit-llvm showdebug.c -$ gdb --quiet --args $BINPATH/lli -use-mcjit showdebug.ll 5 -Reading symbols from $BINPATH/lli...done. -(gdb) b showdebug.c:6 -No source file named showdebug.c. -Make breakpoint pending on future shared library load? (y or [n]) y -Breakpoint 1 (showdebug.c:6) pending. -(gdb) r -Starting program: $BINPATH/lli -use-mcjit showdebug.ll 5 -[Thread debugging using libthread_db enabled] - -Breakpoint 1, compute_factorial (n=5) at showdebug.c:6 -6 int f = n; -(gdb) p n -$1 = 5 -(gdb) p f -$2 = 0 -(gdb) n -7 while (--n > 1) -(gdb) p f -$3 = 5 -(gdb) b showdebug.c:9 -Breakpoint 2 at 0x7ffff7ed404c: file showdebug.c, line 9. -(gdb) c -Continuing. - -Breakpoint 2, compute_factorial (n=1) at showdebug.c:9 -9 return f; -(gdb) p f -$4 = 120 -(gdb) bt -#0 compute_factorial (n=1) at showdebug.c:9 -#1 0x00007ffff7ed40a9 in main (argc=2, argv=0x16677e0) at showdebug.c:18 -#2 0x3500000001652748 in ?? () -#3 0x00000000016677e0 in ?? () -#4 0x0000000000000002 in ?? () -#5 0x0000000000d953b3 in llvm::MCJIT::runFunction (this=0x16151f0, F=0x1603020, ArgValues=...) at /home/ebenders_test/llvm_svn_rw/lib/ExecutionEngine/MCJIT/MCJIT.cpp:161 -#6 0x0000000000dc8872 in llvm::ExecutionEngine::runFunctionAsMain (this=0x16151f0, Fn=0x1603020, argv=..., envp=0x7fffffffe040) - at /home/ebenders_test/llvm_svn_rw/lib/ExecutionEngine/ExecutionEngine.cpp:397 -#7 0x000000000059c583 in main (argc=4, argv=0x7fffffffe018, envp=0x7fffffffe040) at /home/ebenders_test/llvm_svn_rw/tools/lli/lli.cpp:324 -(gdb) finish -Run till exit from #0 compute_factorial (n=1) at showdebug.c:9 -0x00007ffff7ed40a9 in main (argc=2, argv=0x16677e0) at showdebug.c:18 -18 int result = compute_factorial(firstletter - '0'); -Value returned is $5 = 120 -(gdb) p result -$6 = 23406408 -(gdb) n -21 return result; -(gdb) p result -$7 = 120 -(gdb) c -Continuing. - -Program exited with code 0170. -(gdb) - -- -
- -
- - The LLVM Compiler Infrastructure
- Last modified: $Date: 2012-05-13 16:36:15 +0200 (Sun, 13 May 2012) $ - - - diff --git a/docs/DebuggingJITedCode.rst b/docs/DebuggingJITedCode.rst new file mode 100644 index 000000000000..eeb2f7787dae --- /dev/null +++ b/docs/DebuggingJITedCode.rst @@ -0,0 +1,147 @@ +.. _debugging-jited-code: + +============================== +Debugging JIT-ed Code With GDB +============================== + +.. sectionauthor:: Reid Kleckner and Eli Bendersky + +Background +========== + +Without special runtime support, debugging dynamically generated code with +GDB (as well as most debuggers) can be quite painful. Debuggers generally +read debug information from the object file of the code, but for JITed +code, there is no such file to look for. + +In order to communicate the necessary debug info to GDB, an interface for +registering JITed code with debuggers has been designed and implemented for +GDB and LLVM MCJIT. At a high level, whenever MCJIT generates new machine code, +it does so in an in-memory object file that contains the debug information in +DWARF format. MCJIT then adds this in-memory object file to a global list of +dynamically generated object files and calls a special function +(``__jit_debug_register_code``) marked noinline that GDB knows about. When +GDB attaches to a process, it puts a breakpoint in this function and loads all +of the object files in the global list. When MCJIT calls the registration +function, GDB catches the breakpoint signal, loads the new object file from +the inferior's memory, and resumes the execution. In this way, GDB can get the +necessary debug information. + +GDB Version +=========== + +In order to debug code JIT-ed by LLVM, you need GDB 7.0 or newer, which is +available on most modern distributions of Linux. The version of GDB that +Apple ships with Xcode has been frozen at 6.3 for a while. LLDB may be a +better option for debugging JIT-ed code on Mac OS X. + + +Debugging MCJIT-ed code +======================= + +The emerging MCJIT component of LLVM allows full debugging of JIT-ed code with +GDB. This is due to MCJIT's ability to use the MC emitter to provide full +DWARF debugging information to GDB. + +Note that lli has to be passed the ``-use-mcjit`` flag to JIT the code with +MCJIT instead of the old JIT. + +Example +------- + +Consider the following C code (with line numbers added to make the example +easier to follow): + +.. + FIXME: + Sphinx has the ability to automatically number these lines by adding + :linenos: on the line immediately following the `.. code-block:: c`, but + it looks like garbage; the line numbers don't even line up with the + lines. Is this a Sphinx bug, or is it a CSS problem? + +.. code-block:: c + + 1 int compute_factorial(int n) + 2 { + 3 if (n <= 1) + 4 return 1; + 5 + 6 int f = n; + 7 while (--n > 1) + 8 f *= n; + 9 return f; + 10 } + 11 + 12 + 13 int main(int argc, char** argv) + 14 { + 15 if (argc < 2) + 16 return -1; + 17 char firstletter = argv[1][0]; + 18 int result = compute_factorial(firstletter - '0'); + 19 + 20 // Returned result is clipped at 255... + 21 return result; + 22 } + +Here is a sample command line session that shows how to build and run this +code via ``lli`` inside GDB: + +.. code-block:: bash + + $ $BINPATH/clang -cc1 -O0 -g -emit-llvm showdebug.c + $ gdb --quiet --args $BINPATH/lli -use-mcjit showdebug.ll 5 + Reading symbols from $BINPATH/lli...done. + (gdb) b showdebug.c:6 + No source file named showdebug.c. + Make breakpoint pending on future shared library load? (y or [n]) y + Breakpoint 1 (showdebug.c:6) pending. + (gdb) r + Starting program: $BINPATH/lli -use-mcjit showdebug.ll 5 + [Thread debugging using libthread_db enabled] + + Breakpoint 1, compute_factorial (n=5) at showdebug.c:6 + 6 int f = n; + (gdb) p n + $1 = 5 + (gdb) p f + $2 = 0 + (gdb) n + 7 while (--n > 1) + (gdb) p f + $3 = 5 + (gdb) b showdebug.c:9 + Breakpoint 2 at 0x7ffff7ed404c: file showdebug.c, line 9. + (gdb) c + Continuing. + + Breakpoint 2, compute_factorial (n=1) at showdebug.c:9 + 9 return f; + (gdb) p f + $4 = 120 + (gdb) bt + #0 compute_factorial (n=1) at showdebug.c:9 + #1 0x00007ffff7ed40a9 in main (argc=2, argv=0x16677e0) at showdebug.c:18 + #2 0x3500000001652748 in ?? () + #3 0x00000000016677e0 in ?? () + #4 0x0000000000000002 in ?? () + #5 0x0000000000d953b3 in llvm::MCJIT::runFunction (this=0x16151f0, F=0x1603020, ArgValues=...) at /home/ebenders_test/llvm_svn_rw/lib/ExecutionEngine/MCJIT/MCJIT.cpp:161 + #6 0x0000000000dc8872 in llvm::ExecutionEngine::runFunctionAsMain (this=0x16151f0, Fn=0x1603020, argv=..., envp=0x7fffffffe040) + at /home/ebenders_test/llvm_svn_rw/lib/ExecutionEngine/ExecutionEngine.cpp:397 + #7 0x000000000059c583 in main (argc=4, argv=0x7fffffffe018, envp=0x7fffffffe040) at /home/ebenders_test/llvm_svn_rw/tools/lli/lli.cpp:324 + (gdb) finish + Run till exit from #0 compute_factorial (n=1) at showdebug.c:9 + 0x00007ffff7ed40a9 in main (argc=2, argv=0x16677e0) at showdebug.c:18 + 18 int result = compute_factorial(firstletter - '0'); + Value returned is $5 = 120 + (gdb) p result + $6 = 23406408 + (gdb) n + 21 return result; + (gdb) p result + $7 = 120 + (gdb) c + Continuing. + + Program exited with code 0170. + (gdb) diff --git a/docs/DeveloperPolicy.rst b/docs/DeveloperPolicy.rst index cda281a25c12..e35e72955640 100644 --- a/docs/DeveloperPolicy.rst +++ b/docs/DeveloperPolicy.rst @@ -137,6 +137,9 @@ reviewees. If someone is kind enough to review your code, you should return the favor for someone else. Note that anyone is welcome to review and give feedback on a patch, but only people with Subversion write access can approve it. +There is a web based code review tool that can optionally be used +for code reviews. See :doc:`Phabricator`. + Code Owners ----------- @@ -279,7 +282,7 @@ If you have recently been granted commit access, these policies apply: #. You are granted *commit-after-approval* to all parts of LLVM. To get approval, submit a `patch`_ to `llvm-commits
- Extending LLVM: Adding instructions, intrinsics, types, etc. -
- --
-
- Introduction and Warning -
- Adding a new intrinsic function -
- Adding a new instruction -
- Adding a new SelectionDAG node -
- Adding a new type - -
- Introduction and Warning -
- - -
-
-
-
-
-During the course of using LLVM, you may wish to customize it for your -research project or for experimentation. At this point, you may realize that -you need to add something to LLVM, whether it be a new fundamental type, a new -intrinsic function, or a whole new instruction.
- -When you come to this realization, stop and think. Do you really need to -extend LLVM? Is it a new fundamental capability that LLVM does not support at -its current incarnation or can it be synthesized from already pre-existing LLVM -elements? If you are not sure, ask on the LLVM-dev list. The -reason is that extending LLVM will get involved as you need to update all the -different passes that you intend to use with your extension, and there are -many LLVM analyses and transformations, so it may be quite a bit of -work.
- -Adding an intrinsic function is far easier than -adding an instruction, and is transparent to optimization passes. If your added -functionality can be expressed as a -function call, an intrinsic function is the method of choice for LLVM -extension.
- -Before you invest a significant amount of effort into a non-trivial -extension, ask on the list if what you are -looking to do can be done with already-existing infrastructure, or if maybe -someone else is already working on it. You will save yourself a lot of time and -effort by doing so.
- -- Adding a new intrinsic function -
- - -
-
-
-
-
-Adding a new intrinsic function to LLVM is much easier than adding a new -instruction. Almost all extensions to LLVM should start as an intrinsic -function and then be turned into an instruction if warranted.
- --
-
- llvm/docs/LangRef.html: - Document the intrinsic. Decide whether it is code generator specific and - what the restrictions are. Talk to other people about it so that you are - sure it's a good idea. - -
- llvm/include/llvm/Intrinsics*.td: - Add an entry for your intrinsic. Describe its memory access characteristics - for optimization (this controls whether it will be DCE'd, CSE'd, etc). Note - that any intrinsic using the llvm_int_ty type for an argument will - be deemed by tblgen as overloaded and the corresponding suffix - will be required on the intrinsic's name. - -
- llvm/lib/Analysis/ConstantFolding.cpp: If it is possible to - constant fold your intrinsic, add support to it in the - canConstantFoldCallTo and ConstantFoldCall functions. - -
- llvm/test/Regression/*: Add test cases for your test cases to the - test suite -
Once the intrinsic has been added to the system, you must add code generator -support for it. Generally you must do the following steps:
- --
-
-
- Add support to the .td file for the target(s) of your choice in - lib/Target/*/*.td. - -
- This is usually a matter of adding a pattern to the .td file that matches - the intrinsic, though it may obviously require adding the instructions you - want to generate as well. There are lots of examples in the PowerPC and X86 - backend to follow. -
- Adding a new SelectionDAG node -
- - -
-
-
-
-
-As with intrinsics, adding a new SelectionDAG node to LLVM is much easier -than adding a new instruction. New nodes are often added to help represent -instructions common to many targets. These nodes often map to an LLVM -instruction (add, sub) or intrinsic (byteswap, population count). In other -cases, new nodes have been added to allow many targets to perform a common task -(converting between floating point and integer representation) or capture more -complicated behavior in a single node (rotate).
- --
-
- include/llvm/CodeGen/ISDOpcodes.h: - Add an enum value for the new SelectionDAG node. -
- lib/CodeGen/SelectionDAG/SelectionDAG.cpp: - Add code to print the node to getOperationName. If your new node - can be evaluated at compile time when given constant arguments (such as an - add of a constant with another constant), find the getNode method - that takes the appropriate number of arguments, and add a case for your node - to the switch statement that performs constant folding for nodes that take - the same number of arguments as your new node. -
- lib/CodeGen/SelectionDAG/LegalizeDAG.cpp: - Add code to legalize, - promote, and expand the node as necessary. At a minimum, you will need - to add a case statement for your node in LegalizeOp which calls - LegalizeOp on the node's operands, and returns a new node if any of the - operands changed as a result of being legalized. It is likely that not all - targets supported by the SelectionDAG framework will natively support the - new node. In this case, you must also add code in your node's case - statement in LegalizeOp to Expand your node into simpler, legal - operations. The case for ISD::UREM for expanding a remainder into - a divide, multiply, and a subtract is a good example. -
- lib/CodeGen/SelectionDAG/LegalizeDAG.cpp: - If targets may support the new node being added only at certain sizes, you - will also need to add code to your node's case statement in - LegalizeOp to Promote your node's operands to a larger size, and - perform the correct operation. You will also need to add code to - PromoteOp to do this as well. For a good example, see - ISD::BSWAP, - which promotes its operand to a wider size, performs the byteswap, and then - shifts the correct bytes right to emulate the narrower byteswap in the - wider type. -
- lib/CodeGen/SelectionDAG/LegalizeDAG.cpp: - Add a case for your node in ExpandOp to teach the legalizer how to - perform the action represented by the new node on a value that has been - split into high and low halves. This case will be used to support your - node with a 64 bit operand on a 32 bit target. -
- lib/CodeGen/SelectionDAG/DAGCombiner.cpp: - If your node can be combined with itself, or other existing nodes in a - peephole-like fashion, add a visit function for it, and call that function - from . There are several good examples for simple combines you - can do; visitFABS and visitSRL are good starting places. - -
- lib/Target/PowerPC/PPCISelLowering.cpp: - Each target has an implementation of the TargetLowering class, - usually in its own file (although some targets include it in the same - file as the DAGToDAGISel). The default behavior for a target is to - assume that your new node is legal for all types that are legal for - that target. If this target does not natively support your node, then - tell the target to either Promote it (if it is supported at a larger - type) or Expand it. This will cause the code you wrote in - LegalizeOp above to decompose your new node into other legal - nodes for this target. -
- lib/Target/TargetSelectionDAG.td: - Most current targets supported by LLVM generate code using the DAGToDAG - method, where SelectionDAG nodes are pattern matched to target-specific - nodes, which represent individual instructions. In order for the targets - to match an instruction to your new node, you must add a def for that node - to the list in this file, with the appropriate type constraints. Look at - add, bswap, and fadd for examples. -
- lib/Target/PowerPC/PPCInstrInfo.td: - Each target has a tablegen file that describes the target's instruction - set. For targets that use the DAGToDAG instruction selection framework, - add a pattern for your new node that uses one or more target nodes. - Documentation for this is a bit sparse right now, but there are several - decent examples. See the patterns for rotl in - PPCInstrInfo.td. -
- TODO: document complex patterns. -
- llvm/test/Regression/CodeGen/*: Add test cases for your new node - to the test suite. llvm/test/Regression/CodeGen/X86/bswap.ll is - a good example. -
- Adding a new instruction -
- - -
-
-
-
-
-
-WARNING: adding instructions changes the bitcode -format, and it will take some effort to maintain compatibility with -the previous version. Only add an instruction if it is absolutely -necessary.
- --
-
-
- llvm/include/llvm/Instruction.def: - add a number for your instruction and an enum name - -
- llvm/include/llvm/Instructions.h: - add a definition for the class that will represent your instruction - -
- llvm/include/llvm/Support/InstVisitor.h: - add a prototype for a visitor to your new instruction type - -
- llvm/lib/AsmParser/Lexer.l: - add a new token to parse your instruction from assembly text file - -
- llvm/lib/AsmParser/llvmAsmParser.y: - add the grammar on how your instruction can be read and what it will - construct as a result - -
- llvm/lib/Bitcode/Reader/Reader.cpp: - add a case for your instruction and how it will be parsed from bitcode - -
- llvm/lib/VMCore/Instruction.cpp: - add a case for how your instruction will be printed out to assembly - -
- llvm/lib/VMCore/Instructions.cpp: - implement the class you defined in - llvm/include/llvm/Instructions.h - -
- Test your instruction - -
- llvm/lib/Target/*: - Add support for your instruction to code generators, or add a lowering - pass. - -
- llvm/test/Regression/*: add your test cases to the test suite. - -
Also, you need to implement (or modify) any analyses or passes that you want -to understand this new instruction.
- -- Adding a new type -
- - -
-
-
-
-
-
-WARNING: adding new types changes the bitcode -format, and will break compatibility with currently-existing LLVM -installations. Only add new types if it is absolutely necessary.
- - -- Adding a fundamental type -
- -
-
-
-
-
--
-
-
- llvm/include/llvm/Type.h: - add enum for the new type; add static Type* for this type - -
- llvm/lib/VMCore/Type.cpp: - add mapping from TypeID => Type*; - initialize the static Type* - -
- llvm/lib/AsmReader/Lexer.l: - add ability to parse in the type from text assembly - -
- llvm/lib/AsmReader/llvmAsmParser.y: - add a token for that type - -
- Adding a derived type -
- -
-
-
-
--
-
- llvm/include/llvm/Type.h: - add enum for the new type; add a forward declaration of the type - also - -
- llvm/include/llvm/DerivedTypes.h: - add new class to represent new class in the hierarchy; add forward - declaration to the TypeMap value type - -
- llvm/lib/VMCore/Type.cpp:
- add support for derived type to:
--- add necessary member functions for type, and factory methods
-std::string getTypeDescription(const Type &Ty, - std::vector<const Type*> &TypeStack) -bool TypesEqual(const Type *Ty, const Type *Ty2, - std::map<const Type*, const Type*> & EqTypes) -
-
-
- - llvm/lib/AsmReader/Lexer.l: - add ability to parse in the type from text assembly - -
- llvm/lib/BitCode/Writer/Writer.cpp: - modify void BitcodeWriter::outputType(const Type *T) to serialize - your type - -
- llvm/lib/BitCode/Reader/Reader.cpp: - modify const Type *BitcodeReader::ParseType() to read your data - type - -
- llvm/lib/VMCore/AsmWriter.cpp:
- modify
--- to output the new derived type -
-void calcTypeName(const Type *Ty, - std::vector<const Type*> &TypeStack, - std::map<const Type*,std::string> &TypeNames, - std::string & Result) -
-
-
-
-
- -
- - Last modified: $Date: 2012-04-19 22:20:34 +0200 (Thu, 19 Apr 2012) $ - - - - diff --git a/docs/ExtendingLLVM.rst b/docs/ExtendingLLVM.rst new file mode 100644 index 000000000000..6df08eee985a --- /dev/null +++ b/docs/ExtendingLLVM.rst @@ -0,0 +1,306 @@ +.. _extending_llvm: + +============================================================ +Extending LLVM: Adding instructions, intrinsics, types, etc. +============================================================ + +Introduction and Warning +======================== + + +During the course of using LLVM, you may wish to customize it for your research +project or for experimentation. At this point, you may realize that you need to +add something to LLVM, whether it be a new fundamental type, a new intrinsic +function, or a whole new instruction. + +When you come to this realization, stop and think. Do you really need to extend +LLVM? Is it a new fundamental capability that LLVM does not support at its +current incarnation or can it be synthesized from already pre-existing LLVM +elements? If you are not sure, ask on the `LLVM-dev +
LLVM Compiler Infrastructure
- Last modified: $Date: 2012-05-03 17:25:19 +0200 (Thu, 03 May 2012) $ + Last modified: $Date: 2012-10-08 18:39:34 +0200 (Mon, 08 Oct 2012) $