1 ============================
2 "Clang" CFE Internals Manual
3 ============================
11 This document describes some of the more important APIs and internal design
12 decisions made in the Clang C front-end. The purpose of this document is to
13 both capture some of this high level information and also describe some of the
14 design decisions behind it. This is meant for people interested in hacking on
15 Clang, not for end-users. The description below is categorized by libraries,
16 and does not describe any of the clients of the libraries.
21 The LLVM ``libSupport`` library provides many underlying libraries and
22 `data-structures <http://llvm.org/docs/ProgrammersManual.html>`_, including
23 command line option processing, various containers and a system abstraction
24 layer, which is used for file system access.
26 The Clang "Basic" Library
27 =========================
29 This library certainly needs a better name. The "basic" library contains a
30 number of low-level utilities for tracking and manipulating source buffers,
31 locations within the source buffers, diagnostics, tokens, target abstraction,
32 and information about the subset of the language being compiled for.
34 Part of this infrastructure is specific to C (such as the ``TargetInfo``
35 class), other parts could be reused for other non-C-based languages
36 (``SourceLocation``, ``SourceManager``, ``Diagnostics``, ``FileManager``).
37 When and if there is future demand we can figure out if it makes sense to
38 introduce a new library, move the general classes somewhere else, or introduce
41 We describe the roles of these classes in order of their dependencies.
43 The Diagnostics Subsystem
44 -------------------------
46 The Clang Diagnostics subsystem is an important part of how the compiler
47 communicates with the human. Diagnostics are the warnings and errors produced
48 when the code is incorrect or dubious. In Clang, each diagnostic produced has
49 (at the minimum) a unique ID, an English translation associated with it, a
50 :ref:`SourceLocation <SourceLocation>` to "put the caret", and a severity
51 (e.g., ``WARNING`` or ``ERROR``). They can also optionally include a number of
52 arguments to the dianostic (which fill in "%0"'s in the string) as well as a
53 number of source ranges that related to the diagnostic.
55 In this section, we'll be giving examples produced by the Clang command line
56 driver, but diagnostics can be :ref:`rendered in many different ways
57 <DiagnosticClient>` depending on how the ``DiagnosticClient`` interface is
58 implemented. A representative example of a diagnostic is:
62 t.c:38:15: error: invalid operands to binary expression ('int *' and '_Complex float')
66 In this example, you can see the English translation, the severity (error), you
67 can see the source location (the caret ("``^``") and file/line/column info),
68 the source ranges "``~~~~``", arguments to the diagnostic ("``int*``" and
69 "``_Complex float``"). You'll have to believe me that there is a unique ID
70 backing the diagnostic :).
72 Getting all of this to happen has several steps and involves many moving
73 pieces, this section describes them and talks about best practices when adding
76 The ``Diagnostic*Kinds.td`` files
77 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
79 Diagnostics are created by adding an entry to one of the
80 ``clang/Basic/Diagnostic*Kinds.td`` files, depending on what library will be
81 using it. From this file, :program:`tblgen` generates the unique ID of the
82 diagnostic, the severity of the diagnostic and the English translation + format
85 There is little sanity with the naming of the unique ID's right now. Some
86 start with ``err_``, ``warn_``, ``ext_`` to encode the severity into the name.
87 Since the enum is referenced in the C++ code that produces the diagnostic, it
88 is somewhat useful for it to be reasonably short.
90 The severity of the diagnostic comes from the set {``NOTE``, ``REMARK``,
92 ``EXTENSION``, ``EXTWARN``, ``ERROR``}. The ``ERROR`` severity is used for
93 diagnostics indicating the program is never acceptable under any circumstances.
94 When an error is emitted, the AST for the input code may not be fully built.
95 The ``EXTENSION`` and ``EXTWARN`` severities are used for extensions to the
96 language that Clang accepts. This means that Clang fully understands and can
97 represent them in the AST, but we produce diagnostics to tell the user their
98 code is non-portable. The difference is that the former are ignored by
99 default, and the later warn by default. The ``WARNING`` severity is used for
100 constructs that are valid in the currently selected source language but that
101 are dubious in some way. The ``REMARK`` severity provides generic information
102 about the compilation that is not necessarily related to any dubious code. The
103 ``NOTE`` level is used to staple more information onto previous diagnostics.
105 These *severities* are mapped into a smaller set (the ``Diagnostic::Level``
106 enum, {``Ignored``, ``Note``, ``Remark``, ``Warning``, ``Error``, ``Fatal``}) of
108 *levels* by the diagnostics subsystem based on various configuration options.
109 Clang internally supports a fully fine grained mapping mechanism that allows
110 you to map almost any diagnostic to the output level that you want. The only
111 diagnostics that cannot be mapped are ``NOTE``\ s, which always follow the
112 severity of the previously emitted diagnostic and ``ERROR``\ s, which can only
113 be mapped to ``Fatal`` (it is not possible to turn an error into a warning, for
116 Diagnostic mappings are used in many ways. For example, if the user specifies
117 ``-pedantic``, ``EXTENSION`` maps to ``Warning``, if they specify
118 ``-pedantic-errors``, it turns into ``Error``. This is used to implement
119 options like ``-Wunused_macros``, ``-Wundef`` etc.
121 Mapping to ``Fatal`` should only be used for diagnostics that are considered so
122 severe that error recovery won't be able to recover sensibly from them (thus
123 spewing a ton of bogus errors). One example of this class of error are failure
124 to ``#include`` a file.
129 The format string for the diagnostic is very simple, but it has some power. It
130 takes the form of a string in English with markers that indicate where and how
131 arguments to the diagnostic are inserted and formatted. For example, here are
132 some simple format strings:
136 "binary integer literals are an extension"
137 "format string contains '\\0' within the string body"
138 "more '%%' conversions than data arguments"
139 "invalid operands to binary expression (%0 and %1)"
140 "overloaded '%0' must be a %select{unary|binary|unary or binary}2 operator"
141 " (has %1 parameter%s1)"
143 These examples show some important points of format strings. You can use any
144 plain ASCII character in the diagnostic string except "``%``" without a
145 problem, but these are C strings, so you have to use and be aware of all the C
146 escape sequences (as in the second example). If you want to produce a "``%``"
147 in the output, use the "``%%``" escape sequence, like the third diagnostic.
148 Finally, Clang uses the "``%...[digit]``" sequences to specify where and how
149 arguments to the diagnostic are formatted.
151 Arguments to the diagnostic are numbered according to how they are specified by
152 the C++ code that :ref:`produces them <internals-producing-diag>`, and are
153 referenced by ``%0`` .. ``%9``. If you have more than 10 arguments to your
154 diagnostic, you are doing something wrong :). Unlike ``printf``, there is no
155 requirement that arguments to the diagnostic end up in the output in the same
156 order as they are specified, you could have a format string with "``%1 %0``"
157 that swaps them, for example. The text in between the percent and digit are
158 formatting instructions. If there are no instructions, the argument is just
159 turned into a string and substituted in.
161 Here are some "best practices" for writing the English format string:
163 * Keep the string short. It should ideally fit in the 80 column limit of the
164 ``DiagnosticKinds.td`` file. This avoids the diagnostic wrapping when
165 printed, and forces you to think about the important point you are conveying
167 * Take advantage of location information. The user will be able to see the
168 line and location of the caret, so you don't need to tell them that the
169 problem is with the 4th argument to the function: just point to it.
170 * Do not capitalize the diagnostic string, and do not end it with a period.
171 * If you need to quote something in the diagnostic string, use single quotes.
173 Diagnostics should never take random English strings as arguments: you
174 shouldn't use "``you have a problem with %0``" and pass in things like "``your
175 argument``" or "``your return value``" as arguments. Doing this prevents
176 :ref:`translating <internals-diag-translation>` the Clang diagnostics to other
177 languages (because they'll get random English words in their otherwise
178 localized diagnostic). The exceptions to this are C/C++ language keywords
179 (e.g., ``auto``, ``const``, ``mutable``, etc) and C/C++ operators (``/=``).
180 Note that things like "pointer" and "reference" are not keywords. On the other
181 hand, you *can* include anything that comes from the user's source code,
182 including variable names, types, labels, etc. The "``select``" format can be
183 used to achieve this sort of thing in a localizable way, see below.
185 Formatting a Diagnostic Argument
186 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
188 Arguments to diagnostics are fully typed internally, and come from a couple
189 different classes: integers, types, names, and random strings. Depending on
190 the class of the argument, it can be optionally formatted in different ways.
191 This gives the ``DiagnosticClient`` information about what the argument means
192 without requiring it to use a specific presentation (consider this MVC for
195 Here are the different diagnostic argument formats currently supported by
201 ``"requires %1 parameter%s1"``
205 This is a simple formatter for integers that is useful when producing English
206 diagnostics. When the integer is 1, it prints as nothing. When the integer
207 is not 1, it prints as "``s``". This allows some simple grammatical forms to
208 be to be handled correctly, and eliminates the need to use gross things like
209 ``"requires %1 parameter(s)"``.
214 ``"must be a %select{unary|binary|unary or binary}2 operator"``
218 This format specifier is used to merge multiple related diagnostics together
219 into one common one, without requiring the difference to be specified as an
220 English string argument. Instead of specifying the string, the diagnostic
221 gets an integer argument and the format string selects the numbered option.
222 In this case, the "``%2``" value must be an integer in the range [0..2]. If
223 it is 0, it prints "unary", if it is 1 it prints "binary" if it is 2, it
224 prints "unary or binary". This allows other language translations to
225 substitute reasonable words (or entire phrases) based on the semantics of the
226 diagnostic instead of having to do things textually. The selected string
227 does undergo formatting.
232 ``"you have %1 %plural{1:mouse|:mice}1 connected to your computer"``
236 This is a formatter for complex plural forms. It is designed to handle even
237 the requirements of languages with very complex plural forms, as many Baltic
238 languages have. The argument consists of a series of expression/form pairs,
239 separated by ":", where the first form whose expression evaluates to true is
240 the result of the modifier.
242 An expression can be empty, in which case it is always true. See the example
243 at the top. Otherwise, it is a series of one or more numeric conditions,
244 separated by ",". If any condition matches, the expression matches. Each
245 numeric condition can take one of three forms.
247 * number: A simple decimal number matches if the argument is the same as the
248 number. Example: ``"%plural{1:mouse|:mice}4"``
249 * range: A range in square brackets matches if the argument is within the
250 range. Then range is inclusive on both ends. Example:
251 ``"%plural{0:none|1:one|[2,5]:some|:many}2"``
252 * modulo: A modulo operator is followed by a number, and equals sign and
253 either a number or a range. The tests are the same as for plain numbers
254 and ranges, but the argument is taken modulo the number first. Example:
255 ``"%plural{%100=0:even hundred|%100=[1,50]:lower half|:everything else}1"``
257 The parser is very unforgiving. A syntax error, even whitespace, will abort,
258 as will a failure to match the argument against any expression.
263 ``"ambiguity in %ordinal0 argument"``
267 This is a formatter which represents the argument number as an ordinal: the
268 value ``1`` becomes ``1st``, ``3`` becomes ``3rd``, and so on. Values less
269 than ``1`` are not supported. This formatter is currently hard-coded to use
272 **"objcclass" format**
275 ``"method %objcclass0 not found"``
279 This is a simple formatter that indicates the ``DeclarationName`` corresponds
280 to an Objective-C class method selector. As such, it prints the selector
281 with a leading "``+``".
283 **"objcinstance" format**
286 ``"method %objcinstance0 not found"``
290 This is a simple formatter that indicates the ``DeclarationName`` corresponds
291 to an Objective-C instance method selector. As such, it prints the selector
292 with a leading "``-``".
297 ``"candidate found by name lookup is %q0"``
301 This formatter indicates that the fully-qualified name of the declaration
302 should be printed, e.g., "``std::vector``" rather than "``vector``".
307 ``"no known conversion %diff{from $ to $|from argument type to parameter type}1,2"``
311 This formatter takes two ``QualType``\ s and attempts to print a template
312 difference between the two. If tree printing is off, the text inside the
313 braces before the pipe is printed, with the formatted text replacing the $.
314 If tree printing is on, the text after the pipe is printed and a type tree is
315 printed after the diagnostic message.
317 It is really easy to add format specifiers to the Clang diagnostics system, but
318 they should be discussed before they are added. If you are creating a lot of
319 repetitive diagnostics and/or have an idea for a useful formatter, please bring
320 it up on the cfe-dev mailing list.
322 .. _internals-producing-diag:
324 Producing the Diagnostic
325 ^^^^^^^^^^^^^^^^^^^^^^^^
327 Now that you've created the diagnostic in the ``Diagnostic*Kinds.td`` file, you
328 need to write the code that detects the condition in question and emits the new
329 diagnostic. Various components of Clang (e.g., the preprocessor, ``Sema``,
330 etc.) provide a helper function named "``Diag``". It creates a diagnostic and
331 accepts the arguments, ranges, and other information that goes along with it.
333 For example, the binary expression error comes from code like this:
337 if (various things that are bad)
338 Diag(Loc, diag::err_typecheck_invalid_operands)
339 << lex->getType() << rex->getType()
340 << lex->getSourceRange() << rex->getSourceRange();
342 This shows that use of the ``Diag`` method: it takes a location (a
343 :ref:`SourceLocation <SourceLocation>` object) and a diagnostic enum value
344 (which matches the name from ``Diagnostic*Kinds.td``). If the diagnostic takes
345 arguments, they are specified with the ``<<`` operator: the first argument
346 becomes ``%0``, the second becomes ``%1``, etc. The diagnostic interface
347 allows you to specify arguments of many different types, including ``int`` and
348 ``unsigned`` for integer arguments, ``const char*`` and ``std::string`` for
349 string arguments, ``DeclarationName`` and ``const IdentifierInfo *`` for names,
350 ``QualType`` for types, etc. ``SourceRange``\ s are also specified with the
351 ``<<`` operator, but do not have a specific ordering requirement.
353 As you can see, adding and producing a diagnostic is pretty straightforward.
354 The hard part is deciding exactly what you need to say to help the user,
355 picking a suitable wording, and providing the information needed to format it
356 correctly. The good news is that the call site that issues a diagnostic should
357 be completely independent of how the diagnostic is formatted and in what
358 language it is rendered.
363 In some cases, the front end emits diagnostics when it is clear that some small
364 change to the source code would fix the problem. For example, a missing
365 semicolon at the end of a statement or a use of deprecated syntax that is
366 easily rewritten into a more modern form. Clang tries very hard to emit the
367 diagnostic and recover gracefully in these and other cases.
369 However, for these cases where the fix is obvious, the diagnostic can be
370 annotated with a hint (referred to as a "fix-it hint") that describes how to
371 change the code referenced by the diagnostic to fix the problem. For example,
372 it might add the missing semicolon at the end of the statement or rewrite the
373 use of a deprecated construct into something more palatable. Here is one such
374 example from the C++ front end, where we warn about the right-shift operator
375 changing meaning from C++98 to C++11:
379 test.cpp:3:7: warning: use of right-shift operator ('>>') in template argument
380 will require parentheses in C++11
385 Here, the fix-it hint is suggesting that parentheses be added, and showing
386 exactly where those parentheses would be inserted into the source code. The
387 fix-it hints themselves describe what changes to make to the source code in an
388 abstract manner, which the text diagnostic printer renders as a line of
389 "insertions" below the caret line. :ref:`Other diagnostic clients
390 <DiagnosticClient>` might choose to render the code differently (e.g., as
391 markup inline) or even give the user the ability to automatically fix the
394 Fix-it hints on errors and warnings need to obey these rules:
396 * Since they are automatically applied if ``-Xclang -fixit`` is passed to the
397 driver, they should only be used when it's very likely they match the user's
399 * Clang must recover from errors as if the fix-it had been applied.
401 If a fix-it can't obey these rules, put the fix-it on a note. Fix-its on notes
402 are not applied automatically.
404 All fix-it hints are described by the ``FixItHint`` class, instances of which
405 should be attached to the diagnostic using the ``<<`` operator in the same way
406 that highlighted source ranges and arguments are passed to the diagnostic.
407 Fix-it hints can be created with one of three constructors:
409 * ``FixItHint::CreateInsertion(Loc, Code)``
411 Specifies that the given ``Code`` (a string) should be inserted before the
412 source location ``Loc``.
414 * ``FixItHint::CreateRemoval(Range)``
416 Specifies that the code in the given source ``Range`` should be removed.
418 * ``FixItHint::CreateReplacement(Range, Code)``
420 Specifies that the code in the given source ``Range`` should be removed,
421 and replaced with the given ``Code`` string.
423 .. _DiagnosticClient:
425 The ``DiagnosticClient`` Interface
426 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
428 Once code generates a diagnostic with all of the arguments and the rest of the
429 relevant information, Clang needs to know what to do with it. As previously
430 mentioned, the diagnostic machinery goes through some filtering to map a
431 severity onto a diagnostic level, then (assuming the diagnostic is not mapped
432 to "``Ignore``") it invokes an object that implements the ``DiagnosticClient``
433 interface with the information.
435 It is possible to implement this interface in many different ways. For
436 example, the normal Clang ``DiagnosticClient`` (named
437 ``TextDiagnosticPrinter``) turns the arguments into strings (according to the
438 various formatting rules), prints out the file/line/column information and the
439 string, then prints out the line of code, the source ranges, and the caret.
440 However, this behavior isn't required.
442 Another implementation of the ``DiagnosticClient`` interface is the
443 ``TextDiagnosticBuffer`` class, which is used when Clang is in ``-verify``
444 mode. Instead of formatting and printing out the diagnostics, this
445 implementation just captures and remembers the diagnostics as they fly by.
446 Then ``-verify`` compares the list of produced diagnostics to the list of
447 expected ones. If they disagree, it prints out its own output. Full
448 documentation for the ``-verify`` mode can be found in the Clang API
449 documentation for `VerifyDiagnosticConsumer
450 </doxygen/classclang_1_1VerifyDiagnosticConsumer.html#details>`_.
452 There are many other possible implementations of this interface, and this is
453 why we prefer diagnostics to pass down rich structured information in
454 arguments. For example, an HTML output might want declaration names be
455 linkified to where they come from in the source. Another example is that a GUI
456 might let you click on typedefs to expand them. This application would want to
457 pass significantly more information about types through to the GUI than a
458 simple flat string. The interface allows this to happen.
460 .. _internals-diag-translation:
462 Adding Translations to Clang
463 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
465 Not possible yet! Diagnostic strings should be written in UTF-8, the client can
466 translate to the relevant code page if needed. Each translation completely
467 replaces the format string for the diagnostic.
472 The ``SourceLocation`` and ``SourceManager`` classes
473 ----------------------------------------------------
475 Strangely enough, the ``SourceLocation`` class represents a location within the
476 source code of the program. Important design points include:
478 #. ``sizeof(SourceLocation)`` must be extremely small, as these are embedded
479 into many AST nodes and are passed around often. Currently it is 32 bits.
480 #. ``SourceLocation`` must be a simple value object that can be efficiently
482 #. We should be able to represent a source location for any byte of any input
483 file. This includes in the middle of tokens, in whitespace, in trigraphs,
485 #. A ``SourceLocation`` must encode the current ``#include`` stack that was
486 active when the location was processed. For example, if the location
487 corresponds to a token, it should contain the set of ``#include``\ s active
488 when the token was lexed. This allows us to print the ``#include`` stack
490 #. ``SourceLocation`` must be able to describe macro expansions, capturing both
491 the ultimate instantiation point and the source of the original character
494 In practice, the ``SourceLocation`` works together with the ``SourceManager``
495 class to encode two pieces of information about a location: its spelling
496 location and its instantiation location. For most tokens, these will be the
497 same. However, for a macro expansion (or tokens that came from a ``_Pragma``
498 directive) these will describe the location of the characters corresponding to
499 the token and the location where the token was used (i.e., the macro
500 instantiation point or the location of the ``_Pragma`` itself).
502 The Clang front-end inherently depends on the location of a token being tracked
503 correctly. If it is ever incorrect, the front-end may get confused and die.
504 The reason for this is that the notion of the "spelling" of a ``Token`` in
505 Clang depends on being able to find the original input characters for the
506 token. This concept maps directly to the "spelling location" for the token.
508 ``SourceRange`` and ``CharSourceRange``
509 ---------------------------------------
511 .. mostly taken from http://lists.cs.uiuc.edu/pipermail/cfe-dev/2010-August/010595.html
513 Clang represents most source ranges by [first, last], where "first" and "last"
514 each point to the beginning of their respective tokens. For example consider
515 the ``SourceRange`` of the following statement:
522 To map from this representation to a character-based representation, the "last"
523 location needs to be adjusted to point to (or past) the end of that token with
524 either ``Lexer::MeasureTokenLength()`` or ``Lexer::getLocForEndOfToken()``. For
525 the rare cases where character-level source ranges information is needed we use
526 the ``CharSourceRange`` class.
531 The clang Driver and library are documented :doc:`here <DriverInternals>`.
536 Clang supports two implementations of precompiled headers. The default
537 implementation, precompiled headers (:doc:`PCH <PCHInternals>`) uses a
538 serialized representation of Clang's internal data structures, encoded with the
539 `LLVM bitstream format <http://llvm.org/docs/BitCodeFormat.html>`_.
540 Pretokenized headers (:doc:`PTH <PTHInternals>`), on the other hand, contain a
541 serialized representation of the tokens encountered when preprocessing a header
542 (and anything that header includes).
547 The Frontend library contains functionality useful for building tools on top of
548 the Clang libraries, for example several methods for outputting diagnostics.
550 The Lexer and Preprocessor Library
551 ==================================
553 The Lexer library contains several tightly-connected classes that are involved
554 with the nasty process of lexing and preprocessing C source code. The main
555 interface to this library for outside clients is the large ``Preprocessor``
556 class. It contains the various pieces of state that are required to coherently
557 read tokens out of a translation unit.
559 The core interface to the ``Preprocessor`` object (once it is set up) is the
560 ``Preprocessor::Lex`` method, which returns the next :ref:`Token <Token>` from
561 the preprocessor stream. There are two types of token providers that the
562 preprocessor is capable of reading from: a buffer lexer (provided by the
563 :ref:`Lexer <Lexer>` class) and a buffered token stream (provided by the
564 :ref:`TokenLexer <TokenLexer>` class).
571 The ``Token`` class is used to represent a single lexed token. Tokens are
572 intended to be used by the lexer/preprocess and parser libraries, but are not
573 intended to live beyond them (for example, they should not live in the ASTs).
575 Tokens most often live on the stack (or some other location that is efficient
576 to access) as the parser is running, but occasionally do get buffered up. For
577 example, macro definitions are stored as a series of tokens, and the C++
578 front-end periodically needs to buffer tokens up for tentative parsing and
579 various pieces of look-ahead. As such, the size of a ``Token`` matters. On a
580 32-bit system, ``sizeof(Token)`` is currently 16 bytes.
582 Tokens occur in two forms: :ref:`annotation tokens <AnnotationToken>` and
583 normal tokens. Normal tokens are those returned by the lexer, annotation
584 tokens represent semantic information and are produced by the parser, replacing
585 normal tokens in the token stream. Normal tokens contain the following
588 * **A SourceLocation** --- This indicates the location of the start of the
591 * **A length** --- This stores the length of the token as stored in the
592 ``SourceBuffer``. For tokens that include them, this length includes
593 trigraphs and escaped newlines which are ignored by later phases of the
594 compiler. By pointing into the original source buffer, it is always possible
595 to get the original spelling of a token completely accurately.
597 * **IdentifierInfo** --- If a token takes the form of an identifier, and if
598 identifier lookup was enabled when the token was lexed (e.g., the lexer was
599 not reading in "raw" mode) this contains a pointer to the unique hash value
600 for the identifier. Because the lookup happens before keyword
601 identification, this field is set even for language keywords like "``for``".
603 * **TokenKind** --- This indicates the kind of token as classified by the
604 lexer. This includes things like ``tok::starequal`` (for the "``*=``"
605 operator), ``tok::ampamp`` for the "``&&``" token, and keyword values (e.g.,
606 ``tok::kw_for``) for identifiers that correspond to keywords. Note that
607 some tokens can be spelled multiple ways. For example, C++ supports
608 "operator keywords", where things like "``and``" are treated exactly like the
609 "``&&``" operator. In these cases, the kind value is set to ``tok::ampamp``,
610 which is good for the parser, which doesn't have to consider both forms. For
611 something that cares about which form is used (e.g., the preprocessor
612 "stringize" operator) the spelling indicates the original form.
614 * **Flags** --- There are currently four flags tracked by the
615 lexer/preprocessor system on a per-token basis:
617 #. **StartOfLine** --- This was the first token that occurred on its input
619 #. **LeadingSpace** --- There was a space character either immediately before
620 the token or transitively before the token as it was expanded through a
621 macro. The definition of this flag is very closely defined by the
622 stringizing requirements of the preprocessor.
623 #. **DisableExpand** --- This flag is used internally to the preprocessor to
624 represent identifier tokens which have macro expansion disabled. This
625 prevents them from being considered as candidates for macro expansion ever
627 #. **NeedsCleaning** --- This flag is set if the original spelling for the
628 token includes a trigraph or escaped newline. Since this is uncommon,
629 many pieces of code can fast-path on tokens that did not need cleaning.
631 One interesting (and somewhat unusual) aspect of normal tokens is that they
632 don't contain any semantic information about the lexed value. For example, if
633 the token was a pp-number token, we do not represent the value of the number
634 that was lexed (this is left for later pieces of code to decide).
635 Additionally, the lexer library has no notion of typedef names vs variable
636 names: both are returned as identifiers, and the parser is left to decide
637 whether a specific identifier is a typedef or a variable (tracking this
638 requires scope information among other things). The parser can do this
639 translation by replacing tokens returned by the preprocessor with "Annotation
647 Annotation tokens are tokens that are synthesized by the parser and injected
648 into the preprocessor's token stream (replacing existing tokens) to record
649 semantic information found by the parser. For example, if "``foo``" is found
650 to be a typedef, the "``foo``" ``tok::identifier`` token is replaced with an
651 ``tok::annot_typename``. This is useful for a couple of reasons: 1) this makes
652 it easy to handle qualified type names (e.g., "``foo::bar::baz<42>::t``") in
653 C++ as a single "token" in the parser. 2) if the parser backtracks, the
654 reparse does not need to redo semantic analysis to determine whether a token
655 sequence is a variable, type, template, etc.
657 Annotation tokens are created by the parser and reinjected into the parser's
658 token stream (when backtracking is enabled). Because they can only exist in
659 tokens that the preprocessor-proper is done with, it doesn't need to keep
660 around flags like "start of line" that the preprocessor uses to do its job.
661 Additionally, an annotation token may "cover" a sequence of preprocessor tokens
662 (e.g., "``a::b::c``" is five preprocessor tokens). As such, the valid fields
663 of an annotation token are different than the fields for a normal token (but
664 they are multiplexed into the normal ``Token`` fields):
666 * **SourceLocation "Location"** --- The ``SourceLocation`` for the annotation
667 token indicates the first token replaced by the annotation token. In the
668 example above, it would be the location of the "``a``" identifier.
669 * **SourceLocation "AnnotationEndLoc"** --- This holds the location of the last
670 token replaced with the annotation token. In the example above, it would be
671 the location of the "``c``" identifier.
672 * **void* "AnnotationValue"** --- This contains an opaque object that the
673 parser gets from ``Sema``. The parser merely preserves the information for
674 ``Sema`` to later interpret based on the annotation token kind.
675 * **TokenKind "Kind"** --- This indicates the kind of Annotation token this is.
676 See below for the different valid kinds.
678 Annotation tokens currently come in three kinds:
680 #. **tok::annot_typename**: This annotation token represents a resolved
681 typename token that is potentially qualified. The ``AnnotationValue`` field
682 contains the ``QualType`` returned by ``Sema::getTypeName()``, possibly with
683 source location information attached.
684 #. **tok::annot_cxxscope**: This annotation token represents a C++ scope
685 specifier, such as "``A::B::``". This corresponds to the grammar
686 productions "*::*" and "*:: [opt] nested-name-specifier*". The
687 ``AnnotationValue`` pointer is a ``NestedNameSpecifier *`` returned by the
688 ``Sema::ActOnCXXGlobalScopeSpecifier`` and
689 ``Sema::ActOnCXXNestedNameSpecifier`` callbacks.
690 #. **tok::annot_template_id**: This annotation token represents a C++
691 template-id such as "``foo<int, 4>``", where "``foo``" is the name of a
692 template. The ``AnnotationValue`` pointer is a pointer to a ``malloc``'d
693 ``TemplateIdAnnotation`` object. Depending on the context, a parsed
694 template-id that names a type might become a typename annotation token (if
695 all we care about is the named type, e.g., because it occurs in a type
696 specifier) or might remain a template-id token (if we want to retain more
697 source location information or produce a new type, e.g., in a declaration of
698 a class template specialization). template-id annotation tokens that refer
699 to a type can be "upgraded" to typename annotation tokens by the parser.
701 As mentioned above, annotation tokens are not returned by the preprocessor,
702 they are formed on demand by the parser. This means that the parser has to be
703 aware of cases where an annotation could occur and form it where appropriate.
704 This is somewhat similar to how the parser handles Translation Phase 6 of C99:
705 String Concatenation (see C99 5.1.1.2). In the case of string concatenation,
706 the preprocessor just returns distinct ``tok::string_literal`` and
707 ``tok::wide_string_literal`` tokens and the parser eats a sequence of them
708 wherever the grammar indicates that a string literal can occur.
710 In order to do this, whenever the parser expects a ``tok::identifier`` or
711 ``tok::coloncolon``, it should call the ``TryAnnotateTypeOrScopeToken`` or
712 ``TryAnnotateCXXScopeToken`` methods to form the annotation token. These
713 methods will maximally form the specified annotation tokens and replace the
714 current token with them, if applicable. If the current tokens is not valid for
715 an annotation token, it will remain an identifier or "``::``" token.
722 The ``Lexer`` class provides the mechanics of lexing tokens out of a source
723 buffer and deciding what they mean. The ``Lexer`` is complicated by the fact
724 that it operates on raw buffers that have not had spelling eliminated (this is
725 a necessity to get decent performance), but this is countered with careful
726 coding as well as standard performance techniques (for example, the comment
727 handling code is vectorized on X86 and PowerPC hosts).
729 The lexer has a couple of interesting modal features:
731 * The lexer can operate in "raw" mode. This mode has several features that
732 make it possible to quickly lex the file (e.g., it stops identifier lookup,
733 doesn't specially handle preprocessor tokens, handles EOF differently, etc).
734 This mode is used for lexing within an "``#if 0``" block, for example.
735 * The lexer can capture and return comments as tokens. This is required to
736 support the ``-C`` preprocessor mode, which passes comments through, and is
737 used by the diagnostic checker to identifier expect-error annotations.
738 * The lexer can be in ``ParsingFilename`` mode, which happens when
739 preprocessing after reading a ``#include`` directive. This mode changes the
740 parsing of "``<``" to return an "angled string" instead of a bunch of tokens
741 for each thing within the filename.
742 * When parsing a preprocessor directive (after "``#``") the
743 ``ParsingPreprocessorDirective`` mode is entered. This changes the parser to
744 return EOD at a newline.
745 * The ``Lexer`` uses a ``LangOptions`` object to know whether trigraphs are
746 enabled, whether C++ or ObjC keywords are recognized, etc.
748 In addition to these modes, the lexer keeps track of a couple of other features
749 that are local to a lexed buffer, which change as the buffer is lexed:
751 * The ``Lexer`` uses ``BufferPtr`` to keep track of the current character being
753 * The ``Lexer`` uses ``IsAtStartOfLine`` to keep track of whether the next
754 lexed token will start with its "start of line" bit set.
755 * The ``Lexer`` keeps track of the current "``#if``" directives that are active
756 (which can be nested).
757 * The ``Lexer`` keeps track of an :ref:`MultipleIncludeOpt
758 <MultipleIncludeOpt>` object, which is used to detect whether the buffer uses
759 the standard "``#ifndef XX`` / ``#define XX``" idiom to prevent multiple
760 inclusion. If a buffer does, subsequent includes can be ignored if the
761 "``XX``" macro is defined.
765 The ``TokenLexer`` class
766 ------------------------
768 The ``TokenLexer`` class is a token provider that returns tokens from a list of
769 tokens that came from somewhere else. It typically used for two things: 1)
770 returning tokens from a macro definition as it is being expanded 2) returning
771 tokens from an arbitrary buffer of tokens. The later use is used by
772 ``_Pragma`` and will most likely be used to handle unbounded look-ahead for the
775 .. _MultipleIncludeOpt:
777 The ``MultipleIncludeOpt`` class
778 --------------------------------
780 The ``MultipleIncludeOpt`` class implements a really simple little state
781 machine that is used to detect the standard "``#ifndef XX`` / ``#define XX``"
782 idiom that people typically use to prevent multiple inclusion of headers. If a
783 buffer uses this idiom and is subsequently ``#include``'d, the preprocessor can
784 simply check to see whether the guarding condition is defined or not. If so,
785 the preprocessor can completely ignore the include of the header.
792 This library contains a recursive-descent parser that polls tokens from the
793 preprocessor and notifies a client of the parsing progress.
795 Historically, the parser used to talk to an abstract ``Action`` interface that
796 had virtual methods for parse events, for example ``ActOnBinOp()``. When Clang
797 grew C++ support, the parser stopped supporting general ``Action`` clients --
798 it now always talks to the :ref:`Sema libray <Sema>`. However, the Parser
799 still accesses AST objects only through opaque types like ``ExprResult`` and
800 ``StmtResult``. Only :ref:`Sema <Sema>` looks at the AST node contents of these
810 The ``Type`` class and its subclasses
811 -------------------------------------
813 The ``Type`` class (and its subclasses) are an important part of the AST.
814 Types are accessed through the ``ASTContext`` class, which implicitly creates
815 and uniques them as they are needed. Types have a couple of non-obvious
816 features: 1) they do not capture type qualifiers like ``const`` or ``volatile``
817 (see :ref:`QualType <QualType>`), and 2) they implicitly capture typedef
818 information. Once created, types are immutable (unlike decls).
820 Typedefs in C make semantic analysis a bit more complex than it would be without
821 them. The issue is that we want to capture typedef information and represent it
822 in the AST perfectly, but the semantics of operations need to "see through"
823 typedefs. For example, consider this code:
837 The code above is illegal, and thus we expect there to be diagnostics emitted
838 on the annotated lines. In this example, we expect to get:
842 test.c:6:1: error: indirection requires pointer operand ('foo' invalid)
845 test.c:7:1: error: indirection requires pointer operand ('foo' invalid)
848 test.c:8:1: error: indirection requires pointer operand ('foo' invalid)
852 While this example is somewhat silly, it illustrates the point: we want to
853 retain typedef information where possible, so that we can emit errors about
854 "``std::string``" instead of "``std::basic_string<char, std:...``". Doing this
855 requires properly keeping typedef information (for example, the type of ``X``
856 is "``foo``", not "``int``"), and requires properly propagating it through the
857 various operators (for example, the type of ``*Y`` is "``foo``", not
858 "``int``"). In order to retain this information, the type of these expressions
859 is an instance of the ``TypedefType`` class, which indicates that the type of
860 these expressions is a typedef for "``foo``".
862 Representing types like this is great for diagnostics, because the
863 user-specified type is always immediately available. There are two problems
864 with this: first, various semantic checks need to make judgements about the
865 *actual structure* of a type, ignoring typedefs. Second, we need an efficient
866 way to query whether two types are structurally identical to each other,
867 ignoring typedefs. The solution to both of these problems is the idea of
873 Every instance of the ``Type`` class contains a canonical type pointer. For
874 simple types with no typedefs involved (e.g., "``int``", "``int*``",
875 "``int**``"), the type just points to itself. For types that have a typedef
876 somewhere in their structure (e.g., "``foo``", "``foo*``", "``foo**``",
877 "``bar``"), the canonical type pointer points to their structurally equivalent
878 type without any typedefs (e.g., "``int``", "``int*``", "``int**``", and
879 "``int*``" respectively).
881 This design provides a constant time operation (dereferencing the canonical type
882 pointer) that gives us access to the structure of types. For example, we can
883 trivially tell that "``bar``" and "``foo*``" are the same type by dereferencing
884 their canonical type pointers and doing a pointer comparison (they both point
885 to the single "``int*``" type).
887 Canonical types and typedef types bring up some complexities that must be
888 carefully managed. Specifically, the ``isa``/``cast``/``dyn_cast`` operators
889 generally shouldn't be used in code that is inspecting the AST. For example,
890 when type checking the indirection operator (unary "``*``" on a pointer), the
891 type checker must verify that the operand has a pointer type. It would not be
892 correct to check that with "``isa<PointerType>(SubExpr->getType())``", because
893 this predicate would fail if the subexpression had a typedef type.
895 The solution to this problem are a set of helper methods on ``Type``, used to
896 check their properties. In this case, it would be correct to use
897 "``SubExpr->getType()->isPointerType()``" to do the check. This predicate will
898 return true if the *canonical type is a pointer*, which is true any time the
899 type is structurally a pointer type. The only hard part here is remembering
900 not to use the ``isa``/``cast``/``dyn_cast`` operations.
902 The second problem we face is how to get access to the pointer type once we
903 know it exists. To continue the example, the result type of the indirection
904 operator is the pointee type of the subexpression. In order to determine the
905 type, we need to get the instance of ``PointerType`` that best captures the
906 typedef information in the program. If the type of the expression is literally
907 a ``PointerType``, we can return that, otherwise we have to dig through the
908 typedefs to find the pointer type. For example, if the subexpression had type
909 "``foo*``", we could return that type as the result. If the subexpression had
910 type "``bar``", we want to return "``foo*``" (note that we do *not* want
911 "``int*``"). In order to provide all of this, ``Type`` has a
912 ``getAsPointerType()`` method that checks whether the type is structurally a
913 ``PointerType`` and, if so, returns the best one. If not, it returns a null
916 This structure is somewhat mystical, but after meditating on it, it will make
921 The ``QualType`` class
922 ----------------------
924 The ``QualType`` class is designed as a trivial value class that is small,
925 passed by-value and is efficient to query. The idea of ``QualType`` is that it
926 stores the type qualifiers (``const``, ``volatile``, ``restrict``, plus some
927 extended qualifiers required by language extensions) separately from the types
928 themselves. ``QualType`` is conceptually a pair of "``Type*``" and the bits
929 for these type qualifiers.
931 By storing the type qualifiers as bits in the conceptual pair, it is extremely
932 efficient to get the set of qualifiers on a ``QualType`` (just return the field
933 of the pair), add a type qualifier (which is a trivial constant-time operation
934 that sets a bit), and remove one or more type qualifiers (just return a
935 ``QualType`` with the bitfield set to empty).
937 Further, because the bits are stored outside of the type itself, we do not need
938 to create duplicates of types with different sets of qualifiers (i.e. there is
939 only a single heap allocated "``int``" type: "``const int``" and "``volatile
940 const int``" both point to the same heap allocated "``int``" type). This
941 reduces the heap size used to represent bits and also means we do not have to
942 consider qualifiers when uniquing types (:ref:`Type <Type>` does not even
945 In practice, the two most common type qualifiers (``const`` and ``restrict``)
946 are stored in the low bits of the pointer to the ``Type`` object, together with
947 a flag indicating whether extended qualifiers are present (which must be
948 heap-allocated). This means that ``QualType`` is exactly the same size as a
956 The ``DeclarationName`` class represents the name of a declaration in Clang.
957 Declarations in the C family of languages can take several different forms.
958 Most declarations are named by simple identifiers, e.g., "``f``" and "``x``" in
959 the function declaration ``f(int x)``. In C++, declaration names can also name
960 class constructors ("``Class``" in ``struct Class { Class(); }``), class
961 destructors ("``~Class``"), overloaded operator names ("``operator+``"), and
962 conversion functions ("``operator void const *``"). In Objective-C,
963 declaration names can refer to the names of Objective-C methods, which involve
964 the method name and the parameters, collectively called a *selector*, e.g.,
965 "``setWidth:height:``". Since all of these kinds of entities --- variables,
966 functions, Objective-C methods, C++ constructors, destructors, and operators
967 --- are represented as subclasses of Clang's common ``NamedDecl`` class,
968 ``DeclarationName`` is designed to efficiently represent any kind of name.
970 Given a ``DeclarationName`` ``N``, ``N.getNameKind()`` will produce a value
971 that describes what kind of name ``N`` stores. There are 10 options (all of
972 the names are inside the ``DeclarationName`` class).
976 The name is a simple identifier. Use ``N.getAsIdentifierInfo()`` to retrieve
977 the corresponding ``IdentifierInfo*`` pointing to the actual identifier.
979 ``ObjCZeroArgSelector``, ``ObjCOneArgSelector``, ``ObjCMultiArgSelector``
981 The name is an Objective-C selector, which can be retrieved as a ``Selector``
982 instance via ``N.getObjCSelector()``. The three possible name kinds for
983 Objective-C reflect an optimization within the ``DeclarationName`` class:
984 both zero- and one-argument selectors are stored as a masked
985 ``IdentifierInfo`` pointer, and therefore require very little space, since
986 zero- and one-argument selectors are far more common than multi-argument
987 selectors (which use a different structure).
989 ``CXXConstructorName``
991 The name is a C++ constructor name. Use ``N.getCXXNameType()`` to retrieve
992 the :ref:`type <QualType>` that this constructor is meant to construct. The
993 type is always the canonical type, since all constructors for a given type
996 ``CXXDestructorName``
998 The name is a C++ destructor name. Use ``N.getCXXNameType()`` to retrieve
999 the :ref:`type <QualType>` whose destructor is being named. This type is
1000 always a canonical type.
1002 ``CXXConversionFunctionName``
1004 The name is a C++ conversion function. Conversion functions are named
1005 according to the type they convert to, e.g., "``operator void const *``".
1006 Use ``N.getCXXNameType()`` to retrieve the type that this conversion function
1007 converts to. This type is always a canonical type.
1011 The name is a C++ overloaded operator name. Overloaded operators are named
1012 according to their spelling, e.g., "``operator+``" or "``operator new []``".
1013 Use ``N.getCXXOverloadedOperator()`` to retrieve the overloaded operator (a
1014 value of type ``OverloadedOperatorKind``).
1016 ``CXXLiteralOperatorName``
1018 The name is a C++11 user defined literal operator. User defined
1019 Literal operators are named according to the suffix they define,
1020 e.g., "``_foo``" for "``operator "" _foo``". Use
1021 ``N.getCXXLiteralIdentifier()`` to retrieve the corresponding
1022 ``IdentifierInfo*`` pointing to the identifier.
1024 ``CXXUsingDirective``
1026 The name is a C++ using directive. Using directives are not really
1027 NamedDecls, in that they all have the same name, but they are
1028 implemented as such in order to store them in DeclContext
1031 ``DeclarationName``\ s are cheap to create, copy, and compare. They require
1032 only a single pointer's worth of storage in the common cases (identifiers,
1033 zero- and one-argument Objective-C selectors) and use dense, uniqued storage
1034 for the other kinds of names. Two ``DeclarationName``\ s can be compared for
1035 equality (``==``, ``!=``) using a simple bitwise comparison, can be ordered
1036 with ``<``, ``>``, ``<=``, and ``>=`` (which provide a lexicographical ordering
1037 for normal identifiers but an unspecified ordering for other kinds of names),
1038 and can be placed into LLVM ``DenseMap``\ s and ``DenseSet``\ s.
1040 ``DeclarationName`` instances can be created in different ways depending on
1041 what kind of name the instance will store. Normal identifiers
1042 (``IdentifierInfo`` pointers) and Objective-C selectors (``Selector``) can be
1043 implicitly converted to ``DeclarationNames``. Names for C++ constructors,
1044 destructors, conversion functions, and overloaded operators can be retrieved
1045 from the ``DeclarationNameTable``, an instance of which is available as
1046 ``ASTContext::DeclarationNames``. The member functions
1047 ``getCXXConstructorName``, ``getCXXDestructorName``,
1048 ``getCXXConversionFunctionName``, and ``getCXXOperatorName``, respectively,
1049 return ``DeclarationName`` instances for the four kinds of C++ special function
1054 Declaration contexts
1055 --------------------
1057 Every declaration in a program exists within some *declaration context*, such
1058 as a translation unit, namespace, class, or function. Declaration contexts in
1059 Clang are represented by the ``DeclContext`` class, from which the various
1060 declaration-context AST nodes (``TranslationUnitDecl``, ``NamespaceDecl``,
1061 ``RecordDecl``, ``FunctionDecl``, etc.) will derive. The ``DeclContext`` class
1062 provides several facilities common to each declaration context:
1064 Source-centric vs. Semantics-centric View of Declarations
1066 ``DeclContext`` provides two views of the declarations stored within a
1067 declaration context. The source-centric view accurately represents the
1068 program source code as written, including multiple declarations of entities
1069 where present (see the section :ref:`Redeclarations and Overloads
1070 <Redeclarations>`), while the semantics-centric view represents the program
1071 semantics. The two views are kept synchronized by semantic analysis while
1072 the ASTs are being constructed.
1074 Storage of declarations within that context
1076 Every declaration context can contain some number of declarations. For
1077 example, a C++ class (represented by ``RecordDecl``) contains various member
1078 functions, fields, nested types, and so on. All of these declarations will
1079 be stored within the ``DeclContext``, and one can iterate over the
1080 declarations via [``DeclContext::decls_begin()``,
1081 ``DeclContext::decls_end()``). This mechanism provides the source-centric
1082 view of declarations in the context.
1084 Lookup of declarations within that context
1086 The ``DeclContext`` structure provides efficient name lookup for names within
1087 that declaration context. For example, if ``N`` is a namespace we can look
1088 for the name ``N::f`` using ``DeclContext::lookup``. The lookup itself is
1089 based on a lazily-constructed array (for declaration contexts with a small
1090 number of declarations) or hash table (for declaration contexts with more
1091 declarations). The lookup operation provides the semantics-centric view of
1092 the declarations in the context.
1094 Ownership of declarations
1096 The ``DeclContext`` owns all of the declarations that were declared within
1097 its declaration context, and is responsible for the management of their
1098 memory as well as their (de-)serialization.
1100 All declarations are stored within a declaration context, and one can query
1101 information about the context in which each declaration lives. One can
1102 retrieve the ``DeclContext`` that contains a particular ``Decl`` using
1103 ``Decl::getDeclContext``. However, see the section
1104 :ref:`LexicalAndSemanticContexts` for more information about how to interpret
1105 this context information.
1109 Redeclarations and Overloads
1110 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1112 Within a translation unit, it is common for an entity to be declared several
1113 times. For example, we might declare a function "``f``" and then later
1114 re-declare it as part of an inlined definition:
1118 void f(int x, int y, int z = 1);
1120 inline void f(int x, int y, int z) { /* ... */ }
1122 The representation of "``f``" differs in the source-centric and
1123 semantics-centric views of a declaration context. In the source-centric view,
1124 all redeclarations will be present, in the order they occurred in the source
1125 code, making this view suitable for clients that wish to see the structure of
1126 the source code. In the semantics-centric view, only the most recent "``f``"
1127 will be found by the lookup, since it effectively replaces the first
1128 declaration of "``f``".
1130 In the semantics-centric view, overloading of functions is represented
1131 explicitly. For example, given two declarations of a function "``g``" that are
1139 the ``DeclContext::lookup`` operation will return a
1140 ``DeclContext::lookup_result`` that contains a range of iterators over
1141 declarations of "``g``". Clients that perform semantic analysis on a program
1142 that is not concerned with the actual source code will primarily use this
1143 semantics-centric view.
1145 .. _LexicalAndSemanticContexts:
1147 Lexical and Semantic Contexts
1148 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1150 Each declaration has two potentially different declaration contexts: a
1151 *lexical* context, which corresponds to the source-centric view of the
1152 declaration context, and a *semantic* context, which corresponds to the
1153 semantics-centric view. The lexical context is accessible via
1154 ``Decl::getLexicalDeclContext`` while the semantic context is accessible via
1155 ``Decl::getDeclContext``, both of which return ``DeclContext`` pointers. For
1156 most declarations, the two contexts are identical. For example:
1165 Here, the semantic and lexical contexts of ``X::f`` are the ``DeclContext``
1166 associated with the class ``X`` (itself stored as a ``RecordDecl`` AST node).
1167 However, we can now define ``X::f`` out-of-line:
1171 void X::f(int x = 17) { /* ... */ }
1173 This definition of "``f``" has different lexical and semantic contexts. The
1174 lexical context corresponds to the declaration context in which the actual
1175 declaration occurred in the source code, e.g., the translation unit containing
1176 ``X``. Thus, this declaration of ``X::f`` can be found by traversing the
1177 declarations provided by [``decls_begin()``, ``decls_end()``) in the
1180 The semantic context of ``X::f`` corresponds to the class ``X``, since this
1181 member function is (semantically) a member of ``X``. Lookup of the name ``f``
1182 into the ``DeclContext`` associated with ``X`` will then return the definition
1183 of ``X::f`` (including information about the default argument).
1185 Transparent Declaration Contexts
1186 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1188 In C and C++, there are several contexts in which names that are logically
1189 declared inside another declaration will actually "leak" out into the enclosing
1190 scope from the perspective of name lookup. The most obvious instance of this
1191 behavior is in enumeration types, e.g.,
1201 Here, ``Color`` is an enumeration, which is a declaration context that contains
1202 the enumerators ``Red``, ``Green``, and ``Blue``. Thus, traversing the list of
1203 declarations contained in the enumeration ``Color`` will yield ``Red``,
1204 ``Green``, and ``Blue``. However, outside of the scope of ``Color`` one can
1205 name the enumerator ``Red`` without qualifying the name, e.g.,
1211 There are other entities in C++ that provide similar behavior. For example,
1212 linkage specifications that use curly braces:
1220 // f and g are visible here
1222 For source-level accuracy, we treat the linkage specification and enumeration
1223 type as a declaration context in which its enclosed declarations ("``Red``",
1224 "``Green``", and "``Blue``"; "``f``" and "``g``") are declared. However, these
1225 declarations are visible outside of the scope of the declaration context.
1227 These language features (and several others, described below) have roughly the
1228 same set of requirements: declarations are declared within a particular lexical
1229 context, but the declarations are also found via name lookup in scopes
1230 enclosing the declaration itself. This feature is implemented via
1231 *transparent* declaration contexts (see
1232 ``DeclContext::isTransparentContext()``), whose declarations are visible in the
1233 nearest enclosing non-transparent declaration context. This means that the
1234 lexical context of the declaration (e.g., an enumerator) will be the
1235 transparent ``DeclContext`` itself, as will the semantic context, but the
1236 declaration will be visible in every outer context up to and including the
1237 first non-transparent declaration context (since transparent declaration
1238 contexts can be nested).
1240 The transparent ``DeclContext``\ s are:
1242 * Enumerations (but not C++11 "scoped enumerations"):
1251 // Red, Green, and Blue are in scope
1253 * C++ linkage specifications:
1261 // f and g are in scope
1263 * Anonymous unions and structs:
1267 struct LookupTable {
1270 std::vector<Item> *Vector;
1271 std::set<Item> *Set;
1276 LT.Vector = 0; // Okay: finds Vector inside the unnamed union
1278 * C++11 inline namespaces:
1283 inline namespace debug {
1287 mylib::X *xp; // okay: mylib::X refers to mylib::debug::X
1289 .. _MultiDeclContext:
1291 Multiply-Defined Declaration Contexts
1292 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1294 C++ namespaces have the interesting --- and, so far, unique --- property that
1295 the namespace can be defined multiple times, and the declarations provided by
1296 each namespace definition are effectively merged (from the semantic point of
1297 view). For example, the following two code snippets are semantically
1316 In Clang's representation, the source-centric view of declaration contexts will
1317 actually have two separate ``NamespaceDecl`` nodes in Snippet #1, each of which
1318 is a declaration context that contains a single declaration of "``f``".
1319 However, the semantics-centric view provided by name lookup into the namespace
1320 ``N`` for "``f``" will return a ``DeclContext::lookup_result`` that contains a
1321 range of iterators over declarations of "``f``".
1323 ``DeclContext`` manages multiply-defined declaration contexts internally. The
1324 function ``DeclContext::getPrimaryContext`` retrieves the "primary" context for
1325 a given ``DeclContext`` instance, which is the ``DeclContext`` responsible for
1326 maintaining the lookup table used for the semantics-centric view. Given a
1327 DeclContext, one can obtain the set of declaration contexts that are semanticaly
1328 connected to this declaration context, in source order, including this context
1329 (which will be the only result, for non-namespace contexts) via
1330 ``DeclContext::collectAllContexts``. Note that these functions are used
1331 internally within the lookup and insertion methods of the ``DeclContext``, so
1332 the vast majority of clients can ignore them.
1339 The ``CFG`` class is designed to represent a source-level control-flow graph
1340 for a single statement (``Stmt*``). Typically instances of ``CFG`` are
1341 constructed for function bodies (usually an instance of ``CompoundStmt``), but
1342 can also be instantiated to represent the control-flow of any class that
1343 subclasses ``Stmt``, which includes simple expressions. Control-flow graphs
1344 are especially useful for performing `flow- or path-sensitive
1345 <http://en.wikipedia.org/wiki/Data_flow_analysis#Sensitivities>`_ program
1346 analyses on a given function.
1351 Concretely, an instance of ``CFG`` is a collection of basic blocks. Each basic
1352 block is an instance of ``CFGBlock``, which simply contains an ordered sequence
1353 of ``Stmt*`` (each referring to statements in the AST). The ordering of
1354 statements within a block indicates unconditional flow of control from one
1355 statement to the next. :ref:`Conditional control-flow
1356 <ConditionalControlFlow>` is represented using edges between basic blocks. The
1357 statements within a given ``CFGBlock`` can be traversed using the
1358 ``CFGBlock::*iterator`` interface.
1360 A ``CFG`` object owns the instances of ``CFGBlock`` within the control-flow
1361 graph it represents. Each ``CFGBlock`` within a CFG is also uniquely numbered
1362 (accessible via ``CFGBlock::getBlockID()``). Currently the number is based on
1363 the ordering the blocks were created, but no assumptions should be made on how
1364 ``CFGBlocks`` are numbered other than their numbers are unique and that they
1365 are numbered from 0..N-1 (where N is the number of basic blocks in the CFG).
1367 Entry and Exit Blocks
1368 ^^^^^^^^^^^^^^^^^^^^^
1370 Each instance of ``CFG`` contains two special blocks: an *entry* block
1371 (accessible via ``CFG::getEntry()``), which has no incoming edges, and an
1372 *exit* block (accessible via ``CFG::getExit()``), which has no outgoing edges.
1373 Neither block contains any statements, and they serve the role of providing a
1374 clear entrance and exit for a body of code such as a function body. The
1375 presence of these empty blocks greatly simplifies the implementation of many
1376 analyses built on top of CFGs.
1378 .. _ConditionalControlFlow:
1380 Conditional Control-Flow
1381 ^^^^^^^^^^^^^^^^^^^^^^^^
1383 Conditional control-flow (such as those induced by if-statements and loops) is
1384 represented as edges between ``CFGBlocks``. Because different C language
1385 constructs can induce control-flow, each ``CFGBlock`` also records an extra
1386 ``Stmt*`` that represents the *terminator* of the block. A terminator is
1387 simply the statement that caused the control-flow, and is used to identify the
1388 nature of the conditional control-flow between blocks. For example, in the
1389 case of an if-statement, the terminator refers to the ``IfStmt`` object in the
1390 AST that represented the given branch.
1392 To illustrate, consider the following code example:
1408 After invoking the parser+semantic analyzer on this code fragment, the AST of
1409 the body of ``foo`` is referenced by a single ``Stmt*``. We can then construct
1410 an instance of ``CFG`` representing the control-flow graph of this function
1411 body by single call to a static class method:
1416 std::unique_ptr<CFG> FooCFG = CFG::buildCFG(FooBody);
1418 Along with providing an interface to iterate over its ``CFGBlocks``, the
1419 ``CFG`` class also provides methods that are useful for debugging and
1420 visualizing CFGs. For example, the method ``CFG::dump()`` dumps a
1421 pretty-printed version of the CFG to standard error. This is especially useful
1422 when one is using a debugger such as gdb. For example, here is the output of
1435 Predecessors (1): B5
1436 Successors (2): B3 B2
1440 Predecessors (1): B4
1446 Predecessors (1): B4
1451 Predecessors (2): B2 B3
1455 Predecessors (1): B1
1458 For each block, the pretty-printed output displays for each block the number of
1459 *predecessor* blocks (blocks that have outgoing control-flow to the given
1460 block) and *successor* blocks (blocks that have control-flow that have incoming
1461 control-flow from the given block). We can also clearly see the special entry
1462 and exit blocks at the beginning and end of the pretty-printed output. For the
1463 entry block (block B5), the number of predecessor blocks is 0, while for the
1464 exit block (block B0) the number of successor blocks is 0.
1466 The most interesting block here is B4, whose outgoing control-flow represents
1467 the branching caused by the sole if-statement in ``foo``. Of particular
1468 interest is the second statement in the block, ``(x > 2)``, and the terminator,
1469 printed as ``if [B4.2]``. The second statement represents the evaluation of
1470 the condition of the if-statement, which occurs before the actual branching of
1471 control-flow. Within the ``CFGBlock`` for B4, the ``Stmt*`` for the second
1472 statement refers to the actual expression in the AST for ``(x > 2)``. Thus
1473 pointers to subclasses of ``Expr`` can appear in the list of statements in a
1474 block, and not just subclasses of ``Stmt`` that refer to proper C statements.
1476 The terminator of block B4 is a pointer to the ``IfStmt`` object in the AST.
1477 The pretty-printer outputs ``if [B4.2]`` because the condition expression of
1478 the if-statement has an actual place in the basic block, and thus the
1479 terminator is essentially *referring* to the expression that is the second
1480 statement of block B4 (i.e., B4.2). In this manner, conditions for
1481 control-flow (which also includes conditions for loops and switch statements)
1482 are hoisted into the actual basic block.
1484 .. Implicit Control-Flow
1485 .. ^^^^^^^^^^^^^^^^^^^^^
1487 .. A key design principle of the ``CFG`` class was to not require any
1488 .. transformations to the AST in order to represent control-flow. Thus the
1489 .. ``CFG`` does not perform any "lowering" of the statements in an AST: loops
1490 .. are not transformed into guarded gotos, short-circuit operations are not
1491 .. converted to a set of if-statements, and so on.
1493 Constant Folding in the Clang AST
1494 ---------------------------------
1496 There are several places where constants and constant folding matter a lot to
1497 the Clang front-end. First, in general, we prefer the AST to retain the source
1498 code as close to how the user wrote it as possible. This means that if they
1499 wrote "``5+4``", we want to keep the addition and two constants in the AST, we
1500 don't want to fold to "``9``". This means that constant folding in various
1501 ways turns into a tree walk that needs to handle the various cases.
1503 However, there are places in both C and C++ that require constants to be
1504 folded. For example, the C standard defines what an "integer constant
1505 expression" (i-c-e) is with very precise and specific requirements. The
1506 language then requires i-c-e's in a lot of places (for example, the size of a
1507 bitfield, the value for a case statement, etc). For these, we have to be able
1508 to constant fold the constants, to do semantic checks (e.g., verify bitfield
1509 size is non-negative and that case statements aren't duplicated). We aim for
1510 Clang to be very pedantic about this, diagnosing cases when the code does not
1511 use an i-c-e where one is required, but accepting the code unless running with
1512 ``-pedantic-errors``.
1514 Things get a little bit more tricky when it comes to compatibility with
1515 real-world source code. Specifically, GCC has historically accepted a huge
1516 superset of expressions as i-c-e's, and a lot of real world code depends on
1517 this unfortuate accident of history (including, e.g., the glibc system
1518 headers). GCC accepts anything its "fold" optimizer is capable of reducing to
1519 an integer constant, which means that the definition of what it accepts changes
1520 as its optimizer does. One example is that GCC accepts things like "``case
1521 X-X:``" even when ``X`` is a variable, because it can fold this to 0.
1523 Another issue are how constants interact with the extensions we support, such
1524 as ``__builtin_constant_p``, ``__builtin_inf``, ``__extension__`` and many
1525 others. C99 obviously does not specify the semantics of any of these
1526 extensions, and the definition of i-c-e does not include them. However, these
1527 extensions are often used in real code, and we have to have a way to reason
1530 Finally, this is not just a problem for semantic analysis. The code generator
1531 and other clients have to be able to fold constants (e.g., to initialize global
1532 variables) and has to handle a superset of what C99 allows. Further, these
1533 clients can benefit from extended information. For example, we know that
1534 "``foo() || 1``" always evaluates to ``true``, but we can't replace the
1535 expression with ``true`` because it has side effects.
1537 Implementation Approach
1538 ^^^^^^^^^^^^^^^^^^^^^^^
1540 After trying several different approaches, we've finally converged on a design
1541 (Note, at the time of this writing, not all of this has been implemented,
1542 consider this a design goal!). Our basic approach is to define a single
1543 recursive method evaluation method (``Expr::Evaluate``), which is implemented
1544 in ``AST/ExprConstant.cpp``. Given an expression with "scalar" type (integer,
1545 fp, complex, or pointer) this method returns the following information:
1547 * Whether the expression is an integer constant expression, a general constant
1548 that was folded but has no side effects, a general constant that was folded
1549 but that does have side effects, or an uncomputable/unfoldable value.
1550 * If the expression was computable in any way, this method returns the
1551 ``APValue`` for the result of the expression.
1552 * If the expression is not evaluatable at all, this method returns information
1553 on one of the problems with the expression. This includes a
1554 ``SourceLocation`` for where the problem is, and a diagnostic ID that explains
1555 the problem. The diagnostic should have ``ERROR`` type.
1556 * If the expression is not an integer constant expression, this method returns
1557 information on one of the problems with the expression. This includes a
1558 ``SourceLocation`` for where the problem is, and a diagnostic ID that
1559 explains the problem. The diagnostic should have ``EXTENSION`` type.
1561 This information gives various clients the flexibility that they want, and we
1562 will eventually have some helper methods for various extensions. For example,
1563 ``Sema`` should have a ``Sema::VerifyIntegerConstantExpression`` method, which
1564 calls ``Evaluate`` on the expression. If the expression is not foldable, the
1565 error is emitted, and it would return ``true``. If the expression is not an
1566 i-c-e, the ``EXTENSION`` diagnostic is emitted. Finally it would return
1567 ``false`` to indicate that the AST is OK.
1569 Other clients can use the information in other ways, for example, codegen can
1570 just use expressions that are foldable in any way.
1575 This section describes how some of the various extensions Clang supports
1576 interacts with constant evaluation:
1578 * ``__extension__``: The expression form of this extension causes any
1579 evaluatable subexpression to be accepted as an integer constant expression.
1580 * ``__builtin_constant_p``: This returns true (as an integer constant
1581 expression) if the operand evaluates to either a numeric value (that is, not
1582 a pointer cast to integral type) of integral, enumeration, floating or
1583 complex type, or if it evaluates to the address of the first character of a
1584 string literal (possibly cast to some other type). As a special case, if
1585 ``__builtin_constant_p`` is the (potentially parenthesized) condition of a
1586 conditional operator expression ("``?:``"), only the true side of the
1587 conditional operator is considered, and it is evaluated with full constant
1589 * ``__builtin_choose_expr``: The condition is required to be an integer
1590 constant expression, but we accept any constant as an "extension of an
1591 extension". This only evaluates one operand depending on which way the
1592 condition evaluates.
1593 * ``__builtin_classify_type``: This always returns an integer constant
1595 * ``__builtin_inf, nan, ...``: These are treated just like a floating-point
1597 * ``__builtin_abs, copysign, ...``: These are constant folded as general
1598 constant expressions.
1599 * ``__builtin_strlen`` and ``strlen``: These are constant folded as integer
1600 constant expressions if the argument is a string literal.
1607 This library is called by the :ref:`Parser library <Parser>` during parsing to
1608 do semantic analysis of the input. For valid programs, Sema builds an AST for
1616 CodeGen takes an :ref:`AST <AST>` as input and produces `LLVM IR code
1617 <//llvm.org/docs/LangRef.html>`_ from it.
1622 How to add an attribute
1623 -----------------------
1624 Attributes are a form of metadata that can be attached to a program construct,
1625 allowing the programmer to pass semantic information along to the compiler for
1626 various uses. For example, attributes may be used to alter the code generation
1627 for a program construct, or to provide extra semantic information for static
1628 analysis. This document explains how to add a custom attribute to Clang.
1629 Documentation on existing attributes can be found `here
1630 <//clang.llvm.org/docs/AttributeReference.html>`_.
1634 Attributes in Clang are handled in three stages: parsing into a parsed attribute
1635 representation, conversion from a parsed attribute into a semantic attribute,
1636 and then the semantic handling of the attribute.
1638 Parsing of the attribute is determined by the various syntactic forms attributes
1639 can take, such as GNU, C++11, and Microsoft style attributes, as well as other
1640 information provided by the table definition of the attribute. Ultimately, the
1641 parsed representation of an attribute object is an ``AttributeList`` object.
1642 These parsed attributes chain together as a list of parsed attributes attached
1643 to a declarator or declaration specifier. The parsing of attributes is handled
1644 automatically by Clang, except for attributes spelled as keywords. When
1645 implementing a keyword attribute, the parsing of the keyword and creation of the
1646 ``AttributeList`` object must be done manually.
1648 Eventually, ``Sema::ProcessDeclAttributeList()`` is called with a ``Decl`` and
1649 an ``AttributeList``, at which point the parsed attribute can be transformed
1650 into a semantic attribute. The process by which a parsed attribute is converted
1651 into a semantic attribute depends on the attribute definition and semantic
1652 requirements of the attribute. The end result, however, is that the semantic
1653 attribute object is attached to the ``Decl`` object, and can be obtained by a
1654 call to ``Decl::getAttr<T>()``.
1656 The structure of the semantic attribute is also governed by the attribute
1657 definition given in Attr.td. This definition is used to automatically generate
1658 functionality used for the implementation of the attribute, such as a class
1659 derived from ``clang::Attr``, information for the parser to use, automated
1660 semantic checking for some attributes, etc.
1663 ``include/clang/Basic/Attr.td``
1664 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1665 The first step to adding a new attribute to Clang is to add its definition to
1666 `include/clang/Basic/Attr.td
1667 <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/Attr.td?view=markup>`_.
1668 This tablegen definition must derive from the ``Attr`` (tablegen, not
1669 semantic) type, or one of its derivatives. Most attributes will derive from the
1670 ``InheritableAttr`` type, which specifies that the attribute can be inherited by
1671 later redeclarations of the ``Decl`` it is associated with.
1672 ``InheritableParamAttr`` is similar to ``InheritableAttr``, except that the
1673 attribute is written on a parameter instead of a declaration. If the attribute
1674 is intended to apply to a type instead of a declaration, such an attribute
1675 should derive from ``TypeAttr``, and will generally not be given an AST
1676 representation. (Note that this document does not cover the creation of type
1677 attributes.) An attribute that inherits from ``IgnoredAttr`` is parsed, but will
1678 generate an ignored attribute diagnostic when used, which may be useful when an
1679 attribute is supported by another vendor but not supported by clang.
1681 The definition will specify several key pieces of information, such as the
1682 semantic name of the attribute, the spellings the attribute supports, the
1683 arguments the attribute expects, and more. Most members of the ``Attr`` tablegen
1684 type do not require definitions in the derived definition as the default
1685 suffice. However, every attribute must specify at least a spelling list, a
1686 subject list, and a documentation list.
1690 All attributes are required to specify a spelling list that denotes the ways in
1691 which the attribute can be spelled. For instance, a single semantic attribute
1692 may have a keyword spelling, as well as a C++11 spelling and a GNU spelling. An
1693 empty spelling list is also permissible and may be useful for attributes which
1694 are created implicitly. The following spellings are accepted:
1696 ============ ================================================================
1697 Spelling Description
1698 ============ ================================================================
1699 ``GNU`` Spelled with a GNU-style ``__attribute__((attr))`` syntax and
1701 ``CXX11`` Spelled with a C++-style ``[[attr]]`` syntax. If the attribute
1702 is meant to be used by Clang, it should set the namespace to
1704 ``Declspec`` Spelled with a Microsoft-style ``__declspec(attr)`` syntax.
1705 ``Keyword`` The attribute is spelled as a keyword, and required custom
1707 ``GCC`` Specifies two spellings: the first is a GNU-style spelling, and
1708 the second is a C++-style spelling with the ``gnu`` namespace.
1709 Attributes should only specify this spelling for attributes
1711 ``Pragma`` The attribute is spelled as a ``#pragma``, and requires custom
1712 processing within the preprocessor. If the attribute is meant to
1713 be used by Clang, it should set the namespace to ``"clang"``.
1714 Note that this spelling is not used for declaration attributes.
1715 ============ ================================================================
1719 Attributes appertain to one or more ``Decl`` subjects. If the attribute attempts
1720 to attach to a subject that is not in the subject list, a diagnostic is issued
1721 automatically. Whether the diagnostic is a warning or an error depends on how
1722 the attribute's ``SubjectList`` is defined, but the default behavior is to warn.
1723 The diagnostics displayed to the user are automatically determined based on the
1724 subjects in the list, but a custom diagnostic parameter can also be specified in
1725 the ``SubjectList``. The diagnostics generated for subject list violations are
1726 either ``diag::warn_attribute_wrong_decl_type`` or
1727 ``diag::err_attribute_wrong_decl_type``, and the parameter enumeration is found
1728 in `include/clang/Sema/AttributeList.h
1729 <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Sema/AttributeList.h?view=markup>`_
1730 If a previously unused Decl node is added to the ``SubjectList``, the logic used
1731 to automatically determine the diagnostic parameter in `utils/TableGen/ClangAttrEmitter.cpp
1732 <http://llvm.org/viewvc/llvm-project/cfe/trunk/utils/TableGen/ClangAttrEmitter.cpp?view=markup>`_
1733 may need to be updated.
1735 By default, all subjects in the SubjectList must either be a Decl node defined
1736 in ``DeclNodes.td``, or a statement node defined in ``StmtNodes.td``. However,
1737 more complex subjects can be created by creating a ``SubsetSubject`` object.
1738 Each such object has a base subject which it appertains to (which must be a
1739 Decl or Stmt node, and not a SubsetSubject node), and some custom code which is
1740 called when determining whether an attribute appertains to the subject. For
1741 instance, a ``NonBitField`` SubsetSubject appertains to a ``FieldDecl``, and
1742 tests whether the given FieldDecl is a bit field. When a SubsetSubject is
1743 specified in a SubjectList, a custom diagnostic parameter must also be provided.
1745 Diagnostic checking for attribute subject lists is automated except when
1746 ``HasCustomParsing`` is set to ``1``.
1750 All attributes must have some form of documentation associated with them.
1751 Documentation is table generated on the public web server by a server-side
1752 process that runs daily. Generally, the documentation for an attribute is a
1753 stand-alone definition in `include/clang/Basic/AttrDocs.td
1754 <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/AttdDocs.td?view=markup>`_
1755 that is named after the attribute being documented.
1757 If the attribute is not for public consumption, or is an implicitly-created
1758 attribute that has no visible spelling, the documentation list can specify the
1759 ``Undocumented`` object. Otherwise, the attribute should have its documentation
1760 added to AttrDocs.td.
1762 Documentation derives from the ``Documentation`` tablegen type. All derived
1763 types must specify a documentation category and the actual documentation itself.
1764 Additionally, it can specify a custom heading for the attribute, though a
1765 default heading will be chosen when possible.
1767 There are four predefined documentation categories: ``DocCatFunction`` for
1768 attributes that appertain to function-like subjects, ``DocCatVariable`` for
1769 attributes that appertain to variable-like subjects, ``DocCatType`` for type
1770 attributes, and ``DocCatStmt`` for statement attributes. A custom documentation
1771 category should be used for groups of attributes with similar functionality.
1772 Custom categories are good for providing overview information for the attributes
1773 grouped under it. For instance, the consumed annotation attributes define a
1774 custom category, ``DocCatConsumed``, that explains what consumed annotations are
1777 Documentation content (whether it is for an attribute or a category) is written
1778 using reStructuredText (RST) syntax.
1780 After writing the documentation for the attribute, it should be locally tested
1781 to ensure that there are no issues generating the documentation on the server.
1782 Local testing requires a fresh build of clang-tblgen. To generate the attribute
1783 documentation, execute the following command::
1785 clang-tblgen -gen-attr-docs -I /path/to/clang/include /path/to/clang/include/clang/Basic/Attr.td -o /path/to/clang/docs/AttributeReference.rst
1787 When testing locally, *do not* commit changes to ``AttributeReference.rst``.
1788 This file is generated by the server automatically, and any changes made to this
1789 file will be overwritten.
1793 Attributes may optionally specify a list of arguments that can be passed to the
1794 attribute. Attribute arguments specify both the parsed form and the semantic
1795 form of the attribute. For example, if ``Args`` is
1796 ``[StringArgument<"Arg1">, IntArgument<"Arg2">]`` then
1797 ``__attribute__((myattribute("Hello", 3)))`` will be a valid use; it requires
1798 two arguments while parsing, and the Attr subclass' constructor for the
1799 semantic attribute will require a string and integer argument.
1801 All arguments have a name and a flag that specifies whether the argument is
1802 optional. The associated C++ type of the argument is determined by the argument
1803 definition type. If the existing argument types are insufficient, new types can
1804 be created, but it requires modifying `utils/TableGen/ClangAttrEmitter.cpp
1805 <http://llvm.org/viewvc/llvm-project/cfe/trunk/utils/TableGen/ClangAttrEmitter.cpp?view=markup>`_
1806 to properly support the type.
1810 The ``Attr`` definition has other members which control the behavior of the
1811 attribute. Many of them are special-purpose and beyond the scope of this
1812 document, however a few deserve mention.
1814 If the parsed form of the attribute is more complex, or differs from the
1815 semantic form, the ``HasCustomParsing`` bit can be set to ``1`` for the class,
1816 and the parsing code in `Parser::ParseGNUAttributeArgs()
1817 <http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Parse/ParseDecl.cpp?view=markup>`_
1818 can be updated for the special case. Note that this only applies to arguments
1819 with a GNU spelling -- attributes with a __declspec spelling currently ignore
1820 this flag and are handled by ``Parser::ParseMicrosoftDeclSpec``.
1822 Note that setting this member to 1 will opt out of common attribute semantic
1823 handling, requiring extra implementation efforts to ensure the attribute
1824 appertains to the appropriate subject, etc.
1826 If the attribute should not be propagated from from a template declaration to an
1827 instantiation of the template, set the ``Clone`` member to 0. By default, all
1828 attributes will be cloned to template instantiations.
1830 Attributes that do not require an AST node should set the ``ASTNode`` field to
1831 ``0`` to avoid polluting the AST. Note that anything inheriting from
1832 ``TypeAttr`` or ``IgnoredAttr`` automatically do not generate an AST node. All
1833 other attributes generate an AST node by default. The AST node is the semantic
1834 representation of the attribute.
1836 The ``LangOpts`` field specifies a list of language options required by the
1837 attribute. For instance, all of the CUDA-specific attributes specify ``[CUDA]``
1838 for the ``LangOpts`` field, and when the CUDA language option is not enabled, an
1839 "attribute ignored" warning diagnostic is emitted. Since language options are
1840 not table generated nodes, new language options must be created manually and
1841 should specify the spelling used by ``LangOptions`` class.
1843 Custom accessors can be generated for an attribute based on the spelling list
1844 for that attribute. For instance, if an attribute has two different spellings:
1845 'Foo' and 'Bar', accessors can be created:
1846 ``[Accessor<"isFoo", [GNU<"Foo">]>, Accessor<"isBar", [GNU<"Bar">]>]``
1847 These accessors will be generated on the semantic form of the attribute,
1848 accepting no arguments and returning a ``bool``.
1850 Attributes that do not require custom semantic handling should set the
1851 ``SemaHandler`` field to ``0``. Note that anything inheriting from
1852 ``IgnoredAttr`` automatically do not get a semantic handler. All other
1853 attributes are assumed to use a semantic handler by default. Attributes
1854 without a semantic handler are not given a parsed attribute ``Kind`` enumerator.
1856 Target-specific attributes may share a spelling with other attributes in
1857 different targets. For instance, the ARM and MSP430 targets both have an
1858 attribute spelled ``GNU<"interrupt">``, but with different parsing and semantic
1859 requirements. To support this feature, an attribute inheriting from
1860 ``TargetSpecificAttribute`` may specify a ``ParseKind`` field. This field
1861 should be the same value between all arguments sharing a spelling, and
1862 corresponds to the parsed attribute's ``Kind`` enumerator. This allows
1863 attributes to share a parsed attribute kind, but have distinct semantic
1864 attribute classes. For instance, ``AttributeList::AT_Interrupt`` is the shared
1865 parsed attribute kind, but ARMInterruptAttr and MSP430InterruptAttr are the
1866 semantic attributes generated.
1868 By default, when declarations are merging attributes, an attribute will not be
1869 duplicated. However, if an attribute can be duplicated during this merging
1870 stage, set ``DuplicatesAllowedWhileMerging`` to ``1``, and the attribute will
1873 By default, attribute arguments are parsed in an evaluated context. If the
1874 arguments for an attribute should be parsed in an unevaluated context (akin to
1875 the way the argument to a ``sizeof`` expression is parsed), set
1876 ``ParseArgumentsAsUnevaluated`` to ``1``.
1878 If additional functionality is desired for the semantic form of the attribute,
1879 the ``AdditionalMembers`` field specifies code to be copied verbatim into the
1880 semantic attribute class object, with ``public`` access.
1884 All semantic processing of declaration attributes happens in `lib/Sema/SemaDeclAttr.cpp
1885 <http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Sema/SemaDeclAttr.cpp?view=markup>`_,
1886 and generally starts in the ``ProcessDeclAttribute()`` function. If the
1887 attribute is a "simple" attribute -- meaning that it requires no custom semantic
1888 processing aside from what is automatically provided, add a call to
1889 ``handleSimpleAttribute<YourAttr>(S, D, Attr);`` to the switch statement.
1890 Otherwise, write a new ``handleYourAttr()`` function, and add that to the switch
1891 statement. Please do not implement handling logic directly in the ``case`` for
1894 Unless otherwise specified by the attribute definition, common semantic checking
1895 of the parsed attribute is handled automatically. This includes diagnosing
1896 parsed attributes that do not appertain to the given ``Decl``, ensuring the
1897 correct minimum number of arguments are passed, etc.
1899 If the attribute adds additional warnings, define a ``DiagGroup`` in
1900 `include/clang/Basic/DiagnosticGroups.td
1901 <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/DiagnosticGroups.td?view=markup>`_
1902 named after the attribute's ``Spelling`` with "_"s replaced by "-"s. If there
1903 is only a single diagnostic, it is permissible to use ``InGroup<DiagGroup<"your-attribute">>``
1904 directly in `DiagnosticSemaKinds.td
1905 <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/DiagnosticSemaKinds.td?view=markup>`_
1907 All semantic diagnostics generated for your attribute, including automatically-
1908 generated ones (such as subjects and argument counts), should have a
1909 corresponding test case.
1913 Most attributes are implemented to have some effect on the compiler. For
1914 instance, to modify the way code is generated, or to add extra semantic checks
1915 for an analysis pass, etc. Having added the attribute definition and conversion
1916 to the semantic representation for the attribute, what remains is to implement
1917 the custom logic requiring use of the attribute.
1919 The ``clang::Decl`` object can be queried for the presence or absence of an
1920 attribute using ``hasAttr<T>()``. To obtain a pointer to the semantic
1921 representation of the attribute, ``getAttr<T>`` may be used.
1923 How to add an expression or statement
1924 -------------------------------------
1926 Expressions and statements are one of the most fundamental constructs within a
1927 compiler, because they interact with many different parts of the AST, semantic
1928 analysis, and IR generation. Therefore, adding a new expression or statement
1929 kind into Clang requires some care. The following list details the various
1930 places in Clang where an expression or statement needs to be introduced, along
1931 with patterns to follow to ensure that the new expression or statement works
1932 well across all of the C languages. We focus on expressions, but statements
1935 #. Introduce parsing actions into the parser. Recursive-descent parsing is
1936 mostly self-explanatory, but there are a few things that are worth keeping
1939 * Keep as much source location information as possible! You'll want it later
1940 to produce great diagnostics and support Clang's various features that map
1941 between source code and the AST.
1942 * Write tests for all of the "bad" parsing cases, to make sure your recovery
1943 is good. If you have matched delimiters (e.g., parentheses, square
1944 brackets, etc.), use ``Parser::BalancedDelimiterTracker`` to give nice
1945 diagnostics when things go wrong.
1947 #. Introduce semantic analysis actions into ``Sema``. Semantic analysis should
1948 always involve two functions: an ``ActOnXXX`` function that will be called
1949 directly from the parser, and a ``BuildXXX`` function that performs the
1950 actual semantic analysis and will (eventually!) build the AST node. It's
1951 fairly common for the ``ActOnCXX`` function to do very little (often just
1952 some minor translation from the parser's representation to ``Sema``'s
1953 representation of the same thing), but the separation is still important:
1954 C++ template instantiation, for example, should always call the ``BuildXXX``
1955 variant. Several notes on semantic analysis before we get into construction
1958 * Your expression probably involves some types and some subexpressions.
1959 Make sure to fully check that those types, and the types of those
1960 subexpressions, meet your expectations. Add implicit conversions where
1961 necessary to make sure that all of the types line up exactly the way you
1962 want them. Write extensive tests to check that you're getting good
1963 diagnostics for mistakes and that you can use various forms of
1964 subexpressions with your expression.
1965 * When type-checking a type or subexpression, make sure to first check
1966 whether the type is "dependent" (``Type::isDependentType()``) or whether a
1967 subexpression is type-dependent (``Expr::isTypeDependent()``). If any of
1968 these return ``true``, then you're inside a template and you can't do much
1969 type-checking now. That's normal, and your AST node (when you get there)
1970 will have to deal with this case. At this point, you can write tests that
1971 use your expression within templates, but don't try to instantiate the
1973 * For each subexpression, be sure to call ``Sema::CheckPlaceholderExpr()``
1974 to deal with "weird" expressions that don't behave well as subexpressions.
1975 Then, determine whether you need to perform lvalue-to-rvalue conversions
1976 (``Sema::DefaultLvalueConversions``) or the usual unary conversions
1977 (``Sema::UsualUnaryConversions``), for places where the subexpression is
1978 producing a value you intend to use.
1979 * Your ``BuildXXX`` function will probably just return ``ExprError()`` at
1980 this point, since you don't have an AST. That's perfectly fine, and
1981 shouldn't impact your testing.
1983 #. Introduce an AST node for your new expression. This starts with declaring
1984 the node in ``include/Basic/StmtNodes.td`` and creating a new class for your
1985 expression in the appropriate ``include/AST/Expr*.h`` header. It's best to
1986 look at the class for a similar expression to get ideas, and there are some
1987 specific things to watch for:
1989 * If you need to allocate memory, use the ``ASTContext`` allocator to
1990 allocate memory. Never use raw ``malloc`` or ``new``, and never hold any
1991 resources in an AST node, because the destructor of an AST node is never
1993 * Make sure that ``getSourceRange()`` covers the exact source range of your
1994 expression. This is needed for diagnostics and for IDE support.
1995 * Make sure that ``children()`` visits all of the subexpressions. This is
1996 important for a number of features (e.g., IDE support, C++ variadic
1997 templates). If you have sub-types, you'll also need to visit those
1998 sub-types in ``RecursiveASTVisitor`` and ``DataRecursiveASTVisitor``.
1999 * Add printing support (``StmtPrinter.cpp``) for your expression.
2000 * Add profiling support (``StmtProfile.cpp``) for your AST node, noting the
2001 distinguishing (non-source location) characteristics of an instance of
2002 your expression. Omitting this step will lead to hard-to-diagnose
2003 failures regarding matching of template declarations.
2004 * Add serialization support (``ASTReaderStmt.cpp``, ``ASTWriterStmt.cpp``)
2007 #. Teach semantic analysis to build your AST node. At this point, you can wire
2008 up your ``Sema::BuildXXX`` function to actually create your AST. A few
2009 things to check at this point:
2011 * If your expression can construct a new C++ class or return a new
2012 Objective-C object, be sure to update and then call
2013 ``Sema::MaybeBindToTemporary`` for your just-created AST node to be sure
2014 that the object gets properly destructed. An easy way to test this is to
2015 return a C++ class with a private destructor: semantic analysis should
2016 flag an error here with the attempt to call the destructor.
2017 * Inspect the generated AST by printing it using ``clang -cc1 -ast-print``,
2018 to make sure you're capturing all of the important information about how
2019 the AST was written.
2020 * Inspect the generated AST under ``clang -cc1 -ast-dump`` to verify that
2021 all of the types in the generated AST line up the way you want them.
2022 Remember that clients of the AST should never have to "think" to
2023 understand what's going on. For example, all implicit conversions should
2024 show up explicitly in the AST.
2025 * Write tests that use your expression as a subexpression of other,
2026 well-known expressions. Can you call a function using your expression as
2027 an argument? Can you use the ternary operator?
2029 #. Teach code generation to create IR to your AST node. This step is the first
2030 (and only) that requires knowledge of LLVM IR. There are several things to
2033 * Code generation is separated into scalar/aggregate/complex and
2034 lvalue/rvalue paths, depending on what kind of result your expression
2035 produces. On occasion, this requires some careful factoring of code to
2037 * ``CodeGenFunction`` contains functions ``ConvertType`` and
2038 ``ConvertTypeForMem`` that convert Clang's types (``clang::Type*`` or
2039 ``clang::QualType``) to LLVM types. Use the former for values, and the
2040 later for memory locations: test with the C++ "``bool``" type to check
2041 this. If you find that you are having to use LLVM bitcasts to make the
2042 subexpressions of your expression have the type that your expression
2043 expects, STOP! Go fix semantic analysis and the AST so that you don't
2044 need these bitcasts.
2045 * The ``CodeGenFunction`` class has a number of helper functions to make
2046 certain operations easy, such as generating code to produce an lvalue or
2047 an rvalue, or to initialize a memory location with a given value. Prefer
2048 to use these functions rather than directly writing loads and stores,
2049 because these functions take care of some of the tricky details for you
2050 (e.g., for exceptions).
2051 * If your expression requires some special behavior in the event of an
2052 exception, look at the ``push*Cleanup`` functions in ``CodeGenFunction``
2053 to introduce a cleanup. You shouldn't have to deal with
2054 exception-handling directly.
2055 * Testing is extremely important in IR generation. Use ``clang -cc1
2056 -emit-llvm`` and `FileCheck
2057 <http://llvm.org/docs/CommandGuide/FileCheck.html>`_ to verify that you're
2058 generating the right IR.
2060 #. Teach template instantiation how to cope with your AST node, which requires
2061 some fairly simple code:
2063 * Make sure that your expression's constructor properly computes the flags
2064 for type dependence (i.e., the type your expression produces can change
2065 from one instantiation to the next), value dependence (i.e., the constant
2066 value your expression produces can change from one instantiation to the
2067 next), instantiation dependence (i.e., a template parameter occurs
2068 anywhere in your expression), and whether your expression contains a
2069 parameter pack (for variadic templates). Often, computing these flags
2070 just means combining the results from the various types and
2072 * Add ``TransformXXX`` and ``RebuildXXX`` functions to the ``TreeTransform``
2073 class template in ``Sema``. ``TransformXXX`` should (recursively)
2074 transform all of the subexpressions and types within your expression,
2075 using ``getDerived().TransformYYY``. If all of the subexpressions and
2076 types transform without error, it will then call the ``RebuildXXX``
2077 function, which will in turn call ``getSema().BuildXXX`` to perform
2078 semantic analysis and build your expression.
2079 * To test template instantiation, take those tests you wrote to make sure
2080 that you were type checking with type-dependent expressions and dependent
2081 types (from step #2) and instantiate those templates with various types,
2082 some of which type-check and some that don't, and test the error messages
2085 #. There are some "extras" that make other features work better. It's worth
2086 handling these extras to give your expression complete integration into
2089 * Add code completion support for your expression in
2090 ``SemaCodeComplete.cpp``.
2091 * If your expression has types in it, or has any "interesting" features
2092 other than subexpressions, extend libclang's ``CursorVisitor`` to provide
2093 proper visitation for your expression, enabling various IDE features such
2094 as syntax highlighting, cross-referencing, and so on. The
2095 ``c-index-test`` helper program can be used to test these features.