Parse ( US: UK: ), syntax analysis or syntactic analysis is the process of analyzing a series of symbols, either in natural language, computer language or data structure, in accordance with formal grammar rules. The term parsing is derived from the Latin pars ( orationis ), meaning the part (speech).
This term has a slightly different meaning in the various branches of linguistics and computer science. Traditional deciphering of sentences is often done as a method to understand the exact meaning of a sentence or word, sometimes with the help of a device like a sentence diagram. Usually emphasizes the importance of grammatical divisions such as subject and predicate.
In computational linguistics the term is used to refer to formal analysis by computer sentences or other word sequences into its constituents, producing parse trees that show their syntactic relation to each other, which may also contain semantic and other information.
The term is also used in psycholinguistics when describing language comprehension. In this context, parsing refers to the way humans analyze sentences or phrases (in spoken language or text) "in terms of grammatical constituencies, identifying parts of speech, syntactic relationships, etc." This term is very common when discussing what linguistic signals help the speaker to interpret the garden-line phrase.
In computer science, this term is used in computer language analysis, referring to the syntactic analysis of the input code to its component parts to facilitate compiler and interpreter writing. This term can also be used to describe separation or segregation.
Video Parsing
Bahasa manusia
Metode tradisional
The traditional grammar exercise of parsing, sometimes known as clause analysis , involves breaking the text into parts of its words with an explanation of the shape, function, and syntactic relationships of each part. This is determined largely from the study of language conjugation and language decline, which can be very complicated for languages ​​that are widely altered. To parse phrases such as' human dog bite 'involves noting that a single noun' human 'is the subject of a sentence, the verb' bite 'is the sole third person of the current verb form of the verb' bite ', and the singular noun' dog 'is the object of the sentence. Techniques such as sentence diagrams are sometimes used to show relationships between elements in a sentence.
Parsing was formerly a grammar teaching center throughout the English-speaking world, and is widely regarded as the basis for the use and understanding of written languages. However, the general teaching of such techniques is no longer current.
Computational methods
In some machine translation and natural language processing systems, written text in human language is outlined by a computer program. The human sentence is not easily elucidated by the program, because there is substantial ambiguity in the structure of human language, whose use is to convey the meaning (or semantics) among possibilities of infinite possibilities but only some of which are closely related to a particular case. So the saying "Man bites dog" versus "Dog bites man" is definitely on one detail but in other languages ​​may appear as "Male dog bites" with dependence on the larger context to distinguish between the two possibilities, if indeed the difference is attention. It is difficult to prepare a formal rule to describe informal behavior although it is clear that some rules are being followed.
To parse natural language data, the researcher must first approve the grammar to be used. The choice of syntax is influenced by linguistic and computational concerns; for example some parsing systems use lexical functional grammar, but in general, the decomposition for this type of grammar is known as NP-complete. The grammar-driven structure of the phrase head is another linguistic formalism that has been popular in parsing communities, but other research efforts have focused on the less complex formalism used in Penn Treebank. A shallow decomposition aims to find only the boundaries of the main constituents such as the noun phrase. Another popular strategy to avoid linguistic controversy is the decomposition of grammar dependencies.
Most modern parsers are at least partially statistically; that is, they rely on the corpus of the annotated training data (handcrafted). This approach allows the system to gather information about the frequency at which various constructions occur in a particular context. (See machine learning.) The approaches used include direct PCFGs (probabilistic context free grammars), maximum entropy, and nerve webs. Most of the more successful systems use lexical statistics (that is, they consider the identity of the words involved, as well as the part of their speech). However such systems are vulnerable to overfitting and require some kind of refinement to be effective.
Parsing algorithms for natural language can not rely on grammars that have 'good' properties as well as manually designed grammars for programming languages. As mentioned earlier, some grammatical formalizations are very difficult to parse computationally computationally; in general, even if the desired structure is not context-free, a sort of context-free approach to grammar is used to perform the first pass. Algorithms that use context-free grammars often rely on several variants of the CYK algorithm, usually with multiple heuristics to trim away analysis is not possible to save time. (See graph parsing.) However some systems trade speed for accuracy using, for example, the linear time-versions of the shift-reducing algorithm. A somewhat recent development has parsed reranking in which parsers propose a large number of analyzes, and more complex systems choose the best option.
Psycholinguistics
In psycholinguistics, parsing involves not only the assignment of words to categories, but the evaluation of the meaning of sentences according to the syntactical rules is drawn on the conclusions made from each word in the sentence. This usually happens when words are heard or read. Consequently, the psycholinguistic model of parsing is an additional imperative, meaning that they construct interpretations as sentences being processed, which are usually expressed in the form of partial syntactic structures. The creation of the structure was initially wrong when interpreting the sentence of the garden path.
Maps Parsing
Computer language
Parser
A parser is a software component that takes input data (often text) and builds a data structure - often a parse tree, an abstract syntax tree or other hierarchical structure, provides a structural representation of the input while checking the correct syntax. The decomposition may be preceded or followed by another step, or it can be combined into one step. The parser is often preceded by a separate lexical analysis, which creates the token of the input character sequence; alternatively, it can be combined in un-scanned parsing. The parser can be programmed by hand or it can be automatically or semi-automatically generated by the parser generator. Parsing is a complement to templating, which results in formatted output. These can be applied to different domains, but often appear together, such as scanf/printf pair, or input (front end parsing) and output (back end code generation) compiler stages.
Input to parser is often text in some computer languages, but can also be text in natural language or textual data that is less structured, which in this case is generally only a certain part of the extracted text, rather than the parsing tree that is being constructed. Parsers range from very simple functions such as scanf, to complex programs such as frontend of C compilers or HTML parsers from web browsers. An important class of simple parsing is done using regular expressions, where a group of regular expressions define regular languages ​​and regular expression engines automatically generate parsers for that language, enabling pattern matching and text extraction. In other contexts, regular expressions are used instead of parsing, as a lexing step whose results are then used by parsers.
The use of parsers varies by input. In the case of data languages, parsers are often found as a facility of reading files from a program, such as reading in HTML or XML text; these examples are markup languages. In the case of a programming language, the parser is a component of the compiler or interpreter, which parses the source code of the computer programming language to create some form of internal representation; parser is a key step in the compiler frontend. Programming languages ​​tend to be determined in terms of deterministic context free grammars because fast and efficient parsers can be written for them. For the compiler, the parsing itself can be done in one pass or multiple pass - see the compiler one-pass and multi-pass compiler.
The implied losses of the one-pass compiler can be largely overcome by adding fixes, where provisions are made for code relocation during the forward pass, and improvements are applied backwards when the current program segment has been fully recognized. An example where such a remedial mechanism would be useful would be the forward Goto statement, where the GOTO target is not known until the program segment is completed. In this case, the implementation of the fix will be postponed until the target GOTO is recognized. In contrast, GOTO withdrawal does not require repair, since its location is already known.
Context-free grammar is limited to the extent to which they can express all language requirements. Informally, the reason is that such language memory is limited. Grammar can not remember the presence of a construct over input that is too long; this is necessary for the language in which, for example, the name must be declared before it may be referenced. A stronger grammar that can reveal this constraint, however, can not be described efficiently. Thus, it is a common strategy to create casual parsers for context-free grammars that accept the superset of the desired language constructs (that is, it accepts some invalid constructs); then, unwanted constructs can be filtered at the semantic analysis stage (contextual analysis).
For example, in Python, the following is a valid code syntactically:
The following code, however, is a valid syntax in terms of context-free grammar, generating syntactic trees with the same structure as before but syntactically invalid in terms of context-sensitive grammar, which requires that variables be initialized before using:
Instead of being analyzed at the parsing stage, it is caught by examining the values ​​ in the syntax tree, then as part of the semantics analysis: the syntactically-sensitive context in practice is often easier to analyze as semantic.
Process overview
The following example shows a common case of parsing computer languages ​​with two levels of grammar: lexical and syntax.
The first stage is the token generation, or lexical analysis, in which the input character stream is divided into meaningful symbols defined by the regular expression grammar. For example, the calculator program will see inputs like " 12 * (3 4) ^ 2 " and divide them into tokens 12 , * , ( 3 , , 4 , ) , ^ , 2 , each of which is a meaningful symbol in the context of arithmetic expressions. Lexer will contain a rule to say that the characters * , , ^ , ( and ) marks the start of a new token, so meaningless tokens like " 12 * " or " (3 " will not be generated.
The next stage is a parsing or syntactic analysis, which checks that tokens form a permissible expression. This is usually done by referring to context-free grammars that recursively define the components that can shape the expressions and the order in which they should appear. However, not all rules that define a programming language can be expressed by context-free grammar only, for example type validity and an appropriate identifier declaration. These rules can be formally expressed with attribute grammar.
The final phase is semantic analysis or decomposition, which implements the implications of a recently validated expression and takes appropriate action. In the case of a calculator or interpreter, the action is to evaluate the expression or program, the compiler, on the other hand, will generate some sort of code. Attribute grammar can also be used to determine this action.
Type parser
The task of the parser is basically to determine if and how the input can be derived from the initial symbol of the grammar. This can be done in two ways:
- Top-down parsing - Top-down parsing can be seen as an attempt to find the leftmost derivation of the input stream by searching the parse tree using a top-down extension of the given formal grammar rules. Token consumed from left to right. The inclusive option is used to accommodate ambiguity by extending all the right-hand side alternatives to grammar rules.
- Bottom-up parsing - The parser can start with input and try to rewrite it to the initial symbol. Intuitively, the parser seeks to find the most basic elements, then the elements that contain this, and so on. The LR parser is an example of a bottom-up parser. Another term used for this type of parser is Shift-Reduce parsing.
The LL parser and recursive-descent parser are examples of top-down parsers that can not accommodate left recursive production rules. Although it has been believed that a simple implementation of top-down decomposition can not accommodate both direct and indirect left recurrences and may require exponential time and space complexity when deciphering grammar without ambiguous context, more sophisticated algorithms for top-down parsing have been created by Frost , Hafiz, and Callaghan that accommodate ambiguity and abandon recursion in polynomial time and which results in a polynomial size representation of the number of potential parse trees. Their algorithm is capable of deriving the leftmost and the far right input derivation with regard to the context-free grammar provided.
The important difference with respect to the parser is whether the parser produces the leftmost derivation or the rightmost derivation (see grammar without context). The LL parser will produce the leftmost derivative and the LR parser will produce the rightmost derivation (though usually reversed).
Some of the graphical parsing algorithms have been designed for visual programming languages. Parsers for visual languages ​​are sometimes based on grammar graphs.
Parser development software
Some well-known parser development tools include the following. Also see comparison of parser generator.
Lookahead
Lookahead specifies the maximum token the parser can use to decide which rule to use. Lookahead is highly relevant to LL, LR, and LALR parsers, which are often explicitly indicated by applying a lookahead to the algorithm name in parentheses, such as LALR (1).
Most programming languages, the main target parser, are carefully defined in such a way that parsers with limited lookahead, usually one, can decipher it, because parsers with limited lookaheads are often more efficient. One important change to this trend occurred in 1990 when Terence Parr created the ANTLR for his Ph.D. thesis, generator parser for efficient LL parser, k is any fixed value.
The parser usually has only a few actions after viewing each token. They shift (add this token to the stack for subtraction later), subtract (pop tokens from stack and construct syntax form), end, error (no known rules apply) or conflict (do not know whether to shift or decrease).
Lookahead has two advantages.
- This helps the parser take the correct action in case of a conflict. For example, describing an if statement in case of another clause.
- Eliminates many duplicate states and eases additional stack loads. Parser A non-lookahead C language will have about 10,000 states. A lookahead parser will have about 300 states.
Example: Parse Expression 1 2 * 3
Most programming languages ​​(except some such as APL and Smalltalk) and algebraic formulas give higher priority to multiplication than additions, in this case the correct interpretation of the above example is 1 (2 * 3) . Note that Rule4 above is a semantic rule. It is possible to rewrite grammar to include this into syntax. However, not all of these rules can be translated into syntax.
- Simple non-lookahead parser action
Initially Input = [1,, 2, *, 3]
- Shift "1" to the stack of the input (to anticipate rule3). Input = [, 2, *, 3] Stack = [1]
- Reduce "1" for "E" expressions based on rule3. Stack = [E]
- Slide "" to the stack of inputs (to anticipate rule1). Input = [2, *, 3] Stack = [E,]
- Move "2" to the stack of inputs (to anticipate rule3). Input = [*, 3] Stack = [E ,, 2]
- Reduce the stack element "2" into "E" Expression based on rule3. Stack = [E,, E]
- Reduce stack items [E ,, E] and new "E" entries to "E" based on rule1. Stack = [E]
- Move "*" to the stack of inputs (to anticipate rule2). Input = [3] Stack = [E, *]
- Move "3" to the stack of inputs (to anticipate rule3). Input = [] (empty) Stack = [E, *, 3]
- Reduce the stack element "3" for the expression "E" based on rule3. Stack = [E, *, E]
- Subtract stack items [E, *, E] and enter new "E" to "E" based on rule2. Stack = [E]
The parsing tree and the code generated from it are incorrect according to the language semantics.
To parse correctly without lookahead, there are three solutions:
- The user must attach the expression in parentheses. This is often not a feasible solution.
- The parser should have more logic to back off and try again whenever the rule is violated or incomplete. A similar method is followed in the LL parser.
- Alternatively, parsers or grammars must have extra logic to delay the reduction and only reduce when absolutely sure which rule should be reduced first. This method is used in the LR parser. It correctly parses the expression but with more status and increases the depth of the stack.
- Lookahead parser actions
- Shift 1 to the stack on input 1 to anticipate rule3. It is not directly reduced.
- Reduce stack item 1 into a simple Expression on the input based on rule3. Lookahead is, so we are on the path to E, so we can reduce the stack to E.
- Shift to the stack on the input to anticipate rule1.
- Shift 2 to the stack in input 2 in anticipation of rule3.
- Subtract item 2 on Expression on input * by rule3. Lookahead * only expect E before.
- Now the stack has E E and the input is still *. It has two options now, either to switch based on rule2 or subtraction based on rule1. Since * has a higher priority than rule4, we shift * to the stack to anticipate rule2.
- Shift 3 to the stack in input 3 to anticipate rule3.
- Subtract stack item 3 into Expression after viewing the end of input based on rule3.
- Subtract item stack E * E into E based on rule2.
- Subtract item E from E to E based on rule1.
The resulting parsing tree is correct and more efficient than a non-lookahead parser. This is the strategy followed in the LALR parser.
See also
References
Further reading
External links
- LALR Lemon Grinder
- Stanford Parser Parser Stanford
- Turin University Parser A natural language breaker for Italian, open source, developed in Common Lisp by Leonardo Lesmo, University of Torino, Italy.
- Short history of parser construction
- Spoon: Library for analyzing, modifying, rewriting, and browsing the Java source code. It parses source files to build well-designed ASTs with robust API analysis and transformation.
Source of the article : Wikipedia
0 Comments