Intersection and Union Hierarchies of Deterministic Context-Free Languages and Pumping Lemmas

Abstract

We study the computational complexity of finite intersections and unions of deterministic context-free languages. Earlier, Wotschke (1978) demonstrated that intersections of deterministic context-free languages are in general more powerful than intersections of d deterministic context-free languages for any positive integer d based on the hierarchy separation of Liu and Weiner (1973). The argument of Liu and Weiner, however, works only on bounded languages of particular forms, and therefore Wotschke’s result cannot be extended to disprove any other language to be written in the form of an intersection of d deterministic context-free languages. To deal with the non-membership of a wide range of languages, we circumvent their proof argument and instead devise a new, practical technical tool: a pumping lemma for finite unions of deterministic context-free languages. Since the family of deterministic context-free languages is closed under complementation, this pumping lemma enables us to show a non-membership relation of languages made up with finite intersections of even non-bounded languages as well. We also refer to a relationship to Hibbard’s limited automata.

A Historical Account and an Overview of Contributions

Intersection and Union Hierarchies and Historical Background

In formal language theory, context-free languages constitute a fundamental family , which is situated in between the family of regular languages and that of context-sensitive languages. It has been well known that this family is closed under an operation of union but not closed under intersection. As a quick example, the language is not context-free but it can be expressed as an intersection of two context-free languages. This non-closure property can be further generalized to any intersection of d () context-free languages. For later notational convenience, we here write for the family of such languages, namely, the d intersection closure of (see, e.g., [13]). With this notation, the above language belongs to . Similarly, the language over an alphabet falls into because can be expressed as an intersection of d context-free languages of the form (). In 1973, Liu and Weiner [8] gave a contrived proof to their key statement that (*) is outside of for any index . Therefore, the collection truly forms an infinite hierarchy.

Deterministic context-free (dcf) languages have been a focal point in since a systematic study of Ginsburg and Greibach [1]. The importance of such languages can be exemplified by the facts that dcf languages are easy to parse and that every context-free language is simply the homomorphic image of a dcf language. Unlike , the family of dcf languages is closed under neither union nor intersection. We use the terms of d-intersection deterministic context-free (dcf) languages and d-union deterministic context-free (dcf) languages to express intersections of d dcf languages and unions of d dcf languages, respectively. For brevity, we write and respectively for the family of all d-intersection dcf languages and that of all d-union dcf languages, while Wotschke [11, 12] earlier referred to the d-intersection closure of . In particular, we obtain . Since is closed under complementation, it follows that the complement of coincides with . For our convenience, we call two hierarchies and the intersection and union hierarchies of dcf languages, respectively. Concerning these hierarchies, we set to be the intersection closure of , which is . In a similar way, we write for the union closure of , that is, .

Wotschke [11, 12] noted that the aforementioned result (*) of Liu and Weiner leads to the conclusion that truly forms an infinite hierarchy. To be more precise, since the language belongs to , the statement (*) implies , which instantly yields . Wotschke’s argument, nonetheless, heavily relies on the separation result of Liu and Weiner, who employed a notion of stratified semi-linear set to prove the statement (*). Notice that the proof technique of Liu and Weiner was developed only for a particular form of bounded languages1 and it is therefore applicable to specific languages, such as . In fact, the key idea of the proof of Liu and Weiner for is to focus on the number of the occurrences of each base symbol in appearing in each given string w and to translate into a set of Parikh images in order to exploit the semi-linearity of , where expresses the total number of symbols in a string w.

Because of the aforementioned limitation of Liu and Weiner’s proof technique, the scope of their proof cannot be extended to other forms of languages. Simple examples of such languages include , where [d] denotes the set . This is a bounded language expanding but its Parikh images do not have semi-linearity. As another example, let us take a look at a “non-palindrome” language , where expresses the reversal of . This is not even a bounded language. Therefore, Liu and Weiner’s argument is not directly applicable to verify that neither nor belongs to unless we dextrously pick up its core strings that form a certain bounded language. With no such contrived argument, how can we prove and to be outside of ? Moreover, given a language, how can we verify that it is not in ? We can ask similar questions for d-union dcf languages and the union hierarchy of dcf languages. Ginsburg and Greibach [1] remarked with no proof that the context-free language for any non-unary alphabet is not in . It is natural to call for a formal proof of the remark of Ginsburg and Greibach. Using a quite different language , however, Wotschke [11, 12] actually proved that does not belong to (more strongly, the Boolean closure of ) by employing the closure property of under inverse gsm mappings as well as complementation and intersection with regular languages. Wotschke’s proof relies on the following two facts. (i) The language can be expressed as the inverse gsm map of the language , restricted to . (ii) is expressed as the complement of , restricted to a certain regular language. Together with these facts, the final conclusion comes from the aforementioned result (*) of Liu and Weiner because implies by (i) and (ii). To our surprise, the fundamental results on that we have discussed so far are merely “corollaries” of the main result (*) of Liu and Weiner!

For further study on and answering more general non-membership questions to , we need to divert from Liu and Weiner’s contrived argument targeting the statement (*) and to develop a completely different, new, more practical technical tool. The sole purpose of this exposition is, therefore, set to (i) develop a new proof technique, which can be applicable to many other languages, (ii) present an alternative proof for the fact that the intersection and union hierarchies of DCFL are infinite hierarchies, and (iii) present other languages in that do not belong to (in part, verifying Ginsburg and Greibach’s remark for the first time).

In relevance to the union hierarchy of dcf languages, there is another known extension of using a different machine model called limited automata,2 which are originally invented by Hibbard [3] and later discussed extensively in, e.g., [9, 14]. Of all such machines, a d-limited deterministic automaton (or a d-lda, for short) is a deterministic Turing machine that can rewrite each tape cell in between two endmarkers only during the first d visits (except that making a turn of a tape head counts as double visits). We can raise a question of whether there is any relationship between the union hierarchy and d-lda’s.

Overview of Main Contributions

In Sect. 1.1, we have noted that fundamental properties associated with heavily rely on the single separation result (*) of Liu and Weiner. However, Liu and Weiner’s technical tool that leads to their main result does not seem to withstand a wide variety of direct applications. It is thus desirable to develop a new, simple, and practical technical tool that can find numerous applications for a future study on and . Thus, our main contribution of this exposition is to present a simple but powerful, practical technical tool, called the pumping lemma of languages in with , which also enriches our understanding of as well as . Notice that there have been numerous forms of so-called pumping lemmas (or iteration theorems) for variants of context-free languages in the past literature, e.g., [2, 4–7, 10, 15]. Our pumping lemma is a crucial addition to the list of such lemmas.

For a string x of length n and any number , x[i] stands for the ith symbol of x and for the i repetitions of x.

Lemma 1

(Pumping Lemma for DCFL[d]). Let d be any positive integer and let L be any d-union dcf language over an alphabet . There exist a constant such that, for any strings , if has the form for strings with and for any pair , then there exists two distinct indices for which the following conditions (1)–(2) hold. Let .

1. If , then either (a) or (b) holds.

   1. There is a factorization with and such that is in L for any number .
   2. There are two factorizations and with and such that is in L for any number .

2. In the case of , either (a) or (b) holds.

   1. There is a factorization with and such that, for each , is in L for any .
   2. Let and . There are three factorizations , , and with and such that and are in L for any number .

As a special case of , we obtain Yu’s pumping lemma [15, Lemma 1] from Lemma 1. Since there have been few machine-based analyses to prove various pumping lemmas in the past literature, one of the important aspects of this exposition is a clear demonstration of the first alternative proof to Yu’s pumping lemma, which is solely founded on an analysis of behaviors of 1dpda’s instead of derivation trees of grammars as in [15]. The proof of Lemma 1, in fact, exploits early results of [14] on an ideal shape form (Sect. 2.3) together with a new approach of -enhanced machines by analyzing transitions of crossing state-stack pairs (Sect. 2.4). These notions will be explained in Sect. 2 and their basic properties will be explored therein.

Using our pumping lemma (Lemma 1), we can expand the scope of the statement (*) of Liu and Weiner [8] targeting specific bounded languages to other types of languages, including and for each index .

Theorem 1

Let be any index.

1. The language is not in .

2. The language is not in .

Since Lemma 1 concerns with , in our proof of Theorem 1, we first take the complements of the above languages, restricted to suitable regular languages, and we then apply Lemma 1 appropriately to them. The proof sketch of this theorem will be given in Sect. 3. From Theorem 1, we instantly obtain the following consequences of Wotschke [11, 12].

Corollary 1

[11, 12] The intersection hierarchy of dcf languages and the union hierarchy of dcf languages are both infinite hierarchies.

Concerning the limitation of and in recognition power, since all unary context-free languages are also regular languages and the family of regular languages is closed under intersection, all unary languages in are regular as well. It is thus easy to find languages that are not in . Such languages, nevertheless, cannot serve themselves to separate from . As noted in Sect. 1.1, Ginsburg and Greibach [1] remarked with no proof that the context-free language does not belong to (as well as ). As another direct application of our pumping lemma, we give a formal written proof of their remark.

Theorem 2

The context-free language Pal is not in .

As an immediate consequence of the above theorem, we obtain Wotschke’s separation of from . Here, we stress that, unlike the work of Wotschke [11, 12], our proof does not depend on the main result (*) of Liu and Weiner.

Corollary 2

[11, 12] and .

We turn our interest to limited automata. Let us write for the family of all languages recognized by d-limited deterministic automata, in which their tape heads are allowed to rewrite tape symbols only during the first d accesses (except that, in the case of tape heads making a turn, we treat each turn as double visits). Hibbard [3] demonstrated that for any . A slightly modified language of his, which separates from , also belongs to the -th level of the union hierarchy of dcf languages but not in the -th level. We thus obtain the following separation.

Proposition 1

For any , .

The proofs of all the above assertions will be given after introducing necessary notions and notation in the subsequent section.

Preparations: Notions and Notation

Fundamental Notions and Notation

The set of all natural numbers (including 0) is denoted by . An integer interval for two integers m, n with is the set . In particular, for any integer , is abbreviated as [n]. For any string x, |x| indicates the total number of symbols in x. The special symbol is used to denote the empty string of length 0. For a language L over alphabet , denotes , the complement of L. Given a family of languages, expresses the complement family, which consists of languages for any .

Deterministic Pushdown Automata

A one-way deterministic pushdown automaton (or a 1dpda, for short) M is a tuple , where Q is a finite set of inner states, is an input alphabet with , is a stack alphabet, is a deterministic transition function from to , is the initial state in Q, is the bottom marker in , and and are subsets of Q. The symbols and respectively express the left-endmarker and the right-endmarker. Let . We assume that, if is defined, then is undefined for all symbols . Moreover, we require for any and . Each content of a stack is expressed as in which is the topmost stack symbol, is the bottom marker , and all others are placed in order from the top to the bottom of the stack.

Given , a d-intersection deterministic context-free (dcf) language is an intersection of d deterministic context-free (dcf) languages. Let denote the family of all d-intersection dcf languages. Similarly, we define d-union dcf languages and by substituting “union” for “intersection” in the above definitions. Note that because .

For two language families and , the notation (resp., ) denotes the family of all languages L for which there are two languages and over the same alphabet satisfying (resp., ).

Lemma 2

[11, 12] is closed under union, intersection with . In other words, and . A similar statement holds for .

Lemma 3

Let be any natural number.

1. iff .

2. If , then it follows that for any .

From Lemma 3(1) follows Corollary 1, provided that Theorem 1 is true. Theorem 1 itself will be proven in Sect. 3.

Ideal Shape

Let us recall from [14] a special “pop-controlled form” (called an ideal shape), in which the pop operations always take place by first reading an input symbol and then making a series (one or more) of the pop operations without reading any further input symbol. This notion was originally introduced for one-way probabilistic pushdown automata (or 1ppda’s); however, in this exposition, we apply this notion only to 1dpda’s. To be more formal, a 1dpda in an ideal shape is a 1dpda restricted to take only the following transitions. (1) Scanning , preserve the topmost stack symbol (called a stationary operation). (2) Scanning , push a new symbol u () without changing any other symbol in the stack. (3) Scanning , pop the topmost stack symbol. (4) Without scanning an input symbol (i.e., -move), pop the topmost stack symbol. (5) The stack operations (4) comes only after either (3) or (4).

It was shown in [14] that any 1ppda can be converted into its “error-equivalent” 1ppda in an ideal shape. In Lemma 4, we restate this result for 1dpda’s. We say that two 1dpda’s are (computationally) equivalent if, for any input x, their acceptance/rejection coincide. The push size of a 1ppda is the maximum length of any string pushed into a stack by any single move.

Lemma 4

(Ideal Shape Lemma for 1dpda’s). (cf. [14]) Let . Any n-state 1dpda M with stack alphabet size m and push size e can be converted into another (computationally) equivalent 1dpda N in an ideal shape with states and stack alphabet size .

Boundaries and Crossing State-Stack Pairs

We want to define two basic notions of boundaries and crossing state-stacks. For this purpose, we visualize a single move of a 1dpda M as three consecutive actions: (i) firstly replacing the topmost stack symbol, (ii) updating an inner state, and (iii) thirdly either moving a tape head or staying still.

A boundary is a borderline between two consecutive tape cells. We index all such boundaries from 0 to as follows. The boundary 0 is located at the left of cell 0 and boundary is in between cell i and for every index . When a string xy is written in |xy| consecutive cells, the (x, y)-boundary indicates the boundary , which separates between x and y. A boundary block between boundaries and with is a consecutive series of boundaries between and (including and ). These and are called ends of this boundary block. For brevity, we write to denote a boundary block between and . For two boundaries with , the -region refers to the consecutive cells located in the boundary block . When an input string x is written in the -region, we conveniently call this region the x-region unless the region is unclear from the context.

The stack height of M at boundary t is the length of the stack content while passing the boundary t. E.g., a stack content has stack height k.

A boundary block is called convex if there is a boundary s between and (namely, ) such that there is no pop operation in the -region and there is no push operation in the -region. A boundary block is flat if the stack height does not change in the -region. A boundary block with is pseudo-convex if the stack height at every boundary does not go below , where is the stack height at boundary for any . By their definitions, either convex or flat boundary blocks are also pseudo-convex.

A peak is a boundary t such that the stack heights at the boundaries and are smaller than the stack height at the boundary t. A plateau is a boundary block such that any stack height at a boundary is the same. A hill is a boundary block such that (i) the stack height at the boundary t and the stack height at the boundary coincide, (ii) there is at least one peak at a certain boundary , and (iii) both [t, i] and are convex. The height of a hill is the difference between the topmost stack height and the lowest stack height.

Given strings over alphabet , -enhanced strings are strings over the extended alphabet , where is treated as a special input symbol expressing the absence of symbols in . An -enhanced 1dpda (or an -1dpa, for short) is a 1dpda that takes -enhanced strings and works as a standard 1dpda except that a tape head always moves to the right without stopping. This tape head movement is sometimes called “real time”.

Lemma 5

For any 1dpda M, there exists an -1dpda N such that, for any input string x, there is an appropriate -enhanced string for which M accepts (resp., rejects) x iff N accepts (resp., rejects) . Moreover, is identical to x except for the symbol and is uniquely determined from x and M.

Let M be either a 1dpda or an -1dpda, and assume that M is in an ideal shape. A crossing state-stack pair at boundary i is a pair of inner state q and stack content . In a computation of M on input x, a crossing state-stack pair at boundary i refers to the machine’s current status where (1) M is reading an input symbol, say, at cell in a certain state, say, p with the stack content and then M changes its inner state to q, changing a by either pushing another symbol b satisfying or popping a with . Any computation of M on x can be expressed as a series of crossing state-stack pairs at every boundary in the -region.

Two boundaries and with are mutually correlated if there are two crossing state-stack pairs and at the boundaries and , respectively, for which the boundary block is pseudo-convex. Moreover, assume that . Two boundary blocks and are mutually correlated if (i) , , and are all pseudo-convex, (ii) and are crossing state-stack pairs at the boundaries and , respectively, and (iii) and are also crossing state-stack pairs at the boundaries and , respectively, for certain , , and .

If an -1dpda is in an ideal shape, then it pops exactly one stack symbol whenever it reads a single symbol of a given -enhanced input string.

Lemma 6

Let w be any string.

1. Let with . Let be a factorization such that is the -boundary and is the -boundary. If the boundaries and are mutually correlated and inner states at the boundaries and coincide, then it follows that iff for any .

2. Let with . Let such that each is -boundary for each . If two boundary blocks and are mutually correlated, inner states at the boundaries and coincide, and inner states at the boundaries and coincide, then it follows that iff for any number .

Proof Sketches of Three Separation Claims

We intend to present the proof sketches of three separation claims (Theorems 1 and 2 and Proposition 1) before verifying the pumping lemma. To understand our proofs better, we demonstrate a simple and easy example of how to apply Lemma 1 to obtain a separation between and .

Proposition 2

Let and let . It then follows that .

Proof

Let . Clearly, belongs to . Assuming , we apply the pumping lemma (Lemma 1) to . There is a constant that satisfies the lemma. Let and consider for each index . Since each belongs to , we can take an index pair with such that and satisfy the conditions of the lemma.

Since Condition (1) of the lemma is immediate, we hereafter consider Condition (2). Let , , and . Firstly, we consider Case (a) with a factorization with and . Since for any number , we conclude that and . Let and for certain numbers . Note that equals . Hence, for a certain . This implies that . We then obtain , which further implies that and . Similarly, from , it follows that . Thus, . This implies and . Since , we obtain a contradiction.

Next, we consider Case (b) with appropriate factorizations , , and with and such that and for any number . Since , we obtain . Assume that for a certain number . This is impossible because has the form and the exponent of b is not of the form rn for any number .

Proof Sketch of Theorem 1(1). Let be any integer and consider over . It is not difficult to check that . Our goal is, therefore, to show that is not in . To lead to a contradiction, we assume that .

Take in and consider , that is, . Note by Lemma 3(2) that, since , we obtain . Take a pumping-lemma constant that satisfies Lemma 1. We set and consider the set , where and for each index . Lemma 1 guarantees the existence of a specific distinct pair with .

By Lemma 1, since , there are two conditions to consider separately. Condition (1) is not difficult. Next, we consider Condition (2). Let , , and . Note that and . There are three factorizations with and , , and satisfying both and for any number . From follows . Let for a certain . In particular, take . Note that has factors and . Thus, we obtain , a clear contradiction.    

We omit from this exposition the proofs of Theorems 1(2), 2, and Proposition 1. These proofs will be included in its complete version.

Proof Sketch of the Pumping Lemma for DCFL[d]

We are now ready to provide the proof of the pumping lemma for (Lemma 1). Our proof has two different parts depending on the value of d. The first part of the proof targets the basis case of . This special case directly corresponds to Yu’s pumping lemma [15, Lemma 1]. To prove his lemma, Yu utilized a so-called left-part theorem of his for grammars. We intend to re-prove Yu’s lemma using only 1dpda’s with no reference to grammars. Our proof argument is easily extendable to one-way nondeterministic pushdown automata (or 1npda’s) and thus to the pumping lemma for . The second part of the proof deals with the general case of . Hereafter, we give the sketches of these two parts.

Basis Case of : Let be any alphabet and take any infinite dcf language L over . Let us consider an appropriate -1dpda in an ideal shape that recognizes L by Lemmas 4–5. For the desired constant c, we set . Firstly, we take two arbitrary strings xy and over with and .

Our goal is to show that Condition (2) in the basis case of holds. There are four distinct cases to deal with. Hereafter, we intend to discuss them separately. Note that, since M is one-way, every crossing state-stack pair at any boundary in the x-region does not depend on the choice of y and .

Case 1: Consider the case where there are two boundaries with and such that (i) the boundaries and are mutually correlated and (ii) inner states at the boundaries and coincide. In this case, we factorize x into so that and . By Lemma 6(1), it then follows that, for any number , and .

Case 2: Consider the case where there are four boundaries with and and there are , , and for which (i) and are the crossing state-stack pairs respectively at the boundaries and , (ii) and are the crossing state-stack pairs respectively at the boundaries and , and (iii) the boundary block for each index is pseudo-convex. We then take a factorization such that for each . Note that because of and . By an application of Lemma 6(2), we conclude that, for any , for all .

Case 3: Assume that Cases 1–2 fail. For brevity, we set . Consider the case where there is no pop operation in the R-region. Since R-region contains more than boundaries, the R-region includes a certain series of boundaries such that, for certain , , and , there are crossing state-stack pairs of the form at the boundaries , respectively. Note that the boundary blocks are all convex. Clearly, . We choose and so that (i) for each index , and are boundaries in the y-region and in the -region, respectively, satisfying that and , and (ii) for each index , is mutually correlated to in the y-region and also to in the -region. Note that the boundary blocks , are all pseudo-convex. Since , it follows that there is a pair with such that inner states at the boundaries and coincide. Using Lemma 6(2), we can obtain the desired conclusion.

Case 4: Assume that Cases 1–3 fail. In this case, we define a notion of “true gain” in the R-region and estimate its value. Choose and so that , , and the boundary block is pseudo-convex. Let denote the set of boundary blocks with , , for every , and for every such that (i) is pseudo-convex but cannot be flat, (ii) is pseudo-convex (and could be flat), (iii) there are crossing state-stack pairs at the boundaries for every , (iv) the stack height at the boundary is higher than the stack height at the boundary , (v) the boundary is a pit (i.e., the lowest point within its small vicinity). Define the true gain to be . It is possible to prove that . Using this inequality, we can employ an argument similar to Case 3 to obtain the lemma.

General Case of : We begin with proving this case by considering d 1dpda’s . The language recognized by each machine is denoted by . Let us assume that . Take strings in L and assume that each has the form with . Since all ’s are in L, define a function f as follows. Let f(k) denote the minimal index satisfying that but for all . Since there are at most d different languages, there are two distinct indices such that . In what follows, we fix such a pair .

Consider the case of and . Take arbitrary factorizations and . We apply the basis case of again and obtain one of the following (a)–(b). (a) There is a factorization with and such that and for any number . (b) There are factorizations , , and such that , , , and for any number .