Abstract
In this paper, we study the size of depth-two threshold circuits computing the inner product mod 2 function 
(mod 2). First, we reveal that 
can be computed by a depth-two threshold circuit of size significantly smaller than a folklore construction of size 
. Namely, we give a construction of such a circuit (denoted by 
circuit) of size 
. We also give an upper bound of 
for the case that the weights of the top threshold gate are polynomially bounded (denoted by 
circuit). Second, we give new lower bounds on the size of depth-two circuits of some special form; the top gate is an unbounded weight threshold gate and the bottom gates are symmetric gates (denoted by 
circuit). We show that any such circuit computing 
has size 
for every constant 
. This improves the previous bound of 
based on the sign-rank method due to Forster et al. [JCSS ’02, FSTTCS ’01]. Our technique has a unique feature that the lower bound is obtained by giving an explicit feasible solution to (the dual of) a certain linear programming problem. In fact, the problem itself was presented by the author over a decade ago [MFCS ’05], and finding a good solution is an actual contribution of this work.
Keywords: Circuit complexity, Threshold circuits, Linear programming, Upper bounds, Lower bounds
Introduction
The problem of proving strong lower bounds on the size (i.e., the number of gates) of depth-two threshold circuits computing an explicit Boolean function is a big challenge in complexity theory. Currently, we cannot refute that every function in the class NEXP (non-deterministic exponential time) can be computed by a polynomial-size depth-two circuit consisting of threshold gates with unbounded weights (denoted by 
circuit). There is a long line of research aiming for understanding the computational power and the limitation of depth-two threshold circuits (e.g, [5, 9, 10, 13, 14] or see an excellent book [12, Chapter 11.10]). The strongest known lower bound on the size of 
circuits for a function in NP is 
due to Kane and Williams [13].
In this paper, we focus on the size complexity of depth-two threshold circuits for the inner product mod 2 function:
 |
The inner product mod 2 function 
has been widely studied in the context of depth-two threshold circuits (e.g., [7, 10, 13]).
It is a long standing open question whether 
has a polynomial size depth-two threshold circuit with unbounded weights threshold gates in both layers. If we restrict the weights of threshold gates in one of two layers to be polynomial, then strong lower bounds are known. Let 
denote the class of threshold functions whose weights are bounded to be 
. Hajnal et al. [10] proved that every 
circuit computing 
has size 
using the discriminator method. An exponential lower bound were also shown by Nisan [16] using a communication complexity argument. Forster et al. [7, 8] proved that every 
circuit computing 
has size 
by lowerbounding the sign-rank of the communication matrix of 
.
Note that 
has an O(n) size threshold circuit of depth-three; in the first layer, we use n gates to compute 
for each i, and then in the second and third layer, we use O(n) gates to compute the parity of the outputs of them. If the gates at the bottom layer are restricted to be And, Exclusive-or or Symmetric gates, stronger lower bounds for 
are known (see Table 1). Remark that, in recent years, several results providing the separation between depth-two and depth-three threshold circuits were given for real-valued functions (e.g., [6, 18]). However, to the best of our knowledge, the arguments used in these works can not directly be applied for Boolean functions.
Table 1.
Known upper and lower bounds on the size of depth-two circuits using threshold gates that computes 
. Entries marked with (*) are shown in this paper. Unmarked results are folklore.
Our Contributions
The contribution of this work is twofold.
First, we consider upper bounds on the size of depth-two threshold circuits for 
. Although we know that lower bounds are more preferable, we pursuit upper bounds because we think that the lack of knowledge on good upper bounds for the problem is one of the reasons why we could not obtain a good lower bound.
It is folklore that 
can be computed by a 
circuit (hence also by a 
circuit) of size 
by applying the inclusion-exclusion formula. Namely,
 |
To the best of our knowledge, no asymptotically better bound has not been published. Note that 
has 2n input variables and the construction via the DNF representation of 
needs 
gates.
In this work, we show that 
has a depth-two threshold circuit of size significantly smaller than 
. Namely, we give an explicit construction of a 
circuit of size 
as well as a 
circuit of size 
computing 
.
The second contribution of this work is to give a new lower bound on the size of depth-two threshold circuits with some special restriction on the bottom gates. A symmetric gate is a gate that takes Boolean inputs whose output is depending only on the number of one’s in inputs. Let 
denote depth-two circuits consisting of a threshold gate with unbounded weights at the top and symmetric gates at the bottom.
In [7], Forster established a breakthrough result that the sign-rank of the 
Hadamard matrix is 
. Here the sign-rank of a matrix 
with nonzero entries is the least rank of a matrix 
with 
for all i and j. By combining this result and a simple fact that the communication matrix of any symmetric function has rank at most 
, Forster et al. [8] established an 
lower bound on the size of 
circuits for 
.
In this paper, we improve their bound to 
. Although the improvement is somewhat limited, our method has a unique feature; the lower bound is obtained by giving an explicit feasible solution to a certain linear programming problem.
Over a decade ago, building on the work of Basu et al. [3], the author developed an LP-based method to obtain a lower bound on the size of 
circuits for 
[1]. In [1], we showed that the problem of obtaining a lower bound on the size of such circuits can be reduced to the problem of solving a certain linear programming problem. Then we solved an obtained linear programming problem over 
variables using an LP solver to establish a lower bound of 
on the size of 
circuits for 
. However, the problem of determining a highest lower bound that can be obtained by our LP-based method was left as an open problem in [1].
In this work, we show that this limit is in fact 
, surpassing the 
bounds obtained by the sign-rank method. We achieve this by giving an explicit feasible solution to the dual of the linear programming problem presented in [1] and estimating the value of the objective function. Showing this is an actual contribution of the second part of this work.
The rest of the paper is organized as follows. In Sect. 2, we introduce some notations. In Sect. 3, we give new upper bounds on the size of depth-two threshold circuits for 
. Then in Sect. 4, we review an LP-based lower bounds method presented in our previous work [1], and establish a new lower bound on the size of 
circuits for 
.
Preliminaries
For an integer 
, [n] denotes the set 
. The inner product mod 2 function
is a Boolean function over 2n variables defined by
 |
For a Boolean predicate P, let 
denote the Iverson bracket function defined as 
if P is true and 
if P is false.
Let 
be Boolean variables. A linear threshold function is a Boolean function of the form
 |
for some 
. Similarly, an exact threshold function is a Boolean function of the form
 |
We call 
the weights and t the threshold. It is well known that, without loss of generality, we can assume that the weights and the threshold are integers of absolute value 
[15]. Hence, hereafter, we assume that the weights and the threshold are all integers. A gate that computes a linear threshold function is called a threshold gate. The class of all linear threshold functions (exact threshold functions, respectively) is denoted by 
(
, respectively).
As usual, a depth-two circuit such that the top gate computes a function in 
, and every bottom gate computes a function in 
is called a 
circuit. For example, a 
circuit is a depth-two circuit with threshold gates of unbounded weights in both layers. The size of a depth-two circuit is defined to be the number of gates in the bottom layer. The size complexity of a Boolean function f for 
circuits is the minimum size of a 
circuit computing f.
A majority gate is a gate computing a linear threshold function with additional restriction that 
for all i. Here the threshold t can be an arbitrary value, i.e., is not restricted to be the half of the number of input variables. The class of functions computed by a majority gate is denoted by 
. In our definition, a majority gate is allowed to read a variable multiple times. For example, we can say that the function
 |
is computed by a majority gate of fan-in 
. Remark that a majority gate is often defined as a gate that computes a linear threshold function with polynomially bounded weights. If we adapt this definition of majority gates, the size complexity may be reduced by at most a polynomial factor. However, such a difference will not affect all the results described in this paper.
A function 
is called symmetric if the value of f depends only on the number of ones in the input. A gate that computes a symmetric function is called a symmetric gate and the class of all symmetric functions is denoted by 
. Note that a symmetric gate is usually defined as a Boolean gate, i.e., it outputs a binary value. In this paper, we extend the domain from 
to 
. By this extension, the set of symmetric functions turns out to be closed under linear combinations. This property is useful when we view a threshold-of-symmetric circuit as (the sign of) a real polynomial (see Sect. 4.1). Note also that a symmetric gate can simulate all of AND, OR, the modulo gate. It can also simulate a restrict version of the majority gate where the gate reads each variable at most once and all the weights are restricted to be 1.
Upper Bounds
In this section, we give upper bounds on the size of depth-two threshold circuits for 
, which is significantly smaller than a folklore bound of 
.
We begin with two simple lemmas about exact threshold functions. Both lemmas were appeared in [11].
Lemma 1
[11]. Suppose that a Boolean function f can be computed by a 
circuit of size s. Then, f can be computed by a 
circuit of size at most 2s. The same relationship holds for 
and 
circuits.
Lemma 2
[11]. The AND of an arbitrary number of exact threshold functions is also an exact threshold function. In other words, the class of exact threshold functions is closed under the AND operation.
Before stating our upper bounds, we describe an idea of our construction. Consider the function 
. Define two exact threshold functions 
and 
as follows.
 |
It is easy to verify that
 |
where sgn(v) is defined to be 0 if 
and is 1 if 
.
Then, when n is even, 
is given by
 |
1 |
By expanding the product in Eq. (1), we can obtain a polynomial of 
terms in which each term is an AND of exact threshold functions. By Lemma 2, we can express each term by a single 
gate. Therefore, we have a 
circuit of size 
for 
, and also have a 
circuit of the same order by Lemma 1.
It is natural to expect that we can obtain a better bound by considering 
for 
as a base case. These ideas can be summarized as the following theorem.
Theorem 3
Let k be a positive integer. We write 
and 
. Suppose that 
can be represented by the sign of the linear combination of 
exact threshold functions where all weights are integers, i.e.,
 |
where 
and 
for 
. Then,
The size complexity of
for 
circuits as well as 
circuits is 
,The size complexity of
for 
circuits is 
.
Proof (Sketch). First, observe that 
is just a PARITY of n/k copies of 
. Replace each 
with a constructed 
-gate 
circuit. The PARITY of n/k
of 
s can be written as the sign of the product of n/k sums of 
s. Applying distributivity to the product of sums, we get a sum of 
products of 
s. But the product of a bunch of 
s can be written as one 
, so we get a 
of 
s, completing the proof of Statement 1 of the theorem. The proof for Statement 2 is similar. 
With the aid of computers, we found a formula of length 8 for 
as well as a formula of total weight 13 for 
that lead us to the following theorems.
Theorem 4
The size complexity of 
for 
circuits (and also for 
circuits) is 
.
Theorem 5
The size complexity of 
for 
circuits is 
.
Proof of Theorem 4. Let 
denote the input variables for 
. For 
, we write 
. We introduce the following seven exact threshold functions and write them as 
.
 |
It is elementary to verify that
 |
This gives a desired bound by Theorem 3. 
Proof of Theorem 5. Let 
denote the input variables for 
. For 
, we write 
. We introduce the following twelve exact threshold functions and write them as 
.
 |
It is elementary to verify that
 |
This gives a desired bound by Theorem 3. 
It is plausible that our bounds can further be improved by considering 
for 
as a base case. We remark that, for the case of 
circuits, the following argument says that there is a barrier at 
: The proof of Theorem 5 actually gives a construction of 
circuits for 
. By applying the “discriminator lemma" developed in [10] carefully, we can prove an 
lower bound on the size complexity of 
for 
circuits. Currently, we do not know such a barrier for 
circuits.
Lower Bounds for 
Circuits
In this section, we show 
lower bounds on the size of depth-two circuits for 
where the top gate is a threshold gate and the bottom gates are symmetric gates. In Sect. 4.1, we review our LP-based method presented in our previous work [1], and then we establish the lower bound in Sect. 4.2.
Throughout this section, we label the input variables of 
as 
and define 
(mod 2). This indexing is different from the one used in the previous section, but will be convenient for a later discussion.
LP-Based Method for Lower Bounds on Circuit Size
As defined before, we call a depth-two circuit with unbounded weights threshold gate at the top and symmetric gates at the bottom as a 
circuit. For a Boolean function f, the size complexity of 
for 
circuits is simply denoted by s(f). Throughout of this section, we treat a 
circuit as the sign of a real polynomial.
Definition 6
We say that a real polynomial 
sign represents a Boolean function f on n variables if, for every 
,
 |
We consider a polynomial 
 |
2 |
where 
and 
is a symmetric function over the set of variables S. The support of P is defined by 
. Obviously, s(f) is equal to the minimum size of the support of a polynomial P of the form (2) that sign represents f.
A point of our method is to define the parameter 
, which gives a lower bound on s(f), by introducing a certain linear programming problem.
Recall that the input variables of 
is 
.
Let 
. For 
, the parameter 
is defined inductively (on k) such that 
is the minimum value of the objective function of the following linear programming problem. Let 
be the first 2k variables of X. The program has 
real-valued variables 
and 
constraints.
 |
3 |
The key observation is the following.
Fact 7
([1]). Suppose that 
. Let 
and 
be real numbers such that 
and 
for every n. Let 
be the minimum value of the objective function of the LP problem (3). Then 
The following corollary is immediate from Fact 7.
Corollary 8
([1]). For every 
, 
. 
In the following, we give a sketch of the proof of Fact 7 for completeness.
Let 
be a real function and 
be a partial assignment to X. Let 
denote the set of variables that assigned a constant by 
, i.e., 
. The restriction of f by 
, denoted by 
, is the function obtained by setting 
to 
if 
and leaving 
as a variable otherwise.
The restriction of a polynomial P of the form (2), denoted by 
, is defined similarly. First, replace each 
in P by 
, which is a symmetric function over the set of variables 
. Then, if there are two (or more) functions 
and 
such that 
, then they are merged into a single symmetric function. This is possible by the fact that the linear combination of two (or more) symmetric functions over the same set of variables is also a symmetric function.
For a polynomial P of the form (2), we decompose P into 
’s for 
in such a way that
 |
Let 
be the number of terms in 
. Note that
 |
and
 |
We use the following fact that is easy to verify but useful.
Fact 9
([1]). Let 
and 
be two partial assignments such that 
. Then, 
implies 
. 
Proof of Fact 7 (sketch). Suppose that a polynomial P of the form (2) sign-represents 
. In what follows, we consider two types of pairs of partial assignments.
Choose 
and then choose 
. The unchosen variable in 
is denoted by v. Let 
and 
be two partial assignments such that 
, 
and 
.
A key observation is that for every such pair of partial assignments 
, we have 
and 
. This implies that the polynomial 
sign represents 
. Fact 9 says that 
is vanished if 
. Hence, we have
 |
4 |
where the last inequality follows from the assumption in the statement of Fact 7. Let 
for 
. By dividing both side of (4) by 
, we have
 |
which is the first constraint in the LP problem (3).
We also consider another type of partial assignments.
Choose 
such that 
, and then choose 
and 
. Let 
and 
be the unchosen variables in 
and 
, respectively. Let 
and 
be two partial assignments such that 
, 
and 
.
Similar to the case of Type 1, we have 
and 
, and hence 
sign represents 
. In addition, 
is vanished if 
is zero or two. Hence, we have
 |
5 |
where the last inequality follows from the assumption of the statement in Fact 7. This inequality is equivalent to
 |
which is the second constraint in the LP problem (3).
If P is an optimal polynomial for 
, then
 |
which is equivalent to
 |
Therefore, the minimum value 
of the objective function of the LP program (3) satisfies 
. This completes the proof of Fact 7. 
The LP problem (3) can easily be generated and solved by using a computer when k is small. In our previous work [1], we have succeeded to solve these problems by an LP solver for 
(see Table 2). During this work, we could extend the table up to 
. The best lower bound obtained in this way is 
, but still weaker than a bound of 
due to Forster et al. [7, 8].
Table 2.
The values of 
and 
for 
. The numbers shown in the table are truncated (not rounded) at the third or fourth decimal places.
| n | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|
|
1.5 | 2 | 2.833 | 4.027 | 5.750 | 8.254 | 11.970 | 17.335 | 25.207 |
|
1.2247 | 1.2599 | 1.2974 | 1.3213 | 1.3384 | 1.3519 | 1.3638 | 1.3729 | 1.3808 |
Obviously, the best possible lower bound that could be obtained by our approach is 
where 
. However, finding the value of 
was left as an open problem in [1].
New Lower Bounds on 
Circuits
In this section, we show that 
establishing a new lower bound on the size complexity of 
for 
circuits.
Theorem 10
For every 
,
 |
Hence, 
for every 
.
Although we only prove the lower bound, we strongly believe that our bound on 
is tight, i.e., 
. Note that 
by the construction described in Introduction and the fact that AND is contained in SYM. To the best of our knowledge, this is the best known upper bound on 
1.
Proof of Theorem 10. The proof is done by giving a feasible solution to the dual of the LP problem (3), and then estimating the value of the objective function.
We define 
to be
 |
For 
and 
, let 
denote the v’s bit of 
.
The dual of (3) is given by
 |
6 |
The LP duality theorem guarantees that the maximum value of the objective function in this dual program (6) equals to 
. Since LP (6) is a maximization problem, any feasible solution gives a lower bound on 
.
Here we present a feasible solution to LP (6) that will be analyzed in the proof. Define
 |
as follows: For 
, let 
if v is odd and 
if v is even. For 
, let 
if both of u and v are odd and 
otherwise. Note that we inspired this solution through actually solving LP (6) using an LP solver.
In order to show the feasibility of 
, it is enough to verify that the first constraint in LP (6) is satisfied. For 
, let
 |
Then, for 
, the first constraint in LP (6) can be written as
 |
This can easily be verified by observing that the LHS of this equation is equal to 
, completing the proof of the feasibility of 
.
We proceed to the estimation of the value of the objective function.
The proof is by the induction on k. For 
, we can verify the theorem by a direct calculation (see Table 2). Suppose that 
. By the definition of 
and the inductive assumption, we have
 |
7 |
By an elementary but somewhat lengthy calculation, we can show that
 |
as desired. The detailed calculations are omitted due to the page restriction and will appear in the full version of the paper. 
Acknowledgement
The author would like to thank anonymous referees for their helpful comments and suggestions. This work is supported in part by JSPS Kakenhi 18K11152 and 18H04090.
Footnotes
Actually, this is true only in an asymptotic sense. For example, an exhaustive computation shows 
, 
, 
and 
.
Contributor Information
Alberto Leporati, Email: alberto.leporati@unimib.it.
Carlos Martín-Vide, Email: carlos.martin@urv.cat.
Dana Shapira, Email: shapird@g.ariel.ac.il.
Claudio Zandron, Email: zandron@disco.unimib.it.
Kazuyuki Amano, Email: amano@gunma-u.ac.jp.
References
- 1.Amano K, Maruoka A. On the complexity of depth-2 circuits with threshold gates. In: Jȩdrzejowicz J, Szepietowski A, editors. Mathematical Foundations of Computer Science 2005; Heidelberg: Springer; 2005. pp. 107–118. [Google Scholar]
- 2.Amano K, Tate S. On XOR lemmas for the weight of polynomial threshold functions. Inf. Comput. 2019;269:104439. doi: 10.1016/j.ic.2019.104439. [DOI] [Google Scholar]
- 3.Basu S, Bhatnagar N, Gopalan P, Lipton R. Polynomials that sign represent parity and descartes’ rule of signs. Comput. Complex. 2008;17(3):377–406. doi: 10.1007/s00037-008-0244-2. [DOI] [Google Scholar]
- 4.Bruck J. Harmonic analysis of polynomial threshold functions. SIAM J. Discrete Math. 1990;3(2):168–177. doi: 10.1137/0403015. [DOI] [Google Scholar]
- 5.Chattopadhyay, A., Mande, N.: A short list of equalities induces large sign rank. In: Proceedings of FOCS 2018, pp. 47–58 (2018)
- 6.Eldan, R., Shamir, O.: The power of depth for feedforward neural networks. In: Proceedings of COLT 2016, pp. 907–940 (2016)
- 7.Forster J. A linear lower bound on the unbounded error probabilistic communication complexity. J. Comput. Syst. Sci. 2002;65(4):612–625. doi: 10.1016/S0022-0000(02)00019-3. [DOI] [Google Scholar]
- 8.Forster J, Krause M, Lokam SV, Mubarakzjanov R, Schmitt N, Simon HU. Relations between communication complexity, linear arrangements, and computational complexity. In: Hariharan R, Vinay V, Mukund M, editors. FST TCS 2001: Foundations of Software Technology and Theoretical Computer Science; Heidelberg: Springer; 2001. pp. 171–182. [Google Scholar]
- 9.Goldmann M, Håstad J, Razborov A. Majority gates vs. general weighted threshold gates. Comput. Complex. 1992;2:277–300. doi: 10.1007/BF01200426. [DOI] [Google Scholar]
- 10.Hajnal A, Maass W, Pudlák P, Szegedy M, Turán G. Threshold circuits of bounded depth. J. Comput. Syst. Sci. 1993;46(2):129–154. doi: 10.1016/0022-0000(93)90001-D. [DOI] [Google Scholar]
- 11.Hansen, K., Podolskii, V.: Exact threshold circuits. In: Proceedings of CCC 2010, pp. 270–279 (2010)
- 12.Jukna S. Boolean Function Complexity, Advances and Frontiers. Heidelberg: Springer; 2012. [Google Scholar]
- 13.Kane, D., Williams, R.: Super-linear gate and super-quadratic wire lower bounds for depth-two and depth-three threshold circuits. In: Proceedings of STOC 2016, pp. 633–643 (2016)
- 14.Krause M, Pudlák P. On the computational power of depth-2 circuits with threshold and modulo gates. Theor. Comput. Sci. 1997;174(1–2):137–156. doi: 10.1016/S0304-3975(96)00019-9. [DOI] [Google Scholar]
- 15.Muroga S. Threshold Logic and Its Applications. Hoboken: Wiley; 1971. [Google Scholar]
- 16.Nisan, N.: The communication complexity of threshold gates. In: Proceedings of “Combinatorics, Paul Erdős is Eighty", pp. 301–315 (1994)
- 17.Roychowdhury V, Orlitsky A, Siu KY. Lower bounds on threshold and related circuits via communication complexity. IEEE Trans. Inf. Theory. 1994;40(2):467–474. doi: 10.1109/18.312169. [DOI] [Google Scholar]
- 18.Safran, I., Eldan, R., Shamir, O.: Depth separations in neural networks: what is actually being separated?. In: Proceedings of COLT 2019, pp. 2664–2666 (2019). (full version at Arxiv:1904.06984)
- 19.Sezener C, Oztop E. Minimal sign representation of boolean functions: algorithms and exact results for low dimensions. Neural Comput. 2015;27(8):1796–1823. doi: 10.1162/NECO_a_00750. [DOI] [PubMed] [Google Scholar]