When the Density of the Noninteracting Reference System is not the Density of the Physical System in Density Functional Theory

Abstract

A major challenge in density functional theory (DFT) is the development of density functional approximations (DFAs) to overcome errors in existing DFA, leading to more complex functionals. For such functionals, we consider roles of the noninteracting reference systems. The electron density of the Kohn-Sham (KS) reference with a local potential has been traditionally defined as being equal to the electron density of the physical system. This key idea has been applied in two ways: the inverse calculation of such a local KS potential for the reference from a given density and the direct calculation of density and energy based on given DFAs. By construction, the inverse calculation can yield a KS reference with the density equal to the input density of the physical system. In application of DFT, however, it is the direct calculation of density and energy from a DFA that plays a central role. For direct calculations, we find that the self-consistent density of the KS reference defined by the optimized effective potential (OEP), is not the density of the physical system, when the DFA is dependent on the external potential. This inequality holds also for the density of generalized KS (GKS) or generalized OEP reference, which allows a nonlocal potential, when the DFA is dependent on the external potential. Instead, the density of the physical system, consistent with a given DFA, is given by the linear response of the total energy with respect to the variation of the external potential. This is a paradigm shift in DFT on the use of noninteracting references: the noninteracting KS or GKS references represents the explicit computational variables for energy minimization, but not the density of the physical system for external potential-dependent DFAs. We develop the expressions for the electron density so defined through the linear response for general DFAs, demonstrate the results for orbital functionals and for many-body perturbation theory within the second-order and the random-phase approximation, and explore the its connections to developments in DFT.

1. Introduction

As the fundamental variable in the density functional theory (DFT)^1-4, the ground-state electron density of a physical system fully characterizes, by virtue of the Hohenberg-Kohn theorem¹, the time-independent Hamiltonian (up to an irrelevant additive constant) of the many-electron physical system, which, in turn, determines the properties of the system. The Kohn-Sham (KS) method² uses the density of the noninteracting reference system to represent the electron density of the physical system and to carry out the energy minimization. It forms the backbone of most applications of DFT. It leads to wide-range applications of local density functional approximation (LDA)², (semi)local generalized gradient approximations (GGA)^5-7 and hybrid functionals^6,8-12.

However, commonly used density functional approaximations (DFAs) such as local density approximation (LDA), GGA and hybrid functionals face challenges in describing van der Waals interactions, reaction barriers, fundamental gaps, and strong correlation^13-16. Hence, constructing better approximations to the exchange-correlation (XC) energy and thereby achieving higher accuracy in practical applications have been the constant pursuit in DFT, with increased complexity of functionals and more variables. Beyond GGA, Hybird functionals^6,8-12 and meta-GGA¹⁷ depend explicitly on the one-electron orbitals. Developments based on many-body perturbation theory led to functionals that are explicitly dependent on the occupied and virtual orbitals and orbital energies of the one-electron equation, which in turn depends on the external potential.Thus the explicit variables in these functionals are the orbitals and the external potential. This comes mainly from the use of the many-electron perturbation theory. The second-order Moller-Plesset (MP2)^18-23, double hybrids^24-27, and random phase approximations (RPA) in both particle-hole (ph) channel^28-37 and particle-particle (pp) channel^36,38-42 significantly improve the performance of DFT on a wide range of properties, such as noncovalent interactions and reaction barriers.

The recently developed localized orbital scaling correction (LOSC)^43,44 and fractional-spin LOSC (FSLOSC)⁴⁵ also depend on orbitals and the external potential (or orbital energies). LOSC functionals reduce the delocalization error and static correlation error inherent in commonly used DFAs by imposing the Perdew-Parr-Levy-Balduz (PPLB) condition^46-49 and the flat-plane condition of Mori-Sanchez, Cohen and Yang⁵⁰ on each local region. Orbitalets, a set of orbitals localized in both physical and energy spaces from mixing canonical orbitals that are close in energy, are used to capture local fractional-charge and fractional-spin information in LOSC and FSLOSC^43,45, thus leading to the dependence on orbitals and the external potential.

While being implicitly a functional of the density, the exchange-correlation functional approximation $E_{\text{xc}}^{\text{DFA}}$ can have explicit dependence on the noninteracting one-particle density matrix ${\rho }_s^{\sigma }(\mathbf{r},{\mathbf{r}}^{\prime })$, or one-electron orbitals {ϕ_pσ(r)}, and the external potential ν_ext(r), namely expressed as $E_{\text{xc}}^{\text{DFA}}[{\rho }_s^{\sigma }(\mathbf{r},{\mathbf{r}}^{\prime }),v_{\text{ext}}(\mathbf{r})]$ or $E_{\text{xc}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})]$, beyond the explicit dependence on the electron density as in LDA and GGA. Note that orbital energies can normally be expressed in terms of {ϕ_pσ(r)} and ν_ext(r) and are thus not independent variables. As always, the minimum of the total energy functional is the corresponding approximation to the ground state energy of the physical system. Is the minimizing noninteracting-system electron density equal to the corresponding approximation to the physical-system electron density associated with the energy functional approximation in use?

Surprisingly, as will be shown in this work, the answer to this key question depends on the functional form and the definition of the minimum, and the difference in the densities can be significant. In particular, for functionals dependent on the external potential, none of the existing direct self-consistent calculation approaches leads to a noninteracting reference density that is the corresponding DFA approximation to the physical-system density.

2. Theory

Let us recall that the electron density of the Kohn-Sham noninteracting reference with a local potential has been traditionally defined as being equal to the density of the physical system within the KS theory². This has been applied in two ways: the inverse calculation of the local potential that defines the Kohn-Sham non-interacting reference system from a given density for the physical system, and the direct calculation of density and energy of the physical system with density functional approximations (DFAs) in the minimization of the total energy functional.

The inverse calculation of the KS local potential from a given electron density of the physical system, independently of how the density is given, can be carried out in various ways^51-54. This approach, which we call now as the inverse optimized effective potential (Inv-OEP) for clarity, is well established – the obtained local potential defines the exact KS system when the exact density is used and yields the input density by definition. The orbitals and potential so generated are the Kohn-Sham orbitals and the local potential consistent with the original Kohn-Sham formulation and with whatever level of theory used for generating the input density (or even with experimental electron density) (Row 1 in Table 1). The inverse approach can also be carried out to generate reference systems with varying strength of electron interactions along the adiabatic connection ^53,55,56.

Table 1.

The agreement between ρ_s(r), the densities of the various noninteracting reference systems, and ρ(r), the densities of the physical system defined by the linear response, consistently with the DFA of various forms. Empty entry indicates that it is not yet known how to conduct the corresponding calculation. The dependence of the DFAs on the variables indicated are all explicit.

	Noninteracting Systems	Functional forms of DFAs
		$E_{\text{xc}}^{\text{DFA}}[{\rho }_s^{\sigma }(\mathbf{r})]$	$E_{\text{xc}}^{\text{DFA}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$	$E_{\text{xc}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})]$
Inverse	1. KS (Inverse-OEP) $v_s^{\sigma }(\mathbf{r})$	yes	yes	yes
Direct	2. KS $v_s^{\sigma }(\mathbf{r})$	yes
Direct	3. OEP $v_s^{\sigma }(\mathbf{r})$	yes	yes	no
Direct	4. GKS $v_s^{\sigma }(\mathbf{r},{\mathbf{r}}^{\prime })$	yes	yes
Direct	5. GOEP/OO $v_s^{\sigma }(\mathbf{r},{\mathbf{r}}^{\prime })$	yes	yes	no

The direct, self-consistent field (SCF) calculation of the ground state density and energy with a DFA is our main consideration in this work with focus particularly on the role of the non-interacting system density.

2.1. Functionals of Electron Density $E_{\text{xc}}^{\text{DFA}}[{\rho }^{\sigma }(\mathbf{r})]$

When the DFA is $E_{\text{xc}}^{\text{DFA}}[{\rho }^{\sigma }(\mathbf{r})]$, an explicit functional of the density, as in LDA and GGA, the derivation of KS theory² shows that the energy minimization can be achieved with the KS noninteracting reference system with self-consistent KS local potential for electrons with spin σ,

$$v_s^{\sigma }(\mathbf{r})=v_{ext}(\mathbf{r})+v_H(\mathbf{r})+\frac{\delta E_{xc}^{\text{DFA}}[{\rho }^{\sigma }]}{\delta {\rho }^{\sigma }(\mathbf{r})},$$ (1)

which is equal to the functional derivative of $E_{\text{xc}}^{\text{DFA}}[{\rho }^{\sigma }(\mathbf{r})]$ in addition to the contribution from the external potential ν_ext(r) and the classical Coulomb energy ν_H(r). The eigenstates of the KS Hamiltonian $h_s^{\sigma }=-\frac{1}{2}{\nabla }^2+v_s^{\sigma }(\mathbf{r})$ are the set of KS orbitals {ϕ_iσ}

$$h_s^{\sigma }{\varphi }_{p\sigma }={\epsilon }_{p\sigma }{\varphi }_{p\sigma },$$ (2)

and the associated noninteracting electron density, expressed in terms of the occupied KS orbitals {ϕ_iσ(r)} is

$${\rho }_s^{\sigma }(\mathbf{r})=\sum _i^{occ}{\varphi }_{i\sigma }^{\ast }(\mathbf{r}){\varphi }_{i\sigma }(\mathbf{r}).$$ (3)

Unless stated otherwise, we use σ for spin index, i, j../a, b.. for occupied/virtual orbitals, and p, q.. for general orbitals.

Clearly from the KS derivation for functional with explicit dependence on density, ${\rho }_s^{\sigma }(\mathbf{r})$ of Eq. (3) is the appropriate approximation to the physical-system electron density for such DFAs, beyond any doubt. Note that in present work, the notation for functional $E_{xc}^{\text{DFA}}[{\rho }^{\sigma }]$ means that $E_{xc}^{\text{DFA}}$ is explicitly expressed in terms of the electron density, either as a local/semilocal functional as in LDA/GGA, or nonlocal functional such as functionals describing van der Waals interactions^57,58. The original KS formulation is entirely applicable for such explicit functionals of electron density, local or nonlocal (Row 2 in Table 1).

2.2. Calculation of Density of a Physical System from General Functional Approximations

Complication arises when a DFA is an implicit functional of density, but depends explicitly on orbitals {ϕ_pσ}, and/or the external potential v_ext(r). In some DFAs, the orbital dependence can also come through the dependence on the one-electron density matrix of the noninteracting system

$${\rho }_s^{\sigma }(\mathbf{r},{\mathbf{r}}^{\prime })=\sum _i^{occ}{\varphi }_{i\sigma }(\mathbf{r}){\varphi }_{i\sigma }^{\ast }({\mathbf{r}}^{\prime }).$$ (4)

The local KS potential $v_s^{\sigma }(\mathbf{r})$ cannot be directly obtained from the explicit functional derivative of the DFA with respect to density. Instead, the commonly used optimized effective potential (OEP) method ^53,59-66, consistent with the original KS framework, provides an effective way of implementing the self-consistent minimization of these functionals of {ϕ_pσ}, and v_ext(r), which are implicit functionals of density. Importantly, the OEP approach yields the same density and local potential as in original KS approach when the DFA is an explicit functional of the density $E_{\text{xc}}^{\text{DFA}}[\rho (\mathbf{r})]$, demonstrating the consistency with the KS theory. In OEP, the local potential defining the reference system can be determined by an additional integral equation⁵⁹ or as the computational variable in the direct optimization of the total energy^54,63,67. For such OEP calculations, what is the approximation to the physical-system electron density consistent with the DFA?

Unlike wavefunction methods where the density can be obtained from the expectation value of the wave function⁶⁸, the wave function of the noninteracting reference system does not serve as an approximation to the wave function of the physical system. Therefore the linear response for N-electron system,

$$\rho (\mathbf{r})=\delta E_{v_{ext}}(N)∕\delta v_{\text{ext}}(\mathbf{r})$$ (5)

equal to the functional derivative of the energy with respect to the external potential, is the natural approach for calculating the density of the physical system in DFT. This has been widely used in wavefunction theory, because the linear response gives the exact density when the exact energy expression is used, or the approximate density consistent with the approximation theory^69,70. The linear response, however, gives a different density from that obtained from the expectation value of the wavefunction in most wavefunction methods such as MP2 and CCSD carried out in post-HF manner. Linear response theory has already been used in DFT to define the density of the physical system^71,72. Now consider a physical system with external potential v_ext(r) and its total energy in a general $\text{DFA}{E}_{\text{xc}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})]$,

$$\begin{gathered} E_{v_{ext}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\}]=T_s[{\rho }_s^{\sigma }(\mathbf{r},{\mathbf{r}}^{\prime })]+J[{\rho }_s(\mathbf{r})]+\int d\mathbf{r}{v}_{\text{ext}}(\mathbf{r}){\rho }_s(\mathbf{r}) \\ +E_{\text{xc}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})], \end{gathered}$$ (6)

where T_s is the noninteracting kinetic energy, ${\rho }_s^{\sigma }(\mathbf{r},{\mathbf{r}}^{\prime })$ is defined through Eq. (4), and J is the Hartree electrostatic energy. The OEP method aims to optimize a local effective potential $v_s^{\sigma }(\mathbf{r})$ in the one-electron eigenvalue equation, so that the resulting ${\rho }_s^{\sigma }(\mathbf{r})={\rho }_s^{\sigma }(\mathbf{r},\mathbf{r})$, ${\rho }_s^{\sigma }(\mathbf{r},{\mathbf{r}}^{\prime })$ and {ϕ_pσ(r)} minimize the total energy of Eq. (6)^59,63:

$$E_{v_{ext}}^{\text{DFA}}(N)=\underset{{\mathrm{v}}_{\mathrm{s}}^{\sigma }(\mathbf{r})}{\text{min}}{E}_{v_{ext}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\}].$$ (7)

The theoretical foundation for the energy optimization of Eq, (7) in terms of $v_s^{\sigma }(\mathbf{r})$, which is the dual variable of the electron density ${\rho }_s^{\sigma }(\mathbf{r})$, was established within the potential functional theory (PFT). PFT is a formulation parallel to KS DFT. It uses the potential as the basic variable instead of the electron density and eliminates the need to consider electron densities that are not ν-representable by a noninteracting system^73,74. An effective OEP solution was formulated by Yang and Wu using the direct optimization with the following construction^63,64 $v_s^{\sigma }(\mathbf{r})=v_0(\mathbf{r})+{\sum }_t{b}_t^{\sigma }g(\mathbf{r})$, where ν₀(r) is a fixed reference potential including ν_ext and a long-rang term, and {$b_t^{\sigma }$} are the coefficients for the linear combination of basis functions (g_t(r)}, with regularization when needed ⁵⁴. The availability of analytic energy derivatives with respect to the optimization variables {$b_t^{\sigma }$} leads to efficient optimization⁶³.

Note it is well known that the energy from the OEP minimization, Eq. 7, is higher than that from the direct minimization of energy with respect to the orbitals or the density matrix.

Following the previous use of linear response theory $\rho (\mathbf{r})=\delta E_{v_{ext}}^{\text{DFA}}∕\delta v_{\text{ext}}(\mathbf{r})$^71,72, we now define the density for the many-electron physical system, independent of the minimization approach and applicable to general DFA functional forms and consistent with the DFA used. This leads to

$$\begin{gathered} \rho (\mathbf{r})={\rho }_s(\mathbf{r})+\sum _{p\sigma }\left[\int d{\mathbf{r}}_3\frac{\delta E^{\text{DFA}}}{\delta {\varphi }_{p\sigma }^{\ast }({\mathbf{r}}_3)}\left(\frac{\delta {\varphi }_{p\sigma }^{\ast }({\mathbf{r}}_3)}{\delta v_{\text{ext}}(\mathbf{r})}\right)+c.c.\right] \\ +\frac{\delta E_{\text{xc}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})]}{\delta v_{\text{ext}}(\mathbf{r})} \\ ={\rho }_s(\mathbf{r})+\sum _{p,q\ne p,\sigma }\left[\int d{\mathbf{r}}_3\frac{\delta E^{\text{DFA}}}{\delta {\varphi }_{p\sigma }^{\ast }({\mathbf{r}}_3)}\sum _{q\ne p}{\varphi }_{q\sigma }^{\ast }({\mathbf{r}}_3)\frac{{\varphi }_{q\sigma }(\mathbf{r}){\varphi }_{p\sigma }^{\ast }(\mathbf{r})}{{\epsilon }_{p\sigma }-{\epsilon }_{q\sigma }}+c.c.\right] \\ +\frac{\delta E_{\text{xc}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})]}{\delta v_{\text{ext}}(\mathbf{r})} \end{gathered}$$ (8)

where the first term, ρ_s(r), comes from the explicit dependence on ν_ext, the second term from the implicit dependence of orbitals on ν_ext, and the third term from the ν_ext dependence of the DFA $E_{\text{xc}}^{\text{DFA}}$. Eq. 8 is our general result. Its detailed derivation and its specific results for different scenarios are given in SI. Note that in this work, the functional derivatives are taken with respect to the explicit variables involved and with other variables kept unchanged, if not otherwise specified

2.3. Functionals of Density Matrix $E_{xc}^{\text{DFA}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$

Now we consider explicit density matrix functional $E_{xc}^{\text{DFA}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$, such as meta GGAs, the exact exchange and hyrbid functionals. We can define the effective Hamiltonian as $\left(\delta E_{v_{ext}}^{\text{DFA}}∕\delta {\varphi }_{i\sigma }^{\ast }(\mathbf{r})\right)=n_{i\sigma }\langle \mathbf{r}\mid h_{eff}^{\sigma }\mid {\varphi }_{i\sigma }\rangle$, where $h_{eff}^{\sigma }$ is independent of the orbital index i and

$$\langle \mathbf{r}\mid h_{eff}^{\sigma }\mid {\mathbf{r}}^{\prime }\rangle =\delta E_{v_{ext}}^{\text{DFA}}[({\mathbf{r}}^{\prime },\mathbf{r})]∕\delta {\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r}).$$

Thus, for $E_{xc}^{\text{DFA}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$, the second term simplifies and the third term vanishes, Eq. 8 becomes (See SI)

$$\rho (\mathbf{r})={\rho }_s(\mathbf{r})+\sum _{i,a,\sigma }\left[n_{i\sigma }\langle {\varphi }_{a\sigma }\mid h_{eff}^{\sigma }\mid {\varphi }_{i\sigma }\rangle \frac{{\varphi }_{a\sigma }(\mathbf{r}){\varphi }_{i\sigma }^{\ast }(\mathbf{r})}{{\epsilon }_{i\sigma }-{\epsilon }_{a\sigma }}+c.c.\right]$$ (9)

The second term of this equation is just functional derivative of $E_{v_{ext}}^{\text{DFA}}[\rho ({\mathbf{r}}^{\prime },\mathbf{r})]$ with respect to ν_ext(r) implicitly through ${\varphi }_{i\sigma }^{\ast }(\mathbf{r})$. This is the sum over spin index of the derivatives

$$\frac{\delta E_{v_{ext}}^{\text{DFA}}[\rho ({\mathbf{r}}^{\prime },\mathbf{r})]}{\delta v_s^{\sigma }(\mathbf{r})}=\sum _{i,a}\left[n_{i\sigma }\langle {\varphi }_{a\sigma }\mid h_{eff}^{\sigma }\mid {\varphi }_{i\sigma }\rangle \frac{{\varphi }_{a\sigma }(\mathbf{r}){\varphi }_{i\sigma }^{\ast }(\mathbf{r})}{{\epsilon }_{i\sigma }-{\epsilon }_{a\sigma }}+c.c.\right]$$ (10)

because at the energy minimum, $v_{\mathrm{s}}^{\sigma }(\mathbf{r})=v_{ext}(\mathbf{r})+v_{Hxc}(\mathbf{r})$. The necessary condition for the OEP is

$$\frac{\delta E_{v_{ext}}^{\text{DFA}}[\rho ({\mathbf{r}}^{\prime },\mathbf{r})]}{\delta v_s^{\sigma }(\mathbf{r})}=0.$$ (11)

Thus theoretically, for OEP calculations of $E_{xc}^{\text{DFA}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$,

$${\rho }_s(\mathbf{r})=\rho (\mathbf{r}).$$ (12)

That is, the density of the OEP noninteracting reference system is equal to the density of the physical systems, as defined by linear response, for all DFAs of density matrix functional, $E_{xc}^{\text{DFA}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$, such as meta GGA, exact exchange and hyrbid functionals.

However, computationally, the OEP stationary condition, Eq. 11 can only be satisfied approximately with finite basis sets or grids. Thus in numerical calculations, the expression, Eq. 9, which shows the deviation from ρ_s(r), remains meaningful as the analytical expression for the density from linear response, regardless of the basis set used.

Another minimization approach, more efficient computationally, for explicit functional of density matrix $E_{\text{xc}}^{\text{DFA}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$ is the generalized Kohn-Sham (GKS) approach, which is commonly used for calculations of meta GGA, and hyrbid functionals. Note that the term GKS was first used in Ref.⁷⁵, but it was based on a different meaning: splitting the exchange contribution to the total energy into a screened, nonlocal and a local density component. However, GKS, as commonly used in application of hybrid functionals⁷⁶, long-range-corrected functionals^77-81 and other density matrix functionals⁸², has a different meaning: it is just the direct minimization of the total energy with full relaxation of the orthonormal one-electron orbitals and without requiring them to be the eigenstates of a local potential, just as in the Hartree-Fock theory and in the so-called Hartree-Fock-Kohn-Sham approach developed much earlier^3,83. It is in this context GKS is now used most frequently (e.g. Ref. ⁸⁴).

The minimizing GKS orbitals are instead eigenstates of a Hamiltonian $h_s^{\sigma }=-\frac{1}{2}{\nabla }^2+v_s^{\sigma }(\mathbf{r},{\mathbf{r}}^{\prime })$, with a nonlocal potential as in Hartree-Fock theory. Furthermore $h_s^{\sigma }=h_{eff}^{\sigma }$, where $\langle \mathbf{r}\mid h_{eff}^{\sigma }\mid {\mathbf{r}}^{\prime }\rangle =\delta E_{v_{ext}}^{\text{DFA}}[({\mathbf{r}}^{\prime },\mathbf{r})]∕\delta {\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})$. In GKS for such an density matrix functional $E_{\text{xc}}^{\text{DFA}}[{\rho }_s^{\sigma }[({\mathbf{r}}^{\prime },\mathbf{r})]$, the density of the physical system from the linear response is still given by our general result of Eq. 8, but its second and third term vanish because the GKS orbitals diagonalize $h_s=h_{eff}^{\sigma }$ and there is no explicit dependence of $E_{\text{xc}}^{\text{DFA}}$ on ν_ext(r). While GKS and KS (defined from Inv-OEP) are two different noninteracting reference systems: GKS with a nonlocal potential ν_s(r, r′) and KS (Inv-OEP) with a local potential $v_{\mathrm{s}}^{\sigma }(\mathbf{r})$. But both give the same density of the physical system consistent with the DFA (Row 4 and Row 1 in Table 1).

2.4. The Comparison of GKS with OEP for Density Matrix Functionals $E_{xc}^{\text{DFA}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$

For the direct calculation with $E_{\text{xc}}^{\text{DFA}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$, we compare further the two approaches, OEP and GKS noninteracting references. It is well known that GKS leads to lower total energies because of the additional flexibility of nonlocal one-electron potential, as in Hartree-Fock theory compared with the OEP for the exact exchange functional (which is just the Hartree-Fock exchange energy)⁵⁹.

The current work clarifies that, while the electron densities from GKS and OEP are different, they are both the density of the physical system, as defined by the linear response and consistent with the DFAs $E_{xc}^{\text{DFA}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$. There is a difference for finite basis set calculations of OEP as expressed in Eq. 9.

Previousely, GKS with nonlocal $v_s^{\sigma }(\mathbf{r},{\mathbf{r}}^{\prime })$ has been proved to have the HOMO (LOMO) eigenvalue equal to the chemical potential of electron removal (addition), ${\epsilon }_{HOMO}={\left(\partial E_{v_{ext}}^{DFA}∕\partial N\right)}_{v_{ext}}^{-}\left({\epsilon }_{LUMO}={\left(\partial E_{v_{ext}}^{DFA}∕\partial N\right)}_{v_{ext}}^{+}\right)$ by Cohen, Mori-Sánchez, and Yang⁸⁴. Chemical potentials are key physical properties of the system: Perdew, Parr, Levy, and Balduz proved that, for the exact functional, the chemical potential for electron removal (addition) is equal to −IP (−EA)⁴⁶. These two works combined^46,84 leads to the important conclusion that the HOMO-LUMO gap in GKS is the DFA prediction for the fundamental gap: ε_LUMO – ε_HOMO = IP – EA^84-86. Note that how well the GKS HOMO-LUMO gap ε_LUMO – ε_HOMO approximates the experimental fundamental gap IP – EA just reflects the quality of the DFA $E_{xc}^{\text{DFA}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$ used. Functionals with delocalization error systematically underestimate the fundamental gaps and functionals with reduced delocalization error reduces the error in the prediction of fundamental gaps.^84-86.

The important connections of orbital eigenvalues to the corresponding chemical potentials are established for GKS, not true with the OEP approach, lending support to the GKS approach for calculations with density matrix functionals $E_{xc}^{\text{DFA}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$.

2.5. Functionals of Orbitals and External Potential $E_{\text{xc}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})]$

Consider now $E_{\text{xc}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})]$ with explicit dependence on orbitals and ν_ext(r), as in MP2^18,19, double hybrids^24-27, and ph-RPA^28-37 pp-RPA^39-41 and LOSC^43,45. All these functionals are implicit functionals of ${\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})$, but it is not known how to directly take a functional derivative with respect to ${\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})$ to get an effective Hamiltonian $h_{eff}^{\sigma }$, because of the explicit dependence on {ϕ_pσ(r)} and ν_ext(r).

To find the energy minimum, one can carry out two different calculations: OEP and the orbital optimization (OO)^87-91, or equivalently the optimization of the generalized optimized effective potential (GOEP) $v_s^{\sigma }(\mathbf{r},{\mathbf{r}}^{\prime })$, which is nonlocal⁹².

With OEP, the effective potential is local, and stationary condition is

$$\frac{\delta E_{v_{ext}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})]}{\delta v_s^{\sigma }(\mathbf{r})}=0,$$ (13)

where

$$\begin{gathered} \frac{\delta E_{v_{ext}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})]}{\delta v_s^{\sigma }(\mathbf{r})}= \\ \sum _{p,q\ne p}\left[\int d{\mathbf{r}}_3\frac{\delta E_{v_{ext}}^{\text{DFA}}}{\delta {\varphi }_{p\sigma }^{\ast }({\mathbf{r}}_3)}\sum _{q\ne p}{\varphi }_{q\sigma }^{\ast }({\mathbf{r}}_3)\frac{{\varphi }_{q\sigma }(\mathbf{r}){\varphi }_{p\sigma }^{\ast }(\mathbf{r})}{{\epsilon }_{p\sigma }-{\epsilon }_{q\sigma }}+c.c.\right] \end{gathered}$$ (14)

which is an extension of Eq. (10). The second term of Eq. 8, the general expression for density, is zero theoretically because of Eq. 13. But the third term remains. Therefore, for OEP calculations, we have

$$\rho (\mathbf{r})={\rho }_s(\mathbf{r})+\frac{\delta E_{\text{xc}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})]}{\delta v_{\text{ext}}(\mathbf{r})}.$$ (15)

For functionals $E_{\text{xc}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})]$, OEP does not yield the appropriate approximation to the physical-system electron density ρ(r) from ρ_s(r) of the noninteracting OEP reference. Thus OEP violates the KS foundation that ρ_s(r) = ρ(r) for functionals dependent on the external potential (Row 3 in Table 1).

Now consider OO/GOEP. Note that for $E_{\text{xc}}^{\text{DFA}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$, the OO and GOEP approaches are equivalent to the GKS approach⁹². The GOEP approach was first developed for ph-RPA and pp-RPA and yields accuracy beyond the post self-consistent DFA calculations, particularly for weakly bound systems⁹². Such improvement was not observed with the related OEP approach^93,94.

In GOEP⁹² and OO^87-91, the ground state energy is

$$\begin{gathered} E_v^{\text{DFA}}(N)=\underset{v_s^{\sigma }(\mathbf{r},{\mathbf{r}}^{\prime })}{\text{min}}{E}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})] \\ =\underset{{\varphi }_{p\sigma }(\mathbf{r})}{\text{min}}{E}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})]. \end{gathered}$$ (16)

Note that in the optimization with respect to the orbitals {ϕ_pσ(r)}, the orbital orthonormal constraints are implied. At the stationary point of GOEP or OO, the second term in Eq. 8 vanishes because the orthonormal orbitals are fully relaxed, Eq. 8 becomes Eq. 15. Thus Eq. 15 remain valid for both OEP, and for OO/GOEP, although at the minimum, the total energy of OO/GEOP is lower than that of OEP, and their densities are different. Therefore ρ_s(r) from GOEP is not the same as ρ(r) for ν_ext(r)-dependent functionals, such as MP2 and RPA (Row 5 in Table 1).

The difference in the density of the noninteracting reference and the density from linear response definition was discussed earlier in our work within a multireference approach⁷¹. We note that Furche and coworkers recently derived a similar equation as Eq. 15 to connect the two densities, and developed a new functional self-consistency condition so that the self-consistent calculations of a potential-dependent DFA has the same density as that of the physical system⁷². However, the functional self-consistency condition was only implemented approximately with projections, which cannot completely remove the discrepancy between these two densities⁷². Nonetheless, their method fully orbital-optimizes the RPA energy as the GOEP method⁹², thus the energies of both methods are the same and the resulting densities of the physical system from the linear response should be identical⁷². Our work here focuses on clarifying the densities from existing approaches and highlights the difference of the physical-system density from the densities of reference noninteracting systems.

Finally, we discuss another type of DFAs $E_{\text{xc}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\mathrm{s}}(\mathbf{r})]$, which are derived from many-electron perturbation theory and dependent on the entire set of orbitals {ϕ_pσ} and eigenvalues {${\epsilon }_{p\sigma }^{v_s}$}, where {${\epsilon }_{p\sigma }^{v_s}$} are the eigenvalues of OEP potential ν_s(r) ^23,93,95. It was found that the OEP optimization of MP2 functional can lead to nonphysically low total energies for some systems, because the optimization lead to small HOMO-LUMO gaps in the eigenvalues of OEP potential²³. For such DFA, there is no explicit dependence on ν_ext(r). At the OEP minimum, the energy derivative with respect to ν_s(r) vanishes, similarly to Eq. 13 and thus the density of OEP is the same as the density defined from linear response. We do not make separate column in Table 1 for such type of functionals. Replacing {${\epsilon }_{p\sigma }^{v_s}$} by the eigenvalues of a underlying DFA such as HF, or GGA in the optimization resolves this issue and leads to accurate total energy in the GOEP/OO optimization⁹². However, the eigenvalue dependence then leads to the dependence on ν_ext(r).

2.6. The Density of Noninteracting Reference System in Adiabatic Connections

The adiabatic connections (ACs) in DFT provide the expressions for the exchange-correlation energy in terms of many-electron quantities, such as the wavefunction or the two-particle density matrices^96,97. There are two different channels: the particle-hole channel^96,98 and the particle-particle channel³⁹. In the traditional formulation, the density of the reference systems with varying strengths of electron-electron interaction plays key roles: they are set to equal to the density of the physical system at the noninteracting limit^98,99, or equal to the density of the physical system at all strengths of electron-electron interactions^96,100. This is not inconsistent with the finding of the present work. The traditional ACs are formal and exact construction for the implicit functional of the electron density, without the explicit use of other variables such the noninteracting orbitals and the external potential.

However, when approximations are used and additional explicit variable, the external potential, is introduced in the exchange-correlation energy expression, such as in the many-electron perturbation theory, then the density of noninteracting reference system can no longer be set to equal to the density of the physical system, as shown in the present work. Instead, the linear response definition of Eq. 8 or Eq. 15 should be used. Using the linear response definition for the density of the physical system in the many-electron perturbation theory for the total energy introduces additional variables dependent on the orbitals of the reference system, as can be seen in the case of ph-RPA in Section 3. This certainly leads to significantly more complications for the total energy expression in the AC.

3. Numerical Results on MP2 and RPA Based DFAs and Discussions

To illustrate our idea, we derived and calculated the density of the physical system with ph-RPA, MP2 and pp-RPA within the GOEP or OO formulation and compare it with the noninteracting reference density; see SI for the detailed derivation. The generalized OEP (GOEP) method⁹² used in our calculations is equivalent to orbital optimization (OO)^87-91, with the full minimzation of the energy with respect to the occupied and virtual orbitals in the energy functionals under the orbital orthonormal constraints. For MP2, the one-electron eigenvalues in the energy expression are the eigenvalues of the HF Hamiltonian. For ph- and pp-RPA, the one-electron eigenvalues in the energy expression are the eigenvalues of the DFA one-electron Hamiltonian indicated: for example, ph-RPA@B3LYP for the use of the B3LYP eigenvalues in the ph-RPA energy.

For ph-RPA, the density of the physical system derived from Eq. 15 takes the following expression

$${\rho }^{\text{RPA}}(\mathbf{x})=\sum _i{\varphi }_i(\mathbf{x}){\varphi }_i^{\ast }(\mathbf{x})+\sum _{ij}{D}_{ij}^{\text{RPA}}{\varphi }_j(\mathbf{x}){\varphi }_i^{\ast }(\mathbf{x})+\sum _{ab}{D}_{ab}^{\text{RPA}}{\varphi }_b(\mathbf{x}){\varphi }_a^{\ast }(\mathbf{x}),$$ (17)

where the first term is the density of noninteracting reference system, the index i, j/a, b indicate occupied/unoccupied orbitals where x is a space-spin coordinate, $D_{ij}^{\text{RPA}}$ and $D_{ab}^{\text{RPA}}$ are the occupied-occupied and virtual-virtual blocks of the RPA density matrix (see SI, where we also present the corresponding expressions for pp-RPA).

We examine the quality of the densities by calculating the molecular observables associated with a single-particle local operator. Dipole moments of 7 molecules with different bonding natures were tested (see Table 2). B3LYP^5,6,8 was found to be one of the DFT methods that yield the best densities¹⁰¹, so B3LYP is also included here for comparison. HF cannot correct predict the right direction of the dipole for CO molecule¹⁰², while MP2 erratically overestimates the dipole of CO molecule, which should be attributed to the more serious "intruder state problem" in MP2 theory¹⁰³. Therefore, these results are not included in the statistics in Table 2. As mentioned above, for energy functionals that explicitly depend on the density or density matrix (HF and B3LYP), physical-system densities are the same as GKS noninteracting reference densities. We notice that HF provides the largest MAD, while B3LYP gives much better results. For MP2 and ph-RPA, physical-system densities, derived from the linear response theory, within the GOEP formulation differ from noninteracting reference densities because of the external potential dependence. Our test results show that physical-system densities are always better than the noninteracting reference densities for the testing set, and GOEP-RPA@B3LYP provides the best dipole moments.

Table 2.

Dipole moments calculated with different DFAs and methods. For GOEP-MP2 and GOEP-RPA methods, GOEP was used and “Phys” represents the physical-system electron density calculated from Eq. 15 and “Ref” represents the noninteracting reference electron density. MSD, MAD and MAPD refer to mean signed deviation, mean absolute deviation and mean absolute percentage deviation respectively. The experimental data are from¹⁰⁴. All units are in Debye.

		HF	B3LYP	GOEP-MP2		GOEP-RPA@HF		GOEP-RPA@B3LYP
	Expt	HF/GKS	GKS	Phys	Ref	Phys	Ref	Phys	Ref
H2O	1.855	1.977	1.848	1.838	1.881	1.886	1.911	1.827	1.885
HF	1.820	1.921	1.796	1.786	1.829	1.827	1.853	1.771	1.828
HCl	1.080	1.181	1.087	1.113	1.125	1.100	1.110	1.064	1.090
CO	0.122	NA	0.091	NA	NA	0.039	−0.026	0.124	−0.004
H2S	0.970	1.073	0.971	1.005	1.011	0.986	0.992	0.954	0.970
NH3	1.470	1.630	1.527	1.529	1.555	1.564	1.580	1.520	1.559
LiF	6.284	6.484	6.220	6.287	6.330	6.387	6.407	6.316	6.370
MSD		0.131	−0.009	0.013	0.042	0.027	0.032	−0.003	0.014
MAD		0.131	0.027	0.030	0.042	0.051	0.075	0.027	0.050
MAPD		7.8%	4.7%	2.3%	2.8%	11.7%	20.1%	1.8%	16.2%

For a graphical comparison, errors of densities from different DFAs are provided. In Fig. 1, we plot the physical densities of water molecule with PBE and GOEP-RPA@PBE. The corresponding dipole moments are reported in SI. The CCSD density is used as the reference. By comparison, PBE density shows significant difference from the CCSD density, especially in the region around the oxygen atom. In contrast, the linear-response defined density of the GOEP-RPA@PBE shows much small error around the oxygen atom. Therefore, GOEP-RPA@PBE provides a better density as compared to its parent functional PBE.

Fig. 1.

Error in the density of water molecule compared with CCSD. Left: ρ^PBE – ρ^CCSD, right: ρ^{GOEP–RPA@PBE}–ρ^CCSD. Isosurfaces are chosen at the value 0.002. Red represents positive error and blue represents negative error. As shown in Table I of SI, B3LYP provides even better dipole prediction than SCF-RPA@B3LYP. Apparently, B3LYP combined with pp-RPA can reach higher accuracy than its combination with ph-RPA in the prediction of CO dipole.

Finally, the difference between the physical density and the noninteracting reference density is compared for the difficult case of CO molecule. In Fig. 2, the difference between two densities from GOEP-RPA@B3LYP are plotted. From the figure, we observe a larger deviation of the noninteracting reference density as compared to the density from the linear response, which leads to the overestimation of the dipole moments from the reference density.

Fig. 2.

The difference between the density defined through linear response and noninteracting reference density of CO molecule from GOEP-RPA@B3LYP. ρ_s – ρ is plotted here at the isosurface value of 0.1. Large difference can be observed between the two densities.

4. Summary and Perspectives

In this work, we investigated the density of the physical system from the linear response of the total energy to the external potential, and its relationships to the noninteracting reference densities in OEP, GKS, OO and GOEP formulations with both theoretical derivation and numerical comparison. We summarize our theoretical findings in Table 1.

Independent of any DFA, there is always a Kohn-Sham non-interacting reference system (as given by Inv-OEP) that has the same density as that of the physical system, consistent with the density of the DFA considered, or even the exact functional if the exact density is used as the input in the Inverse-OEP calculation. This confirms the key KS idea that a noninteracting system can represent the electron density of a physical system. However, this KS (Inv-OEP) reference system is not involved with the SCF calculations of physical systems with a given DFA, as in all practical DFT calculations. It is only a fixed-density analysis.

For the direct SCF calculations with a given DFA, our present work leads to a different story. For any explicit density dependent $E_{\text{xc}}^{\text{DFA}}[{\rho }_s^{\sigma }(\mathbf{r})]$, the nonlocal effective potential reduces to a local potential, and GKS, OO, and GOEP are equivalent and give the same density as the conventional KS method. With other DFAs, explicit in the variables ${\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})$, or {ϕ_pσ(r)} and ν_ext(r), they can be all considered implicit functionals of ρ^σ(r), based on the Hohenberg-Kohn theory^1,3. But there is a clear difference between implicit and explicit dependence on ρ^σ(r): Energy minimization in the function space of ${\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})$, or {ϕ_pσ(r)} with GKS or GOEP and OO leads to a lower energy than KS or OEP, when it is not constrained to the KS framework of a local potential $v_s^{\sigma }(\mathbf{r})$.

For DFAs without dependence on the external potential, while the two computational approaches OEP and OO/GOEP lead to different total energies and different noninteracting densities ${\rho }_s^{\sigma }(\mathbf{r})$. The noninteracting density in each case is equal to the corresponding density of the physical system from the linear response for the same DFA. For the most complicated functional form considered, $E_{\text{xc}}^{\text{DFA}}[\{{\varphi }_{p\sigma }(\mathbf{r})\},v_{\text{ext}}(\mathbf{r})]$, none of the existing direct SCF calculation approaches leads to a noninteracting reference density that is the corresponding DFA approximation to the physical-system density.

Furthermore, our numerical tests show that the difference is significant, and is necessary for better density and density-related property prediction. The findings here are in stark contrast to the conventional wisdom for KS/GKS method where the noninteracting reference density has been traditionally considered as equal to the physical-system electron density. This is a fundamental paradigm shift in DFT on the use of the noninteracting reference systems, thus eliminating confusion and providing a theoretical basis for the future development of functionals. The noninteracting KS, GKS, OEP, and GOEP reference systems represents the explicit computational variables for energy minimization, but not the density of the physical system for DFAs dependent on the external potential.

We can also examine how to calculate electron density and the roles of the noninteracting reference systems in non-self-consistent DFT calculations, such as DFA calculations based on Hartree-Fock electron density¹⁰⁵, or the use of multi-reference wavefunctions in DFT to describe strong correlation^106,107. For such non-self-consistent calculations of DFAs with electron density/wavefunction obtained outside of the DFA used, there is no non-interacting reference system associated with self-consistency and the input and output densities are certainly different by definition. However, the linear response calculation of electron density of the physical system considered, Eq. 5, should be always applicable for any total energy expression, evaluated self-consistently or at a given density or wavefunction. In other words, the linear response calculation of electron density, Eq. 5, should lead to electron density that is consistent with the approximation used in the total energy expressions and different from the input electron density from Hartree-Fock calculations¹⁰⁵, or the multi-reference methods ^106,107.

The future development of functional approximations will continue the use of explicit variables like orbitals of the noninteracting reference system and the external potential. The additional degrees of freedom is needed for describing electron interactions more accurately. There is also a deeper mathematical and physical reason related to the form of the exact density functional. The exact density functionals can be defined rigorously^99,108, but they are always implicit, meaning that they are not expressed in terms of the density directly. It has already been shown that the exact exchange-correlation energy functional cannot either be expressed as a differentiable functional of electron density ${\rho }_s^{\sigma }(\mathbf{r})$ ^109,110, nor as a differentiable functional of the density matrix of the noninteracting reference system ${\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})$^50,86, either local or nonlocal. Thus, using ${\rho }_s^{\sigma }(\mathbf{r})$ or ${\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})$ as the only variables in the functionals, $E_{\text{xc}}[{\rho }_s^{\sigma }(\mathbf{r})]$ or $E_{\text{xc}}[{\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})]$ must have derivative discontinuities to describe all physical systems well (including strongly correlated systems)⁸⁶.

The derivative discontinuities are very difficult to construct directly with the variables ${\rho }_s^{\sigma }(\mathbf{r})$ or ${\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})$ ¹¹¹. However, using additional variables such as the orbiatls and the external potential provides the possibility for describing such discontinuities, as shown in capturing the flat-plane conditions of fractional charges and fractional spins⁵⁰ with the pp-RPA functionals ^36,38,39 and with the FSLOSC⁴⁵. This points to the direction of using additional variables beyond ${\rho }_s^{\sigma }(\mathbf{r})$ or ${\rho }_s^{\sigma }({\mathbf{r}}^{\prime },\mathbf{r})$ for future development of DFAs with general applicability. The use of all orbitals and the external potential can also maintain similar low computational cost of commonly used GGA as in LOSC and FSLOSC, without the higher computational scaling of many-electron perturbation theory.