Psi4 1.1: An Open-Source Electronic Structure Program Emphasizing Automation, Advanced Libraries, and Interoperability

Abstract

Psi4 is an ab initio electronic structure program providing methods such as Hartree-Fock, density functional theory, configuration interaction, and coupled-cluster theory. The 1.1 release represents a major update meant to automate complex tasks, such as geometry optimization using complete-basis-set extrapolation or focal-point methods. Conversion of the top-level code to a Python module means that Psi4 can now be used in complex workflows alongside other Python tools. Several new features have been added with the aid of libraries providing easy access to techniques such as density fitting, Cholesky decomposition, and Laplace denominators. The build system has been completely rewritten to simplify interoperability with independent, reusable software components for quantum chemistry. Finally, a wide range of new theoretical methods and analyses have been added to the code base, including functional-group and open-shell symmetry adapted perturbation theory (F-SAPT and O-SAPT), density-fitted coupled cluster with frozen natural orbitals [DF-FNO-CCSD(T)], orbital-optimized perturbation and coupled-cluster methods (e.g., OO-MP2 and OO-CCSD), density-fitted multiconfigurational self-consistent field (DF-MCSCF), density cumulant functional theory (DCT), algebraic-diagrammatic construction [ADC(2)] excited states, improvements to the geometry optimizer, and the “X2C” approach to relativistic corrections, among many other improvements.

Graphical Abstract

1. Introduction

Quantum chemical computations have become an indispensable part of molecular science. They can provide accurate molecular geometries, reaction mechanisms and energetics, and simulated spectra. Quantum chemistry also yields parameters needed to develop force-field models, and as such serves to anchor multiscale modeling efforts. Advances in electronic structure theory and in computer hardware have meant that quantum chemistry is now capable of giving very accurate results for systems with about three dozen atoms or less and reasonably accurate results for hundreds of atoms.

Unfortunately, electronic structure methods are inherently complicated, and the computer programs to implement them are quite lengthy. Fully featured quantum chemistry program packages can easily reach more than one million lines of source code. Additionally, it takes significant effort to optimize these codes so that they run as quickly as possible (an important consideration, given that computations are very time-consuming for larger molecules and/or more accurate models). Thus, most of the popular quantum chemistry programs have been developed over many years through the combined efforts of numerous programmers and often several research groups. This model of large, complex codes developed by multiple programmers over many years presents a tremendous challenge to keeping up with the rapid pace of recent innovations in computer hardware, including the increasing number of central processing unit (CPU) cores per node, the ubiquity of graphics processing units (GPUs), and emerging hardware like Intel’s Xeon Phi. It also presents a challenge to the rapid deployment of new techniques from electronic structure theory, because typically a desirable new feature is re-implemented for each quantum chemistry package.

We believe that this current development model is unsustainable and that the future of quantum chemistry software development lies in a more modular approach, in which small, independent teams develop reusable software components that can be incorporated directly into multiple quantum chemistry packages, for example, as a library or standalone program that has no dependency on any particular quantum chemistry program.¹ For example, libraries like Libint,² libefp,^3,4 CheMPS2,^5–7 simint,^8,9 Libxc,^10,11 dkh,^12,13 and PCMSolver^14,15 are written so that they can be called from multiple quantum chemistry programs, and programs like dftd3¹⁶ and MRCC^17,18 have been developed so that they are straightforward to interface to other packages. Several analysis and visualization tools (e.g., WebMO,¹⁹ Molden,^20,21 and VMD^22,23) have been written to analyze wavefunctions with data written in common formats. Over the past few years, we have substantially restructured and extended key portions of the Psi4 quantum chemistry package to facilitate interoperability with such reusable software components (or even with other large quantum chemistry programs).

Our build system, driver, and binary distribution system have been redesigned to make it easier for Psi4 to call independent software components. At the same time, we have also made Psi4 callable as a Python library, so that it can be incorporated into complex workflows driven by custom Python scripts or interfaced with other scientific Python projects, of which there has been an explosion in the past few years. Wrapping much of the C++ functionality of Psi4 by Python has not only enabled interoperability, but it has also led to more rapid prototyping and development of complex functionality. Interoperability has also been enhanced by the addition of functions to write out data in common file formats used by quantum chemistry analysis tools, e.g., Molden format for molecular orbitals and formatted checkpoint files.

By moving our development system to a publicly viewable GitHub²⁴ repository and by incorporating social coding practices, we have made it easier for independent developers to interface their work to Psi4. Completely independently of the Psi4 developers, interfaces to Psi4 have been developed for a number of projects, including GeomeTRIC,^25,26 a geometry optimization code that includes the translation-rotation-internal coordinate (TRIC) system; JANPA, a Java-based implementation of the Natural Population Analysis (NPA)²⁷ method;^28,29 an implementation³⁰ of the restrained electrostatic potential (RESP) method for fitting atomic charges;³¹ and htmd, a force-field parameterization tool.³²

We have used object-oriented design to make certain key functionalities available through a generic application program interface (API), while allowing optimizations or adaptations to various hardware to be done “under the hood” without necessitating changes to the top-level code that implements a particular quantum chemistry method. For example, this allows us to transparently switch among different libraries to compute electron repulsion integrals, depending on which is optimal for a particular type of hardware. Finally, we have also added libraries to enable techniques that are becoming more important in modern quantum chemistry, such as density fitting^33–41 and Cholesky decomposition.^42–46

Here we summarize our efforts to modularize and extend Psi4 along these lines. We also report on additional new capabilities of the package. This article, and its author list, reflects primarily the changes since the previous Psi4 publication,⁴⁷ written at the stage of an alpha release. The Psi4 1.1 release carries over much of the design, code, and capabilities of that earlier version of Psi4 and also retains some code and capabilities from Psi3.⁴⁸

2. Python Interface

In the alpha release of Psi4 in 2012,⁴⁷ we introduced a Python front-end to what had previously been an exclusively C/C++ program. All computationally intensive modules remain written in C++ for efficiency reasons, but user input parsing and simple, high-level functions that help drive the program are written in Python.

2.1. Using Python to Parse User-Friendly Input and Automate Multi-Step Computations

One of the design goals of Psi4 is to make the program as easy to use as possible, for both beginners and expert users. For beginners, a minimal input suffices for basic computations. Indeed, one does not even need to specify atomic coordinates — for common molecules, these can be obtained from the PubChem database (for nodes with internet connectivity).⁴⁹ An input file to optimize the geometry of the benzene molecule using Hartree-Fock self-consistent-field (SCF) with the STO-3G basis⁵⁰ is as simple as this:

molecule benzene{
pubchem:benzene
}
set basis sto-3g
optimize(“scf”)

For a graphical user interface (GUI), the popular WebMO program¹⁹ is capable of drawing molecules, generating text input files, managing a job queue, and visualizing orbitals and other results.

Although many solutions would have sufficed to parse input files like the example above, we also wished to allow much more complex input files by advanced users, who might want to perform multi-step computations, execute loops to build a potential energy surface, create tables and export data structures of results, or interface with other Python projects, all in the input file. In this, we were inspired by the Molpro program,⁵¹ which enables looping and post-processing operations.

Our approach is to use the Python language itself for input, yet allow free-form specification of common quantum chemical input (e.g., molecules, options, custom basis sets) by intercepting and parsing them into proper Python commands. In this way, basic users submit minimal input files without any required knowledge of Python, while the inclined can draw upon the full scientific Python ecosystem. This combination of simple quantum chemistry directives and user-specified Python code forms a domain-specific language, “Psithon,” that has shown itself an effective balance between power and ease-of-use since its introduction in alpha Psi4. Advanced users of quantum chemistry programs commonly write workflow-automating helper scripts to aggregate computations, drive supporting executables, or extract key quantities. With Psi4 such functionality can be expressed directly in an input file with conventional Python or, if a given functionality is popular, absorbed into the Psi4 driver.

2.1.1. Specifying Levels of Theory, Basis Set Extrapolations, Corrections for Basis Set Superposition Errors, and Many-Body Expansions of Clusters

In the 1.1 release, Psi4 has been significantly overhauled to make many common operations, which would normally be done by custom user scripts, as easy as specifying a single input string. For example, the common notation of computational chemistry that separates method and basis with a slash can be used, e.g., optimize(“SCF/STO-3G”). This idea can be extended to complete-basis-set (CBS) extrapolations, which can be expressed as energy(“SCF/cc-pV[D,T,Q]Z”) to perform an exponential extrapolation of the SCF energy using the cc-pVDZ, cc-pVTZ, and cc-pVQZ basis sets.⁵² A two-point extrapolation of the correlation energy can be specified in a similar way, e.g., energy(“MP2/aug-cc-pV[T,Q]Z”) for second-order Møller-Plesset perturbation theory (MP2) in augmented basis sets.^52,53 Depending on the method (SCF or post-SCF) and number of ζ-levels specified, extrapolation formulae are selected automatically among 2- and 3-point Helgaker schemes for SCF energies⁵⁴ and 2-point Helgaker for correlation energies.⁵⁵ Alternate or user-defined (by writing a Python function) extrapolation equations may be specified through additional keywords. For example, opt(“MP2/aug-cc-pV[T,Q]Z + D:CCSD(T)/cc-pV[D,T]Z”, corl_scheme=myxtpl_2) performs a focal point optimization with the coupled-cluster delta correction employing a custom formula for its 2-point extrapolation atop MP2 using the default 2-point Helgaker formula. Importantly, the 1.1 release also generalizes these extrapolations to work also for geometry optimizations and vibrational frequencies, e.g., frequencies(“MP2/cc-pV[D,T]Z”) for an MP2 extrapolated vibrational frequency computation.

In a similar way, corrections for basis set superposition errors (BSSE) can also be specified very simply: energy(“MP2/aug-cc-pV[T,Q]Z”, bsse_type=“cp”) for an MP2 extrapolated Boys-Bernardi counterpoise correction.⁵⁶ For a molecular cluster, the Valiron-Mayer functional counterpoise (VMFC) correction⁵⁷ is also available. Molecular clusters can also be analyzed in an N-body fashion in terms of dimers, trimers, etc. This is fully automated by Psi4’s Python driver and can be called by altering the bsse_type argument to the usual energy() function. Grep-able, tabulated, and programmatically accessible quantities for each requested fragment and basis set partitioning combination are made available, and multiple BSSE treatments can be computed concurrently.

Handling such complex operations with such minimal user input is the role of the Python driver portion of Psi4. It contains generic functions like energy(), optimize(), and frequencies(), which take a quantum chemistry method as an argument, then set appropriate options based on user specifications and best practices for the method. The driver then decides what C++ modules need to be called, and in what order. Results are passed back from the C++ modules to the Python driver so that they can undergo various types of post-processing (e.g., converting to other units, combining with results from other programs, exporting as data structures, etc.). For composite and otherwise complex methods, the main driver functions are often recursively called.

2.1.2. Passing Wavefunction Information

As the input to Psi4 computations is a single Python script, information may be passed through objects in main memory. The 1.1 release greatly improves information passing over the alpha release by allowing wavefunction objects to be passed between modules. For example, one could use density functional theory (DFT) orbitals as guess orbitals for a complete-active-space self-consistent-field (CASSCF) procedure:

scf_e, scf_wfn = energy(“bp86”, return_wfn=True)
energy(“casscf”, ref_wfn=scf_wfn)

Additionally, one can obtain any method’s wavefunction and pass it to generic objects that can write out or manipulate data: molden(), which writes Molden files; cubeprop(), which writes densities and electrostatic potentials on a grid; and OEprop(), which computes arbitrary one-electron properties. For example, writing the optimized-orbital MP2 orbitals in Molden format can be accomplished by the following:

omp2_e, omp2_wfn = energy(“omp2”, return_wfn=True)
molden(omp2_wfn, “omp2_orbitals.molden”)

2.2. Rapid Prototyping

Much of the C++ infrastructure of Psi4 has been made accessible at the Python layer. This provides opportunities for rapid prototyping of quantum chemical procedures in Python, while leveraging the existing, efficient C++ infrastructure. Indeed, for some procedures it is even possible to write the final, production-level version of the code by organizing C++ calls completely in Python. Interoperability between C++ and Python was previously accomplished using Boost Python,⁵⁸ but in the 1.1 release we have removed Boost due to its large size and build difficulties; we now use the header-only pybind11 library.⁵⁹

Psi4 modules such as the “JK” object that generates Coulomb and exchange matrices are readily available through Python:

# Create a JK object in the primary basis set
jk = psi4.core.JK.build(primary_basis)
# Provide an occupied orbital matrix for the density matrix:
# D = C_left * C_right(T); by default C_right = C_left
jk.add_C_left(Cocc)
# Note that there are many possible sources for Cocc, e.g. sphericalized atomic
# orbitals, a converged SCF, etc., and these details are up to the user
# Perform the computation and get the J and K matrices
jk.compute()
J = jk.J()
K = jk.K()

As JK builds are the bottleneck in SCF computations, a Python-based SCF algorithm can have nearly identical execution time to that of full C++ code. This is one of many cases where the most computationally intensive parts of calculations can be handled by a Psi4 library, resulting in top-level code that is easy to write, easy to understand, and computationally efficient.

The Psi4NumPy project⁶⁰ aims to enable the creation of clear, readable code for development or pedagogical purposes by combining the extensive Python accessibility of Psi4 with the popular Numerical Python (NumPy)⁶¹ library for high-level array manipulation and access to the BLAS library. The latest version of Psi4 adopts code from that project⁶⁰ allowing Psi4 data and NumPy objects to be seamlessly interconverted. For example,

np_array = numpy.zeros((5, 5))
psi4_matrix = psi4.Matrix.from_array(np_array)
new_np_array = numpy.array(psi4_matrix)

2.3. Opportunities for Education in Electronic Structure Theory

Writing quantum chemistry code in this manner allows for the decoupling of learning quantum chemistry from having to simultaneously learn a low-level programming language. As low-level languages typically have much steeper learning curves than high-level languages like Python, this can be quite advantageous. We have leveraged the Python accessibility of much of the Psi4 infrastructure to develop guided projects for students to write their own Hartree-Fock code; several students in Georgia Tech’s Computational Chemistry course have done this as a class project. One of us (D.G.A.S.) offered an introduction to electronic structure theory and programming in the summer of 2016 for summer students and new graduate students, utilizing Psi4 and Psi4NumPy.

2.4. Psi4 as a Python Module

Finally, the most recent improvement in Psi4 is its ability to be loaded into a Python script as a regular Python module. This ability has also opened up numerous opportunities to integrate Psi4 with other powerful Python modules and related tools. For example, we have been working on integrating Psi4 with the popular OpenMM package for molecular mechanics. As both of these programs have Python front-ends, passing data between programs is straightforward. Psi4 can also be run interactively through Jupyter notebooks,⁶² which are like Mathematica notebooks for Python; intermediate results can be graphed and analyzed with standard Python tools. We believe Jupyter notebooks hold promise as “lab notebooks” (or supplementary information) for computational research, as well as for guided computational chemistry laboratory exercises.

3. Interfaces to External Projects

The current model of quantum chemistry software development is one in which each quantum chemistry package is a large “silo” of code standing independently of all the others. While it is convenient for the user to have a package that contains many different features, it is wasteful for different teams of developers to duplicate the effort of adding new features. It would be better for specialists in a particular method to develop a reusable component that could be adopted by multiple programs. Psi4 strives to minimize the barrier to interoperability with these add-ons by keeping their build and maintenance back-ends separate from Psi4, yet maintaining a monolithic front-end for users. Since the beta versions of Psi4, all projects not written by Psi4 core developers for Psi4 have been removed from the code repository; the source instead resides in the author’s primary (upstream) repository where possible. In their place are single “External Project” files that, when activated by ENABLE_<AddOn> options, queue up the task of building that library from external source code using an internal configuration or common set of build parameters and shared dependencies. This “superbuild” framework is driven by the popular CMake tools.⁶³ The External Project scheme ensures that the footprint of each interfaced project within the Psi4 codebase consists of one build file and typically half-a-dozen lines of linking and include directives. For greatest flexibility, all quantum chemistry dependencies and add-ons that require linking or internal code activation can be built from source (downloaded from GitHub or research websites) during the Psi4 build. Or, existing pre-built libraries can be detected and seamlessly linked in to the final Psi4 library and plugin system. Wherever rational, shared and archive (static) libraries are equally supported. Whenever possible, the changes to an external project’s build system required by Psi4 have been accepted upstream to the main project. These typically consist of a few short files that should prove useful to non-Psi4 users of the add-ons also.

Forming a clear delineation between Psi4 code and external projects simplifies build maintenance but also permits better management and reuse of libraries so that core developers and users alike can access a full-featured Psi4 compilation without hassle. Invaluable in this endeavor is Anaconda,⁶⁴ a distribution of Python (and its associated package manager, conda) that focuses on the scientific community and the cross-platform accessibility of Python and Python/C++/Fortran binaries. Conda tools facilitate the construction of relocatable (installable into any directory) and generic (executable on almost any OS distribution) binary packages, which are hosted at https://anaconda.org/psi4 for Mac and Linux (and, through Ubuntu Terminal, recent versions of Microsoft Windows). To make binaries as foolproof as possible, while also maintaining the efficiency so vital to quantum chemistry computations, a number of techniques are employed, including static linking of high-quality math libraries, static linking of system libraries wherever possible,⁶⁵ resolving symbols to a widely available version of glibc, and turning off processor-specific optimizations (users wishing to obtain the full performance of processor-specific optimizations, which vary from negligible to considerable, can activate these by building the project from scratch). Conda packages are built for Psi4 itself and each of its add-ons. Each can be downloaded individually or an environment specification can install a complete set. For the Psi4 developer, pointing CMake to that install location allows all the add-on and dependency libraries to be detected and linked into Psi4 itself, drastically reducing the compile time and resulting in a full-featured Psi4 build. For someone who wishes to use the Psi4 libraries to develop quantum chemistry code as quickly as possible, C++ and Python Psi4 plugins can be developed from conda packages and conda compilers, without a local development environment. For the user, the Psi4 conda package, some half-a-dozen add-on packages, and still more non-quantum-chemistry dependencies are bundled up along with the conda package manager into a standalone installer, available from http://psicode.org/downloads.html, that unpacks into a Miniconda + Psi4 + AddOns release distribution.

For External Projects the Psi4 project will (1) leave control of their code under their purview; (2) maintain any interfacing code needed; (3) regularly run integration tests between Psi4 and their code; (4) build a mostly statically linked conda package so that any of their users can obtain a pre-built binary distribution through conda install addon --channel psi4; (5) provide a development sandbox for their code through Psi4 plugins; and (6) provide conda download counts independent of Psi4. This maintain-in-pieces build-as-whole model is working for add-ons that are executables, libraries, and plugins from languages C, C++, and Fortran (and could be extended to proprietary or restricted license add-ons), as seen in Figure 1.

Figure 1:

Psi4 creates an ecosystem for code, not a silo. Software packages required to build, required to run, or optionally provide additional capabilities to Psi4 are shown. All are available as conda packages from the default or psi4 channels (excepting pybind11, which is built internally, MRCC, ambit, and Forte).

Cognizant that the expanding interconnectivity of quantum chemical software demonstrated by Figure 1 does not end with Psi4 (especially now that Psi4 is a library more than an executable) and that Psi4 has benefited from the flexible licensing terms of our add-ons, the license for Psi4 itself has been relaxed from the GNU General Public License version 2 or later (GPL-2.0+) to the GNU Lesser General Public License version 3 (LGPL-3.0), thereby permitting Psi4 to be incorporated downstream without downstream projects inheriting the GPL. The required dependencies have been minimized so that it is possible to build a (non-full-featured) Psi4 without any GPL involvement, and a conda package meeting those restrictions is distributed.

The following is a list of external projects integrated with Psi4.

• dftd3: This independent executable program by S. Grimme provides damped dispersion corrections for use with density functional theory.¹⁶ The latest version incorporates modified damping parameters [-D3M and -D3M(BJ)] to better model short-range contacts as described by Smith et al.⁶⁶

• gCP: This “geometric counterpoise” executable by S. Grimme provides approximate corrections for basis set superposition error based on the the molecular geometry, allowing the implementation of the semiempirical methods HF-3c and PBEh-3c.^67,68

• PCMSolver: This library by Di Remigio, Frediani, and coworkers^14,15 implements the widely used polarizable continuum model (PCM) for solvation.^69–72 Within PCM, the mutual solute-solvent polarization is represented by an apparent surface charge σ spread over the cavity boundary and computed as the unique solution to the integral equation associated to the classical Poisson problem,⁷³ $\widehat{\mathcal{T}}\sigma =-\widehat{\mathcal{R}}\phi$, where the definition of the operators $\widehat{\mathcal{T}}$ and $\widehat{\mathcal{R}}$ depends on the molecular geometry and the Green’s function for the differential problem at hand. This mathematical framework can be applied to complex environments, such as homogeneous isotropic, ionic, and anisotropic liquids,⁷³ and systems with interfaces.^74–77 In the model, a molecular cavity built from a set of interlocking atom-centered spheres is partitioned into a mesh of (curvilinear or planar) finite elements, to each of which is attached a finite basis set of piecewise-regular polynomials that offer a natural discretization of the above equation into matrix form.

• libefp: This library by Kaliman and Slipchenko^3,4 implements the effective fragment potential (EFP) method of Gordon and co-workers^78,79 to provide ab initio force fields.

• MRCC: The high-order coupled-cluster methods available in the multi-reference coupled-cluster program of Kállay and co-workers^17,18 are accessible from Psi4.

• dkh: This library by Wolf, Reiher, and Hess implements the Douglas-Kroll-Hess relativistic corrections between 2^nd and 4^th order.^12,13

• Integrals: Psi4 is unusual in that it supports multiple electron repulstion integral (ERI) libraries. We continue to support the Libint library (version 1),² originally written for Psi3. However, an interface layer called libmints allows us to replace Libint with other integrals libraries, and we have written an interface to the erd library of Flocke and Lotrich.⁸⁰ Very recently, we have also interfaced to the new simint library of Pritchard,^8,9 optimized for architectures like the Intel Xeon Phi.

• CheMPS2: This library^5–7 presents a spin-adapted implementation of the density matrix renormalization group (DMRG) approach,^81,82 which is an effective way to approximate complete active space configuration interaction (CAS CI) with much larger active spaces (up to about 50 electrons in 50 orbitals for general problems). DMRG-based complete active space self-consistent field (DMRG-CASSCF)⁸³ and CASSCF plus second-order perturbation theory (DMRG-CASPT2)⁸⁴ are available through CheMPS2.

• gdma: Psi4 has been interfaced with the GDMA program of Stone,⁸⁵ which provides distributed multipole analysis of wavefunctions to represent the electrostatic field as a sum of contributions from point charges, dipoles, etc., at atomic centers or other sites.

• v2rdm_casscf: This plugin⁸⁶ (requiring Psi4) implements a semidefinite programming algorithm for the variational optimization of the ground-state two-electron reduced-density matrix (2RDM). This variational 2RDM (v2RDM) approach is most useful as an alternative to CAS CI for describing the electronic structure of an active space. Unlike CAS CI, v2RDM methods can be applied to large active spaces comprised of as many as 50 electrons in 50 orbitals. The plugin also provides a v2RDM-driven CASSCF procedure that exploits density fitting technology for the treatment of large numbers of external orbitals.⁸⁷

• ambit: This library by Turney and co-workers⁸⁸ provides tensors to represent quantum chemistry quantities such as electron repulsion integrals or coupled-cluster amplitudes.

• cfour: We have nearly completed a prototype interface that allows Psi4 to drive computations performed by the cfour package for coupled-cluster theory.⁸⁹

• Forte: This plugin (requiring Psi4) by Li, Hannon, Schriber, Zhang, and Evangelista⁹⁰ implements a number of multireference quantum chemistry methods, including approaches based on the driven similarity renormalization group (DSRG)⁹¹ with perturbative^92,93 and nonperturbative truncation schemes.⁹⁴ Forte also implements two new selected-CI approaches, the adaptive-⁹⁵ and projector-CI.⁹⁶

4. Development Process

Psi4 has adopted tools standard in the industry such as GitHub for collaboration and version control, Travis CI⁹⁷ for continuous integration testing, and CodeCov⁹⁸ for test coverage metrics. This ensures the primary source code retains its reproducibility and all code changes can be reviewed by the entire Psi4 community. New developers feel safe making changes to the base code, as all changes will be publicly visible and reviewed by the core team through the GitHub “pull request” mechanism, and the changes will also be tested automatically by the Travis CI system. The public visibility of the GitHub project helps communication between developers, both within the core team and also with independent developers who want their code to work with Psi4. Several improvements and bug fixes have been contributed back to the Psi4 development team from the broader community.

5. Libraries

5.1. Density fitting

One of the major barriers to efficient electronic structure computations is the large number of ERIs. For a set of one-particle spatial basis functions $\left\{{\varphi }_p\left({\overrightarrow{r}}_1\right)\right\}$ (e.g., Gaussian atomic orbitals, molecular orbitals), the ERIs are defined as

$$\left(pq\mid rs\right)\equiv {\iint }_{{ℝ}^6}{\text{d}}^3{r}_1{\text{d}}^3{r}_2{\varphi }_p\left({\overrightarrow{r}}_1\right){\varphi }_q\left({\overrightarrow{r}}_1\right)\frac{1}{r_{12}}{\varphi }_r\left({\overrightarrow{r}}_2\right){\varphi }_s\left({\overrightarrow{r}}_2\right),$$ (1)

where ${\overrightarrow{r}}_1$ and ${\overrightarrow{r}}_2$ denote the Cartesian coordinates of electrons 1 and 2, respectively, and r₁₂ is the distance between electrons 1 and 2. Each ERI encapsulates the electrostatic interaction between a particle in the “pair-space” basis function ${\rho }_{pq}\left({\overrightarrow{r}}_1\right)\equiv {\varphi }_p\left({\overrightarrow{r}}_1\right){\varphi }_q\left({\overrightarrow{r}}_1\right)$ in coordinate 1 with another particle in pair-space basis function ${\rho }_{rs}\left({\overrightarrow{r}}_2\right)\equiv {\varphi }_r\left({\overrightarrow{r}}_2\right){\varphi }_s\left({\overrightarrow{r}}_2\right)$ in coordinate 2. Despite the existence of a rich literature on the efficient evaluation of the ERIs in Gaussian orbitals,^99–101 the order-4 ERI tensor is a major bottleneck in terms of generation, storage and utilization.

A particularly elegant approach to reducing the impact of the 4-index nature of the ERIs in electronic structure theory begins from the observation that the quadratic-scaling pair-space basis $\left\{{\rho }_{pq}\left({\overrightarrow{r}}_1\right)\right\}$ is highly numerically redundant, particularly in large molecular systems and/or high-quality basis sets. Fortunately, the usual quadratic-scaling pair-space basis can be accurately represented in a linear-scaling “auxiliary basis” $\left\{{\chi }_A\left({\overrightarrow{r}}_1\right)\right\}$, e.g., ${\rho }_{pq}\left({\overrightarrow{r}}_1\right)\approx {\sum }_A{d}_{pq}^A{\chi }_A\left({\overrightarrow{r}}_1\right)$, where $d_{pq}^A$ is an order-3 fitting coefficient tensor. This is the crux of the extremely successful Cholesky decomposition (CD)^42–46 and density fitting (DF)^33–41 approaches. In particular, we focus on the now-ubiquitous “Coulomb-metric” DF approach, in which the ERI is approximated as

$$\left(pq\mid rs\right)\approx \sum _{AB}\left(pq\mid A\right){\left(A\mid B\right)}^{-1}\left(B\mid rs\right)\equiv B_{pq}^C{B}_{rs}^C.$$ (2)

In the last step, and below, we assume that repeated indices are summed over. The use of DF reduces the storage burden of the ERI tensor from order-4 to order-3. In addition, transforming the integrals from the atomic orbital to molecular orbital basis now becomes an $\mathcal{O}\left(n^4\right)$ operation instead of an $\mathcal{O}\left(n^5\right)$ operation. Moreover, the subsequent utilization of the ERI tensor in contractions with density matrices or wavefunction amplitudes is often considerably more efficient than with conventional integrals, due to reduced storage cost and the occasional opportunity to achieve formal scaling reduction by the “unpinning” of the pq and rs indices in the DF representation (i.e., this sometimes allows more efficient ways to order the steps in the tensor multiplications). Coulomb-metric DF is exceedingly accurate, with errors in relative properties of chemical interest (barrier heights, interaction energies, etc.) generally lower than 0.05 kcal mol⁻¹ when using standardized auxiliary fitting basis sets.⁴¹

Coulomb-metric DF is utilized in many electronic structure methods in Psi4, as discussed below. The use of DF to compress the molecular-orbital basis ERIs is so ubiquitous that we have introduced a DFERI helper class to quickly construct the DF factorization for a given set of orbital pair-spaces (including customized pair-spaces). An example of using the DFERI object to build the DF factorization of the hole-particle integrals $\left(ia\mid jb\right)\approx B_{ia}^Q{B}_{jb}^Q$ (where i and j indicate hole orbitals and a and b indicate particle orbitals) is presented below,

// Build a DFERI object with default options/orbital spaces
shared_ptr<DFERI> df = DFERI::build(primary,auxiliary,options);
// Request the decomposition (ia|jb) ~= B_iâQ B_jb^Q for MP2
df->add_pair_space(“MP2”, “ACTIVE_OCC”, “ACTIVE_VIR”);
// Compute the DF factorization
df->compute();
// Grab the DF tensor (on disk) for subsequent use
shared_ptr<Tensor> BiaQ = df->ints[“MP2”];

This helper class shields the user from the specifics of forming the DF integrals: parallelization over multiple cores, using Cauchy-Schwarz spatial sieving to remove insignificant pq pairs in the AO basis DF integrals, applying the atomic orbital to molecular orbital integral transform, locating the pseudoinverse of the fitting metric, and applying the fitting metric. Additionally, if multiple pair-space tasks are requested, the generation of the integrals, first-half integral transformation, and fitting metric inverse are coalesced to avoid redundant effort. Thus, in just a few lines of client code, the user has access to DF factorizations that are tractable on problems with up to ~ 6000 basis functions, and can then focus on the application of these DF tensors to implement their desired method.

5.2. JK object

A fundamental (and often quite computationally expensive) operation in myriad electronic structure methods is the construction of one-electron “Coulomb” (J_pq) or “exchange” (K_pq) matrices corresponding to a pre-specified generalized one-particle density matrix (D_rs):

$$J_{pq}\equiv \sum _{rs}\left(pq\mid rs\right)D_{rs},$$ (3)

$$K_{pq}\equiv \sum _{rs}\left(pr\mid qs\right)D_{rs}.$$ (4)

SCF, coupled-perturbed Hartree-Fock, configuration interaction singles, and many terms in symmetry-adapted perturbation theory and multiconfigurational self-consistent-field (MCSCF) can be formulated in terms of these matrices. There are many approaches to efficiently and/or exactly forming these matrices, including integral-direct, PK supermatrix,¹⁰² or density fitting. To hide the details of these approaches from the user, we have introduced an abstract JK class, implemented by any one of a number of DirectJK, DFJK, DiskJK, etc. classes. After construction, the user interacts with the JK object in an abstract manner, requesting J and K matrices without needing to know the details of how they are constructed. This means that any method coded in terms of the JK object automatically can use DF, Direct, or any other integral technology from the outset. An example of computing J and K matrices at the Python layer was presented above in Section 2.2.

The density matrices supplied to the JK object are actually specified in terms of their underlying factorization in terms of occupied orbital coefficients. This factorization naturally occurs in most places that J and K matrices appear (and can always be achieved by supplying the density and identity matrices as the factor tensors, if needed). However, endowing the JK object with knowledge of the factorization and symmetry of the density matrix often allows for significant performance gains, e.g., in DF Hartree-Fock.⁴¹

6. New Capabilities

6.1. Symmetry-Adapted Perturbation Theory (SAPT)

One of the specialties of Psi4 is a suite of symmetry-adapted perturbation theory (SAPT)^103–105 codes for the analysis of noncovalent interactions.¹⁰⁶ Previously, we had introduced efficient density-fitted and/or natural-orbital-accelerated implementations of SAPT0,^107,108 SAPT2+3,^109,110 and SAPT2+3 with a coupled-cluster doubles treatment of dispersion, SAPT2+3(CCD).¹¹¹ These allow the use of DF-SAPT0 on systems with up to ~ 3500 basis functions and the use of DF-SAPT2+3(CCD) on systems with up to ~ 1200 basis functions. Recently, we have extended the capabilities of the SAPT codes in Psi4 to broaden their applicability beyond closed-shell dimers and to deepen the insight obtained. These developments include the “functional group” SAPT (F-SAPT) decomposition,^112,113 which provides a partition of the SAPT0 energy at the chemical functional group level through partial summations of the SAPT perturbation series in terms of pairs of nuclei and localized electrons. F-SAPT0 has the same computational cost as a standard DF-SAPT0 analysis, but provides deeper insight by quantifying the contribution of each pair of functional groups to each intermolecular interaction energy term. For example, with F-SAPT we can partition the interaction energy in a protein-ligand interaction into contributions between functional groups on the ligand and individual side-chains in the active site.

Another new development is “intramolecular” SAPT (I-SAPT),¹¹⁵ which uses Hartree-Fock-in-Hartree-Fock embedding¹¹⁶ to produce a SAPT0-type interaction energy analysis for two moieties interacting within the presence of a third body, which might be covalently bound to one or both of the moieties in question. I-SAPT thus yields insight into intramolecular noncovalent interactions, previously inaccessible to SAPT. An example of the combined deployment of F-SAPT and I-SAPT is depicted in Figure 2 above. The last new extension of SAPT methods in Psi4 is an efficient implementation of open-shell SAPT0.¹¹⁷ The corresponding equations were only recently published,^118,119 and we present the first publicly available fully density-fitted implementation SAPT0 using unrestricted Hartree-Fock references, allowing computations on high-spin radical dimers of unprecedented sizes. The SAPT0 codes, and their F-SAPT, I-SAPT, and open-shell extensions, make substantial use of the JK object and other libraries described in Section 5, which greatly accelerated the implementation of these methods. Moreover, the use of these common library primitives serve to “standardize” the tractability limits of all of the SAPT0-type methods, meaning that they all are deployable to the same ~ 3500 basis function scale.

Figure 2:

Example F-SAPT analysis of non-covalent stabilization of a transition state in the reaction of benzaldehyde with an enamine intermediate (see Ref. 114). Color coding shows the electrostatic interaction between the phenyl group of benzaldehyde with the three functional groups of the enamine intermediate. A red group is attracted to the other monomer, whereas a blue group is repelled by the other monomer; C-H/π contacts contribute to the favorable red/red interaction between the cyclohexenyl and phenyl groups.

6.2. Density-Fitted and Cholesky Decomposition Coupled-Cluster Theory, with Frozen Natural Orbitals

One of the major additions to Psi4 since the alpha release is coupled-cluster theory utilizing density-fitting or Cholesky decomposition approximations for the ERIs, as provided by the libraries described in section 5.1. The DF/CD approximations substantially reduce the input/output time associated with processing the ERIs. However, the rate-determining step in coupled-cluster singles and doubles (CCSD)¹²⁰ remains the same, scaling as $\mathcal{O}\left(o^2{v}^4\right)$, where o and v are the number of occupied and virtual orbitals, respectively. Indeed, this rate-determining step, the so-called particle-particle ladder term, becomes actually more computationally costly in the DF/CD formalism, as one has to create the all-virtual ERIs from the 3-index quantities in a step that scales as $\mathcal{O}\left(N_{\text{aux }}{v}^4\right)$, where N_aux is the size of the auxiliary index (the number of auxiliary basis functions in the DF approach, or the number of Cholesky vectors in the CD approach). While technically only a fifth-power term, it is a very expensive one, since N_aux and v are both large compared to o.

We addressed this issue by using a synergistic approximation, frozen natural orbitals (FNOs).^121–124 These orbitals are better suited to truncating the virtual orbital subspace; one can delete a significant fraction of the most weakly occupied natural orbitals with only a minor impact on the total energy. Indeed, in our tests of intermolecular interaction energies in the S22 test set,¹²⁵ we performed computations with coupled-cluster through perturbative triples [CCSD(T)] in which we deleted FNO virtual orbitals with occupation numbers less than 10⁻⁵. In an aug-cc-pVDZ basis set,^52,53 the mean absolute error for this approximation was only 0.012 kcal/mol when MP2 is used to approximately correct for the deleted FNO virtual orbitals.¹²⁶ The errors due to density fitting are an order of magnitude smaller than this, about 0.001–0.002 kcal/mol when using the standard double-ζ density fitting auxiliary basis sets.¹²⁷ On the other hand, the combined DF/CD and FNO approximations can lead to substantial speedups in some cases, particularly for intermolecular interactions when counterpoise corrections are applied, because most of the “ghost” functions are weakly occupied and can be effectively truncated by the FNO procedure. To compute the relatively small and subtle three-body contribution to the interaction energy of the benzene trimer, we used a very conservative 10⁻⁷ cutoff on the FNOs, and nevertheless the overall counterpoise-corrected DF-FNO-CCSD(T)/aug-cc-pVDZ computation ran four times faster than the corresponding conventional CCSD(T) computation.¹²⁷ This allowed us to perform a large number of such computations, which resulted in the first definitive resolution of the three-body contribution to the lattice energy of crystalline benzene.¹²⁸ Additional speedups of about 3x are possible using our version of the code optimized for GPUs.¹²⁹ The MOLCAS¹³⁰ and Q-Chem¹³¹ programs have also implemented coupled-cluster codes using DF and/or CD approximations combined with FNOs or similar orbitals.^132,133

6.3. Perturbation theory and coupled-cluster gradients

Analytic gradients have been implemented for MP2 and third-order Møller-Plesset perturbation theory (MP3), using restricted Hartree-Fock (RHF) or unrestricted Hartree-Fock (UHF) orbitals, with conventional integrals or with density fitting. From these codes, it was straightforward¹³⁴ to also implement “MP2.5,” which is a simple average of MP2 and MP3.¹³⁵ This method has been found^134–136 to give rather accurate results for non-covalent interaction energies at a substantially reduced cost compared to CCSD(T).

We have also recently added analytic gradients for CCSD and CCSD(T) with RHF or UHF orbitals. In addition, we published the analytic gradients of density-fitted CCSD.¹³⁷ The density fitting approximation, while speeding up the computation of certain terms, has very little effect on geometries or vibrational frequencies.^137,138

6.4. Optimized-Orbital Methods

In wavefunction methods, one typically uses canonical Hartree-Fock orbitals. However, it is also possible to use orbitals that minimize the energy of the targeted post-Hartree-Fock wavefunction; MCSCF would be a popular example. Minimizing the energy of other kinds of wavefunctions with respect to orbital rotations is not nearly as well explored, but it has been reported in the past for MP2^139–142 and coupled-cluster doubles (CCD).^{120,143–145} The use of optimized orbitals simplifies computation of response properties and means that orbital response terms are already accounted for in analytic gradients. Additionally, optimized-orbital (OO) CCD was found,¹⁴⁴ like Brueckner CCD, to provide much better vibrational frequencies for systems exhibiting artifactual spatial symmetry-breaking like NO₃ and ${\text{O}}_4^{+}$.^146,147

Psi4 provides extensive support for optimized-orbital wavefunction methods, including an improved Newton-Raphson algorithm for the orbital optimization.¹⁴⁸ Optimized-orbital versions of MP2,¹⁴⁸ MP3,¹⁴⁹ MP2.5,¹⁵⁰ linearized CCD,¹⁵¹ and CCD¹⁴⁸ are implemented, as well as standard or asymmetric (T) triples corrections^152,153 to the latter.¹⁵⁴ In addition, analytic energy gradients are available for OO-MP2,¹⁵⁰ OO-MP3,¹⁵⁵ OO-linearized CCD,¹⁵¹ and OO-MP2.5.¹³⁴ Density-fitted versions of the energies and gradients are also available for MP2,¹⁵⁶ MP3,¹⁵⁷ MP2.5,¹⁵⁷ and linearized CCD.¹⁵⁸ For a set of weakly interacting dimers, orbital optimization was found to maintain the good performance of Hartree-Fock-based MP2.5 for closed-shell systems, but for open-shell systems, orbital optimization decreased errors by a factor of 5 (leading to results better than the more expensive CCSD method).¹³⁴

6.5. Density-Fitted Multi-Configurational Self-Consistent-Field

In Psi4 1.1 the MCSCF module has been completely rewritten using an approach similar to the atomic-orbital formulation.¹⁵⁹ This method is able to reduce the most computationally expensive pieces of MCSCF to J and K builds. There are several pieces of the pure atomic-orbital formulation that do not scale well to large active spaces, and these pieces can be supplied either by density-fitting techniques¹⁶⁰ or transforming exact ERIs.

For flexibility, all methods of the MCSCF wavefunction object have been exported to Python so that advanced MCSCF methods can be easily built.

cas_e, cas_wfn = energy(“CASSCF”, return_wfn=True)
# Compute the sigma vector for the current MCSCF state
inp_vec = cas_wfn.reference_civector()
out_vec = cas_wfn.new_civector()
cas_wfn.sigma(inp_vec, out_vec)
# Build an arbitrary two-particle density matrix between two vectors
tpdm = cas_wfn.tpdm(inp_vec, out_vec)

Using these flexible objects allows for the rapid exploration of various post-MCSCF theories. For example, SAPT based on CASSCF wavefunctions is being developed in such a manner.

6.6. Density Cumulant Functional Theory

Psi4 is the only quantum chemistry program that features a publicly available implementation of density cumulant functional theory (DCT). Rather than describing electron correlation using a many-electron wavefunction, as in conventional ab initio theories, DCT uses the cumulant of the two-electron reduced density matrix.¹⁶¹ By parametrizing the cumulant via a set of approximate constraints (also known as N-representability conditions), DCT provides a way to directly obtain electronic density matrices and energies without explicitly constructing the electronic wave function, thus making the computation of molecular properties (e.g., equilibrium structures, dipole moments, vibrational spectra) very efficient. Among other attractive features of DCT are size-extensivity, orbital relaxation, and the ability to efficiently incorporate high-order electron correlation effects. The earlier version of Psi4 presented the first implementation¹⁶² of the original DCT formulation that used a simple set of approximate N-representability conditions (the DC-06 method).¹⁶¹ In the Psi4 1.1 release, several new DCT methods have been implemented, including an improved description of the one-particle density matrix (DC-12),¹⁶³ orbital-optimized DCT formulations (ODC-06 and ODC-12),¹⁶⁴ and more sophisticated N-representability conditions and three-particle correlation effects [ODC-13 and ODC-13(λ₃)].¹⁶⁵ These new methods have a similar computational cost as the original DC-06 method, but exhibit much higher accuracy, especially for systems with unpaired electrons and significant multi-reference effects.^166,167 The new DCT code also features analytic gradients for the DC-06, ODC-06, and ODC-12 methods.^164,168 Very recently, we introduced density fitting and spin-adaptation, which significantly reduce the cost of the two-electron integral transformation for the orbital-optimized DCT methods (e.g., ODC-12).¹⁶⁹ Using this new DF-based implementation, it is now possible to perform DCT computations with all electrons correlated for systems with more than 1000 basis functions.

6.7. Algebraic-Diagrammatic Construction Excitation Energies

The latest version of Psi4 includes code to compute electronic excitation energies using the second-order algebraic-diagrammatic construction scheme [ADC(2)].¹⁷⁰ This can be thought of as an excited state generalization of MP2. The code accounts for point-group symmetry through the libdpd library, and it also includes a “partially renormalized” variant, PR-ADC(2), meant to confer resistance against quasi-degeneracy, described in Ref. 171. It can perform routine computations on systems with ~ 1000 basis functions.

6.8. Geometry Optimizer

By default, the geometry optimizer uses redundant internal coordinates, but the ability to alternatively use Cartesian or delocalized internal coordinates has been added since the alpha release. The default optimization step is now the iterative, restricted one¹⁷² arising from the rational function optimization method.¹⁷³ Cartesian Hessians from any source, and the Hessian guess from Lindh et al.¹⁷⁴ may now be used.

There is improved support for constrained optimizations, including the ability to freeze selected internal and Cartesian coordinates, and to optimize toward a structure with a desired coordinate value using extra forces. The user can also turn on a “dynamic-level” algorithm that incrementally tries more robust optimization methods for difficult systems when poor steps occur. This approach was critical in obtaining optimized geometries in a study of highly-strained alkoxy-substituted 1,8-bis((propyloxyphenyl) ethynyl) naphthalenes.¹⁷⁵

6.9. X2C Relativistic Corrections

One-component scalar theories offer an economical and convenient way to include approximate relativistic effects in electronic structure computations both at the self-consistent-field level (Hartree-Fock, DFT) and in electron correlation methods. Psi4 implements the spin-free one-electron version of the exact-two-component approach,^176–185 abbreviated as X2C. An X2C computation starts with the diagonalization of the one-electron Dirac Hamiltonian using a kinetically balanced basis set. Next, a Foldy-Wouthuysen transformation¹⁸⁶ is performed, which yields an effective one-electron Hamiltonian matrix (h_X2C), which is the sum of one-electron modified kinetic and potential energy matrices.

The choice of basis for X2C computations requires some considerations. In general, contracted basis sets do not offer a balanced description of the large and small components of the wave function, so it is generally recommended to evaluate the X2C Hamiltonian via a decontraction/recontraction procedure. The default procedure used in Psi4 consists in solving the Dirac Hamiltonian in a decontracted atomic orbital basis followed by projection of the solutions onto the contracted basis. Alternatively, the user may perform a computation in a fully decontracted basis.

One of the major advantages of spin-free scalar relativistic approaches like X2C is that they can be easily interfaced with non-relativistic electronic structure methods by replacing the one-electron Hamiltonian matrix with the quantity h_X2C. All other quantities that enter these theories (two-electron integrals, exchange-correlation functionals, etc.) are evaluated using standard non-relativistic equations. The cost of X2C computation has a scaling proportional to the cube of the number of basis functions (the number of primitive basis functions, in the case of decontraction/recontraction procedure) and is generally negligible. An example application of the X2C capabilities of Psi4 is a recently developed X2C/orthogonality-constrained DFT¹⁸⁷ approach to predict near edge X-ray absorption spectra.¹⁸⁸ The use of scalar relativistic Hamiltonians was found to be essential, since already for second-row elements the correction to nonrelativistic excitation energies is as large as 8 eV.

7. Community and Outreach

Finally, the Psi4 team and our collaborators have fostered the growth of a user community. We have spent significant effort in writing and maintaining a clear user manual,¹⁸⁹ which presents not only the features and input file syntax, but also discussions on simplifying and automating more complex computations. The manual is formatted using Sphinx,¹⁹⁰ which facilitates heavy use of cross-referencing for user convenience. To ensure that documentation on API and user options is kept up-to-date, we automatically generate this information directly from the source code. New users can consult a number of tutorial videos posted on YouTube¹⁹¹ and post questions at our user forum.¹⁹² News about Psi4, and downloads, are posted on our website.¹⁹³

Not only do we wish to help experienced users learn about how to use Psi4, but we also want to leverage Psi4 to teach undergraduates and graduate students about computational chemistry. We have formed the Psi4Education group to develop and distribute a set of computational chemistry laboratory exercises using freely available software, including Psi4. Our webpage provides the labs developed to date,¹⁹⁴ and a book chapter describes the project as a whole.¹⁹⁵ We use the popular WebMO graphical front-end,¹⁹ which is interfaced to Psi4. A Psi4Education workshop was held at the Biennial Conference on Chemical Education demonstrating these lab exercises in 2014. In 2016, a joint workshop on quantum chemistry and molecular mechanics simulations at the Florida American Chemical Society meeting introduced attendees to Psi4 integrated with plotting capabilities in Jupyter notebooks. We plan to continue developing additional lab exercises as part of this effort. In addition, as mentioned above in Section 2.3, advanced exercises in electronic structure theory for upper-level undergraduates and beginning graduate students, developed using our user-friendly Python front-end and the NumPy linear algebra library, are available through the Psi4NumPy project.⁶⁰

8. Conclusions

The hybrid C++/Python programming model seems successful in our experience so far. C++ allows fast, efficient code for lower-level, computationally intensive tasks. For the driver and other higher-level portions of the code, Python seems to offer a lower barrier to entry for new programmers, and it can facilitate rapid prototyping thanks to the interface with NumPy. Moreover, the large number of active scientific computing projects in Python is likely to aid interoperability of Psi4 with other projects, which is one of our major goals. To that end, the 1.1 release uses a new build and distribution system to make it easy to connect Psi4 with independently-developed, reusable software components. Finally, we have developed a new set of libraries providing efficient generation of generalized Coulomb and exchange matrices, density-fitted electron repulsion integrals, and related quantities. These libraries have greatly simplified and sped up our development of numerous new features over the past few years.