The expectation consistent (EC) approximation framework is a state-of-the-art approach for solving (generalized) linear inverse problems with random forward operators and i.i.d. signal priors. In image inverse problems, however, both the forward operator and image pixels are structured, which plagues traditional EC implementations. In this work, we propose a novel incarnation of EC that exploits deep neural networks to handle structured operators and signals. For phase-retrieval, we propose a simplified variant called ''deepECpr'' that reduces to iterative denoising. In experiments recovering natural images from phaseless, shot-noise corrupted, coded-diffraction-pattern outputs, we observe accuracy surpassing the state-of-the-art prDeep (Metzler et al., 2018) and Diffusion Posterior Sampling (Chung et al., 2023) approaches with two-orders-of-magnitude complexity reduction.

In light of the widespread success of generative models, a significant amount of research has gone into speeding up their sampling time. However, generative models are often sampled multiple times to obtain a diverse set incurring in a cost that is orthogonal to sampling time. We tackle the question of how to improve diversity and sample efficiency by moving beyond the common assumption of independent samples. For this we propose particle guidance, an extension of diffusion-based generative sampling where a joint-particle time-evolving potential enforces diversity. We analyze theoretically the joint distribution that particle guidance generates, its implications on the choice of potential, and the connections with methods in other disciplines. Empirically, we test the framework both in the setting of conditional image generation, where we are able to increase diversity without affecting quality, and molecular conformer generation, where we reduce the state-of-the-art median error by 13% on average.

The recent advent of diffusion models has led to significant progress in solving inverse problems, leveraging these models as effective generative priors. Nonetheless, challenges related to the ill- posed nature of such problems remain, often due to inherent ambiguities in measurements. Drawing inspiration from the human ability to resolve visual ambiguities through perceptual biases, here we introduce a novel latent diffusion inverse solver by incorporating regularization by texts (TReg). Specifically, TReg applies the textual description of the preconception of the solution during the reverse sampling phase, of which description is dynamically reinforced through null-text optimization for adaptive negation. Our comprehensive experimental results demonstrate that TReg successfully mitigates ambiguity in latent diffusion inverse solvers, enhancing their effectiveness and accuracy.

Solving for the 3D atomic structure of unknown materials is a key problem in materials science. Atomic electron tomography (AET) is a technique capable of reconstructing the 3D position and chemical species of all atoms in a nanoscale sample from a series of 2D projections from different angles. One challenge in AET is carbon contamination that accumulates on the sample while collecting the tomographic projections, creating an unwanted temporal dynamic that degrades reconstruction quality when existing tomography algorithms expect a static sample. In this work, we use an unsupervised implicit neural representation (INR) as a space-time model to computationally remove the contamination and recover a clean 3D reconstruction, and show promising preliminary results on simulated data.

Generative models, including normalizing flows, are gaining popularity in imaging science for tasks such as image reconstruction, posterior sampling, and data sharing. However, training them requires a high-quality dataset of objects, which can be challenging to obtain in fields such as tomographic imaging. This work proposes AmbientFlow, a framework for training flow-based generative models directly from noisy and incomplete data using variational Bayesian methods. The effectiveness of AmbientFlow in learning invertible generative models of objects from noisy, incomplete stylized imaging measurements is demonstrated via numerical studies.

The idea of using generative models to perform posterior sampling for imaging inverse problems has elicited attention from the computational imaging community. The main limitation of the existing generative model-based posterior sampling methods is that they do not provide any information about how uncertain the generative model is. In this work, we propose a quick-to-adopt framework that can transform a given generative model-based posterior sampling method into a statistical model that can quantify the generative model uncertainty. The proposed framework is built upon the principles of Bayesian neural networks with latent variables and uses ensembling to capture the uncertainty on the parameters of a generative model. We evaluate the proposed framework on the computed tomography reconstruction problem and demonstrate its capability to quantify generative model uncertainty with an illustrative example. We also show that the proposed method can improve the quality of the reconstructions and the predictive uncertainty estimates of the generative model-based posterior sampling method used within the proposed framework.

We study generative compressed sensing when the measurement matrix is randomly subsampled from a unitary matrix (with the DFT as an important special case). It was recently shown that $\mathcal{O}(kdn\| \boldsymbol{\alpha}\|_{\infty}^{2})$ uniformly random Fourier measurements are sufficient to recover signals in the range of a neural network $G:\field^k \to \field^n$ of depth $d$, where each component of the so-called local coherence vector $\boldsymbol{\alpha}$ quantifies the alignment of a corresponding Fourier vector with the range of $G$. We construct a model-adapted sampling strategy with an improved sample complexity of $\mathcal{O}(kd\| \boldsymbol{\alpha}\|_{2}^{2})$ measurements. This is enabled by: (1) new theoretical recovery guarantees that we develop for nonuniformly random sampling distributions and then (2) optimizing the sampling distribution to minimize the number of measurements needed for these guarantees. This development offers a sample complexity applicable to natural signal classes, which are often almost maximally coherent with low Fourier frequencies. Finally, we consider a surrogate sampling scheme, and validate its performance in recovery experiments using the CelebA dataset.

An emerging new paradigm for solving inverse problems is via the use of deep learning to learn a regularizer from data. This leads to high-quality results, but often at the cost of provable guarantees. In this work, we show how well-posedness and convergent regularisation arises within the convex-nonconvex (CNC) framework for inverse problems. We introduce a novel input weakly convex neural network (IWCNN) construction to adapt the method of learned adversarial regularization to the CNC framework. Empirically we show that our method overcomes numerical issues of previous adversarial methods.

Many imaging inverse problems---such as image-dependent in-painting and dehazing---are challenging because their forward models are unknown or depend on unknown latent parameters. While one can solve such problems by training a neural network with vast quantities of paired training data, such paired training data is often unavailable. In this paper, we propose a generalized framework for training image reconstruction networks when paired training data is scarce. In particular, we demonstrate the ability of image denoising algorithms and, by extension, denoising diffusion models to supervise network training in the absence of paired training data. (The unabridged version of this manuscript is available at https://arxiv.org/abs/2303.09642}{https://arxiv.org/abs/2303.09642).

We consider hyperparameter optimization (HPO) of approaches that employ outputs of mechanistic models as priors in hybrid modeling for data consistent inversion. An implicit density estimator (DE) models a non-parametric distribution of model input parameters, and the push forward of those generated samples produces a model output distribution that should match a target distribution of observed data. A rejection sampler then filters out “undesirable” samples through a discriminator function. In a samples-generate-reject pipeline with the objective of fitting the push-forward to the observed experimental outputs, several DEs can be employed within the generator and discriminator components. However, the extensive evaluation of these end-to-end inversion frameworks is still lacking. Specifically, this data-consistent model inversion pipeline offers an extra challenge concerning optimization of constituent models. Traditional HPO are often limited to single-model scenarios and might not directly map to frameworks that optimize several models to achieve a single loss. To overcome the time overhead due to summative optimization of each component, and the expanded combinatorial search space, we introduce a method that performs an initial random search to bootstrap a HPO that applies weighted feature importance to gradually update the hyperparameter set, periodically probing the pipeline to track the loss. Our experiments show reduced number of time intensive pipeline runs but with the faster convergence.

We propose to enhance 1-weakly convex ridge regularizers for image reconstruction by incorporating spatial adaptivity. To this end, we resort to a neural network that generates a weighting mask from an initial reconstruction, which is obtained with the baseline regularizer. Empirically, the learned mask can capture long-range dependencies and leads to a smaller penalization of inherent image structures. Our experiments show that spatial adaptivity improves the performance of image denoising and MRI reconstruction.

We provide the first framework to solve inverse problems with diffusion models learned from linearly corrupted data. Our method leverages a generative model trained on one type of corruption (e.g. highly inpainted images) to perform posterior sampling conditioned on measurements from a different forward process (e.g. blurred images). This fully unlocks the potential of ambient diffusion models that are essential in scientific applications where access to fully observed samples is impossible or undesirable. Our experimental evaluation shows that diffusion models trained on corrupted data can even outperform models trained on clean data for image restoration in both speed and performance.

We propose a surrogate function for efficient use of score-based priors for Bayesian inverse imaging. Recent work turned score-based diffusion models into probabilistic priors for solving ill-posed imaging problems by appealing to an ODE-based log-probability function. However, evaluating this function is computationally inefficient and inhibits posterior estimation of high-dimensional images. Our proposed surrogate prior is based on the evidence lower-bound of a score-based diffusion model. We demonstrate the surrogate prior on variational inference for efficient approximate posterior sampling of large images. Compared to the exact prior in previous work, our surrogate prior accelerates optimization of the variational image distribution by at least two orders of magnitude. We also find that our principled approach achieves higher-fidelity images than non-Bayesian baselines that involve hyperparameter-tuning at inference. Our work establishes a practical path forward for using score-based diffusion models as general-purpose priors for imaging.

Phase retrieval (PR) is a crucial problem in many imaging applications. This study focuses on resolving the holographic phase retrieval problem in situations where the measurements are affected by a combination of Poisson and Gaussian noise, which commonly occurs in optical imaging systems. To address this problem, we propose a new algorithm called ``AWFS" that uses the accelerated Wirtinger flow (AWF) with a score function as generative prior. We calculate the gradient of the log-likelihood function for PR and provide an implementable estimate for it. Additionally, we introduce a generative prior in our regularization framework by using score matching to capture information about the gradient of image prior distributions. The results of our simulation experiments on three different datasets show the following. 1) By using the PG likelihood model, the proposed algorithm improves reconstruction compared to algorithms based solely on Gaussian or Poisson likelihood. 2) The proposed score-based image prior method leads to improved reconstruction quality over the method based on denoising diffusion probabilistic model (DDPM), as well as plug-and-play alternating direction method of multipliers (PnP-ADMM) and regularization by denoising (RED).

Uncertainty quantification for imaging-related inverse problems is drawing much attention lately. Existing approaches towards this task define uncertainty regions per pixel while ignoring spatial correlations. In this paper we propose PUQ (Principal Uncertainty Quantification) -- a novel definition of uncertainty that takes into account spatial relationships within the image, thus providing reduced uncertainty volume. Leveraging diffusion models, we derive uncertainty intervals around principal components of the empirical posterior distribution, accompanied by probabilistic guarantees. The proposed approach can operate globally on the entire image, or locally on patches, resulting in informative and interpretable uncertainty regions. We verify our approach on several inverse problems, showing a significantly tighter uncertainty regions compared to baseline methods.

Likelihood analysis is typically limited to normally distributed noise due to the difficulty of determining the probability density function of complex, high-dimensional, non-Gaussian, and anisotropic noise. This work presents Score-based LIkelihood Characterization (SLIC), a framework that resolves this issue by building a data-driven noise model using a set of noise realizations from observations. We show that the approach produces unbiased and precise likelihoods even in the presence of highly non-Gaussian correlated and spatially varying noise. We use diffusion generative models to estimate the gradient of the probability density of noise with respect to data elements. In combination with the Jacobian of the physical model of the signal, we use Langevin sampling to produce independent samples from the unbiased likelihood. We demonstrate the effectiveness of the method using real data from the Hubble Space Telescope and James Webb Space Telescope.

Black-box forward model simulators are widely used in scientific and engineering domains for their exceptional capability to mimic complex physical systems. However, applying current state-of-the-art gradient-based Bayesian inference techniques like Hamiltonian Monte Carlo or Variational Inference with them becomes infeasible due to the opaque nature of these simulators. We address this challenge by introducing a modular approach that combines black-box variational inference (BBVI) with deep generative priors, making it possible to efficiently and accurately perform high-dimensional Bayesian inversion in these settings. Our method introduces a novel gradient correction term and a sampling strategy for BBVI, which collectively diminish gradient errors by several orders of magnitude across different dimensions, even with minimal batch sizes. Furthermore, integrating our method with Generative Adversarial Network (GAN)-based priors significantly enhances the solution of high-dimensional inverse problems. We validate our algorithm's effectiveness on a range of physics-based inverse problems using both simulated and experimental data. In comparison to Markov Chain Monte Carlo (MCMC) methods, our approach consistently delivers superior accuracy and substantial improvements in both statistical and computational efficiency, often by an order of magnitude.

Inverse problems arise in a multitude of applications, where the goal is to recover a clean signal from noisy and possibly (non)linear observations. The difficulty of a reconstruction problem depends on multiple factors, such as the structure of the ground truth signal, the severity of the degradation, the implicit bias of the reconstruction model and the complex interactions between the above factors. This results in natural sample-by-sample variation in the difficulty of a reconstruction task, which is often overlooked by contemporary techniques, resulting in long inference times, subpar performance and wasteful resource allocation. We propose a novel method to estimate the degradation severity of noisy, degraded signals in the latent space of an autoencoder. We show that the estimated severity has strong correlation with the true corruption level and can give useful hints at the difficulty of reconstruction problems on a sample-by-sample basis. Furthermore, we propose a reconstruction method based on latent diffusion models that leverages the predicted degradation severities to fine-tune the reverse diffusion sampling trajectory and thus achieve sample-adaptive inference.

In this work, we introduce a physics-guided self-supervised learning approach to reconstruct dynamic magnetic resonance images (MRI) from sparsely sampled radial data. The architecture incorporates a variable splitting scheme via a quadratic penalty approach consisting of iterative data consistency and denoiser step. To accommodate cardiac motion, the denoiser implements a learnable low-rank and sparse component instead of a conventional convolutional neural network. We compare the proposed model to iterative regularized MRI reconstruction techniques and to other deep neural network approaches adapted to radial data, both in supervised and self-supervised tasks. Our proposed method surpasses the performance of other techniques for a single heartbeat and four heartbeat MRI reconstruction. Furthermore, our approach outperforms other deep neural network reconstruction approaches in both supervision and self-supervision tasks.

Optical multilayer thin film structures are widely used in various photonic applications. Inverse design is an important but difficult step to enable these applications, which seeks to find out the best structure (material & thickness arrangements) given a target optical response. Recently, deep learning-based methods have been developed to solve the inverse design efficiently. However, existing methods usually fix the material arrangements and only design the thickness, which is not versatile for a different material arrangement and may lead to sub-optimal performance. In this study, we resolve this issue by treating the structure as a sequence and using structure tokens to represent the material and thickness simultaneously. Later on, the inverse design problem can be formulated as a common sequence generation task conditioned on the input optical responses. Based on this, we propose OptoGPT to act as a versatile inverse design model that can design material and thickness simultaneously, significantly expanding the design capability. In addition, using probability resampling further provides a versatile method to satisfy fabrication and design requirements in practical applications.

Despite the promise of Neural Posterior Estimation (NPE) methods in astronomy, the adaptation of NPE into the routine inference workflow has been slow. We identify three critical issues: the need for custom featurizer networks tailored to the observed data, the inference inexactness, and the under-specification of physical forward models. To address the first two issues, we introduce a new framework and open-source software \textit{nbi} (\textit{Neural Bayesian Inference}), which supports both amortized and sequential NPE. First, \textit{nbi} provides built-in ``featurizer'' networks with demonstrated efficacy on sequential data, such as light curve and spectra, thus obviating the need for this customization on the user end. Second, we introduce a modified algorithm SNPE-IS, which facilities asymptotically exact inference by using the surrogate posterior under NPE only as a proposal distribution for importance sampling. These features allow \textit{nbi} to be applied off-the-shelf to astronomical inference problems involving light curves and spectra. We discuss how \textit{nbi} may serve as an effective alternative to existing methods such as Nested Sampling. Our package is at [url redacted].

We propose to sample from high-dimensional posterior distributions arising in physics-based inverse problems using conditional score-based generative models. The proposed approach trains a noise-conditional score network to approximate the score function of the posterior distribution. Then, the network is used to sample from the posterior distribution through annealed Langevin dynamics. The proposed method is applicable even when we can only simulate the forward problem. We apply it to two physics-based inverse problems and compare its performance with conditional generative adversarial networks. Results show that conditional score-based generative models can reliably perform Bayesian inference.

Wearable devices offer continuous monitoring of biomarkers, presenting an opportunity to diagnose cardiovascular diseases earlier, potentially reducing their fatality rate. While machine learning holds promise for predicting cardiovascular biomarkers from sensor data, its use often depends on the availability of labeled datasets, which are limited due to technical and ethical constraints. On the other hand, biophysical simulations present a solution to data scarcity but face challenges in model transfer from simulation to reality due to inherent model simplifications and misspecifications. Building on advancements in hybrid learning, we introduce a method that combines a pulse-wave propagation model, rooted in biophysical simulations, with a correction model trained with unlabeled real-world data. This generative model transforms cardiovascular parameters into real-world sensor measurements and, when trained as an auto-encoder, also provides the inverse transformation, mapping measurements to cardiovascular biomarkers. Notably, when assessed using real pulse-wave data, our hybrid method appears to outperform models based solely on simulations in inferring cardiovascular biomarkers, opening new avenues for inferring physiological biomarkers in data-limited scenarios.

In recent years, inverse problems for black-box simulators have enjoyed increased focus of the machine learning community due to their prevalence in science and engineering domains. Such simulators describe a forward process $f: (\psi, x) \rightarrow y$. Here the intent is to optimise simulator parameters $\psi$ to minimise some observation loss on $y$, under some input distribution on $x$. Optimisation of such objectives is often challenging, since it is not trivial to estimate simulator gradients accurately. In settings where multiple related inverse problems need to be solved simultaneously, from-scratch/ab-initio optimisation of each may be infeasible if the forward model is expensive to evaluate. In this paper, we propose a novel method for solving such families of inverse problems with reinforcement learning. We train a policy to guide the optimisation by selecting between gradients estimated numerically from the simulator and gradients estimated from a pre-trained surrogate model. After training the surrogate and the policy, downstream inverse problem optimisations require 10\%-70\% fewer simulator evaluations. Moreover, the policy does successful optimisations on functions where using just simulator gradient estimates fails.

Diffusion-based generative models have been used as powerful priors for magnetic resonance imaging (MRI) reconstruction. We present a learning method to optimize sub-sampling patterns for compressed sensing multi-coil MRI that leverages pre-trained diffusion generative models. Crucially, during training we use a single-step reconstruction based on the posterior mean estimate given by the diffusion model and the MRI measurement process. Experiments across varying acceleration factors and pattern types show that sampling operators learned with our method lead to competitive, and in the case of 2D patterns, improved reconstructions compared to baseline patterns.

Diffusion models have recently delivered state-of-the-art performance for MRI reconstruction with improved robustness. However, these models fail when there is a large distribution shift, and their long inference times impede their clinical utility. Recently, regularization by denoising diffusion process (RED-diff) was introduced for solving general inverse problems. RED-diff uses a variational sampler based on a measurement consistency loss and a score matching regularization. In this paper, we extend RED-diff to MRI reconstruction. RED-diff formulates MRI reconstruction as stochastic optimization, and outperforms diffusion baselines in PSNR/SSIM with $3 \times$ faster inference while using the same amount of memory. The code is publicly available at https://github.com/NVlabs/SMRD.

Blind image deblurring (BID) has been extensively studied in computer visionand adjacent fields. Modern methods for BID can be grouped into two categories:single-instance methods that deal with individual instances using statistical infer-ence and numerical optimization, and data-driven methods that train deep-learningmodels to deblur future instances directly. Data-driven methods can be free fromthe difficulty in deriving accurate blur models, but are fundamentally limited bythe diversity and quality of the training data—collecting sufficiently expressiveand realistic training data is a standing challenge. In this paper, we focus onsingle-instance methods that remain competitive and indispensable, and address thechallenging setting unknown kernel size and substantial noise, failing state-of-the-art (SOTA) methods. We propose a practical BID method that is stable againstboth, the first of its kind. Also, we show that our method, a non-data-drivenmethod, can perform on par with SOTA data-driven methods on similar data thelatter are trained on, and can perform consistently better on novel data.

Proximal operators are ubiquitous in inverse problems, commonly appearing as part of algorithmic strategies to regularize problems that are otherwise ill-posed. Modern deep learning models have been brought to bear for these tasks too, as in the framework of plug-and-play or deep unrolling, where they loosely resemble proximal operators. Yet, these do not provide any guarantee that these general functions, implemented by neural networks, provide a proximal operator of some function, nor do they provide any characterization of the function of which they provide some approximate proximal. Herein we provide a framework to develop learned proximal networks (LPN), which provide exact proximal operators for a data-driven regularizer, and show how a new training strategy, dubbed proximal matching, guarantees that the obtained regularizer recovers the log-prior of the true data distribution. Thus, such LPN provide general, unsupervised, proximal operators that can be used for general inverse problems. We illustrate our results in a series of cases of increasing complexity, demonstrating that these models not only result in state-of-the-art restoration results, but provide a window into the resulting priors learned from data.

Multiplicative speckle noise is an inherent part of coherent imaging systems, such as synthetic aperture radar and digital holography. Speckle noise is mitigated by obtaining multiple measurement vectors with independent speckle noise, a technique commonly referred to as "multi-look", followed by appropriate averaging. However, in many applications, even with multi-look, the achievable performance is not satisfactory. Moreover, in this approach, every look (or every set of measurements) is required to be over-determined,which imposes additional constraints on spatial resolution. In this work, we develop a maximum likelihood based approach for recovering images from a set of compressive measurements contaminated by speckle noise. We propose an iterative multi-look compressive sensing recovery algorithm, DIP-$M^3$, that i) requires no training data, ii) is computationally efficient, and iii) generates high-quality reconstruction images from multi-look, where each look is underdetermined and corrupted by speckle noise.

Deep generative models, such as diffusion models, GANs, and IMLE, have shown impressive capability in tackling inverse problems. However, the validity of model-generated solutions w.r.t. the forward process and the reliability of associated uncertainty estimates remain understudied. This study evaluates recent diffusion-based, GAN-based, and IMLE-based methods on three inverse problems, i.e., 16x super-resolution, colourization, and image decompression. We assess the validity of these models' outputs as solutions to the inverse problems and conduct a thorough analysis of the reliability of the models' estimates of uncertainty over the solution. Overall, we find that the IMLE-based CHIMLE method outperforms other methods in terms of producing valid solutions and reliable uncertainty estimates.

Diffusion models have demonstrated state-of-the-art results in solving inverse problems in various domains including medical imaging. However, existing works generally consider the cases where the forward operator is fully known. Therefore blind inverse problems with unknown forward operator parameters require modifications on existing methods. In this work, we present an extension of the recently developed regularization by denoising diffusion process (RED-diff) algorithm to the blind inverse problems. We test our method in fieldmap-corrected MR image reconstruction and show that the blind RED-diff framework can successfully approximate the unknown forward model parameters and produce fieldmap corrected reconstructions accurately.

For accelerated magnetic resonance imaging (MRI), conditional generative adversarial networks (cGANs), when trained end-to-end with a fixed subsampling mask, have been shown to compete with contemporary diffusion-based techniques while generating samples thousands of times faster. To handle unseen sampling masks at inference, we propose ``guided reconstruction'' (GR), wherein the cGAN code vectors are projected onto the measurement subspace. Using fastMRI brain data, we demonstrate that GR allows a cGAN to successfully handle changes in sampling mask, as well as changes in acceleration rate, yielding faster and more accurate recoveries than the Langevin approach from (Jalal et al., 2021) and the DDRM diffusion approach from (Kawar et al., 2022). Our code will be made available at https://github.com/matt-bendel/rcGAN-agnostic.

Evaluation of MR reconstruction methods is challenged by the need for image quality (IQ) metrics which correlate strongly with radiologist-perceived IQ. We explore Deep Feature Distances (DFDs) as MR reconstruction IQ metrics, whereby distances between ground truth and reconstructed MR images are computed in a lower-dimensional feature space encoded by a CNN. In addition to comparing DFDs to two commonly used pixel-based MR IQ metrics in PSNR and SSIM via correlations to radiologist reader scores of MR image reconstructions, we explore the impact of domain shifts between the DFD encoder training data and the evaluated MR images. In particular, we assess two state-of-the-art but "out-of-domain" DFDs with encoders trained on natural images, an in-domain DFD trained on MR images alone, and propose two domain-adjacent DFDs trained on large medical imaging datasets (not limited to MR data). IQ metric performance is assessed via their correlations to 5 expert radiologist reader scores of MR image reconstructions. We make three striking observations: 1) all DFDs out-perform traditional IQ metrics, 2) DFDs performance approaches that of radiologist inter-reader variability, and, 3) surprisingly, out-of-domain DFDs perform comparably as an MR reconstruction IQ metric to in-domain and domain-adjacent DFDs. These results make it evident that DFDs should be used alongside traditional IQ metrics in evaluating MR reconstruction IQ, and suggest that general vision encoders are able to assess visual IQ across image domains.

Conventional electromagnetic wave simulators often have long simulation times, so are not suitable for computational imaging and photonic inverse problems (e.g. end-to-end design, iterative reconstruction) that require evaluating the forward model many times. Electromagnetic wave simulators based on neural networks promise speed improvements of several orders-of-magnitude, but standard supervised training approaches have difficulty fitting the true physics. Physics-informed approaches help, but existing residual-based methods use only local information and must be used in conjunction with standard supervised loss. In this work, we introduce Time Reversal Consistency (TReC), a new physics-based training method based on the time reversibility of Maxwell's equations. TReC uses a time-reversed, differentiable finite-difference simulator to compare neural network predictions with a known initial condition. TReC provides both global physics guidance and supervision in a single function. When trained only on randomized scatterers, we find that networks trained with TReC generalize well to a range of arbitrary structured media. We validate the method on the inverse design of a set of angle-to-angle couplers, addressing almost two magnitudes more parameters than previous methods, and find that the design quality corresponds closely with designs based on a conventional simulator while requiring 5\% of the design time.

Event cameras offer high frame rates, minimal motion blur, and excellent dynamic range. As a result they excel at reconstructing the geometry of 3D scenes. However, their measurements do not contain absolute intensity information, which can make accurately reconstructing the appearance of 3D scenes from events challenging. In this work, we develop a multimodal neural 3D scene reconstruction framework that simultaneously reconstructs scene geometry from events and scene appearance from grayscale images. Our framework---which is based on neural surface representations, as opposed to the neural radiance fields used in previous works---is able to reconstruct both the structure and appearance of 3D scenes more accurately than existing unimodal reconstruction methods.

Phase retrieval (PR) concerns the recovery of complex phases from complexmagnitudes. We identify the connection between the difficulty level and thenumber and variety of symmetries in PR problems. We focus on the most difficultfar-field PR (FFPR), and propose a novel method using double deep image priors.In realistic evaluation, our method outperforms all competing methods by largemargins. As a single-instance method, our method requires no training data andminimal hyperparameter tuning, and hence enjoys good practicality.

Data-consistent model inversion problems aim to infer distributions of model parameters from distributions of experimental observations. Previous approaches to solving these problems include rejection algorithms, which are impractical for many real-world problems, and generative adversarial networks, which require a differentiable simulation. Here, we introduce a sequential sample refinement algorithm that overcomes these drawbacks. A set of parameters is iteratively refined using density ratio estimates in the model input and output domains, and parameters are resampled by training a generative implicit density estimator. We implement this novel approach using a combination of standard models from artificial intelligence and machine learning, including density estimators, binary classifiers, and diffusion models. To demonstrate the method, we show two examples from computational biology, with different levels of complexity.

In-context learning is one of the surprising and useful features of large language models. How it works is an active area of research. Recently, stylized meta-learning-like setups have been devised that train these models on a sequence of input-output pairs $(x, f(x))$ from a function class using the language modeling loss and observe generalization to unseen functions from the same class. One of the main discoveries in this line of research has been that for several problems such as linear regression, trained transformers (TFs) learn algorithms for learning functions in context. We extend this setup to different types of linear-inverse problems and show that TFs are able to in-context learn these problems as well. Additionally, we show that TFs are able to recover the solutions in fewer-measurements than the number of unknowns, leveraging the structure of these problems and are in accordance with the recovery bounds. Finally, we also discuss the multi-task setup, where the TF is pre-trained on multiple types of linear-inverse problems at once and show that at inference time, given the measurements, they are able to identify the correct problem structure and solve the inverse problem efficiently.

In this paper, we propose DynODE, a method to model the video dynamics by learning the trajectory of independently inverted latent codes from GANs. The entire sequence is seen as discrete-time observations of a continuous trajectory of the initial latent code.The latent codes representing different frames are therefore reformulated as state transitions of the initial frame, which can be modeled by neural ordinary differential equations. Our DynODE learns the holistic geometry of the video dynamic space from given sparse observations and specifies continuous latent states, allowing us to engage in various video applications such as frame interpolation and video editing.Extensive experiments demonstrate that our method achieves state-of-the-art performance but with much less computation. Code is available at https://github.com/weihaox/dynode_released.

The multi-plane encoding approach has been highlighted for its ability to serve as static and dynamic neural radiance fields without sacrificing generality.This approach constructs related features through projection onto learnable planes and interpolating adjacent vertices. This mechanism allows the model to learn fine-grained details rapidly and achieves outstanding performance. However, it has limitations in representing the global context of the scene, such as object shapes and dynamic motion over times when available training poses are sparse. In this work, we propose refined tensorial radiance fields that harness coordinate-based networks known for strong bias toward low-frequency signals.The coordinate-based network is responsible for capturing global context, while the multi-plane network focuses on capturing fine-grained details.We demonstrate that using residual connections effectively preserves their inherent properties.Additionally, the proposed curriculum training scheme accelerates the disentanglement of these two features. We empirically show that the proposed method outperforms others for the task with static and dynamic NeRFs using sparse inputs.In particular, we prove that excessively increasing denoising regularization for multi-plane encoding effectively eliminates artifacts; however, it can lead to artificial details that appear authentic but are not present in the data. On the other hand, we note that the proposed method does not suffer from this issue.