Driven by technology, the trend towards collecting large data has resulted in the need for more flexible models to derive meaningful and actionable insights from such data and fast computational methods for fitting them. My research addresses these needs by investigating ways of speeding up and improving the accuracy of the derivation of classical and Bayesian statistical inference from complex models applied to large data sets. In particular, Bayesian methods offer a flexible and principled approach to inference and prediction that allows one to synthesize data from various sources while quantifying for uncertainty. However, it is often challenging to obtain Bayesian inference from complex models due to the computationally intensive characteristic of Markov Chain Monte Carlo (MCMC) methods. My research studies some important alternatives such as variational approximation and stochastic optimization, as well as ways to improve the rate of convergence of MCMC methods and the EM algorithm for classical statistical inference, by studying the close links between these methods.