The University of Southampton
University of Southampton Institutional Repository

On the use of Cauchy prior distributions for Bayesian logistic regression

On the use of Cauchy prior distributions for Bayesian logistic regression
On the use of Cauchy prior distributions for Bayesian logistic regression

In logistic regression, separation occurs when a linear combination of the predictors can perfectly classify part or all of the observations in the sample, and as a result, finite maximum likelihood estimates of the regression coefficients do not exist. Gelman et al. (2008) recommended independent Cauchy distributions as default priors for the regression coefficients in logistic regression, even in the case of separation, and reported posterior modes in their analyses. As the mean does not exist for the Cauchy prior, a natural question is whether the posterior means of the regression coefficients exist under separation. We prove theorems that provide necessary and sufficient conditions for the existence of posterior means under independent Cauchy priors for the logit link and a general family of link functions, including the probit link. We also study the existence of posterior means under multivariate Cauchy priors. For full Bayesian inference, we develop a Gibbs sampler based on Pólya-Gamma data augmentation to sample from the posterior distribution under independent Student-t priors including Cauchy priors, and provide a companion R package tglm, available at CRAN. We demonstrate empirically that even when the posterior means of the regression coefficients exist under separation, the magnitude of the posterior samples for Cauchy priors may be unusually large, and the corresponding Gibbs sampler shows extremely slow mixing. While alternative algorithms such as the No-U-Turn Sampler (NUTS) in Stan can greatly improve mixing, in order to resolve the issue of extremely heavy tailed posteriors for Cauchy priors under separation, one would need to consider lighter tailed priors such as normal priors or Student-t priors with degrees of freedom larger than one.

Binary regression, Existence of posterior mean, Markov chain monte carlo, Probit regression, Separation, Slow mixing
1936-0975
359-383
Ghosh, Joyee
2df25062-4a81-4e56-a98b-90278e6fa327
Li, Yingbo
085fa181-da79-4e08-ba19-b0952e51f3e6
Mitra, Robin
2b944cd7-5be8-4dd1-ab44-f8ada9a33405
Ghosh, Joyee
2df25062-4a81-4e56-a98b-90278e6fa327
Li, Yingbo
085fa181-da79-4e08-ba19-b0952e51f3e6
Mitra, Robin
2b944cd7-5be8-4dd1-ab44-f8ada9a33405

Ghosh, Joyee, Li, Yingbo and Mitra, Robin (2018) On the use of Cauchy prior distributions for Bayesian logistic regression. Bayesian Analysis, 13 (2), 359-383. (doi:10.1214/17-BA1051).

Record type: Article

Abstract

In logistic regression, separation occurs when a linear combination of the predictors can perfectly classify part or all of the observations in the sample, and as a result, finite maximum likelihood estimates of the regression coefficients do not exist. Gelman et al. (2008) recommended independent Cauchy distributions as default priors for the regression coefficients in logistic regression, even in the case of separation, and reported posterior modes in their analyses. As the mean does not exist for the Cauchy prior, a natural question is whether the posterior means of the regression coefficients exist under separation. We prove theorems that provide necessary and sufficient conditions for the existence of posterior means under independent Cauchy priors for the logit link and a general family of link functions, including the probit link. We also study the existence of posterior means under multivariate Cauchy priors. For full Bayesian inference, we develop a Gibbs sampler based on Pólya-Gamma data augmentation to sample from the posterior distribution under independent Student-t priors including Cauchy priors, and provide a companion R package tglm, available at CRAN. We demonstrate empirically that even when the posterior means of the regression coefficients exist under separation, the magnitude of the posterior samples for Cauchy priors may be unusually large, and the corresponding Gibbs sampler shows extremely slow mixing. While alternative algorithms such as the No-U-Turn Sampler (NUTS) in Stan can greatly improve mixing, in order to resolve the issue of extremely heavy tailed posteriors for Cauchy priors under separation, one would need to consider lighter tailed priors such as normal priors or Student-t priors with degrees of freedom larger than one.

Text
euclid.ba.1488855634 - Version of Record
Available under License Creative Commons Attribution.
Download (732kB)

More information

e-pub ahead of print date: 7 March 2017
Published date: 1 June 2018
Keywords: Binary regression, Existence of posterior mean, Markov chain monte carlo, Probit regression, Separation, Slow mixing

Identifiers

Local EPrints ID: 419196
URI: https://eprints.soton.ac.uk/id/eprint/419196
ISSN: 1936-0975
PURE UUID: 9342cdea-cadb-4a2d-a00c-0818c71ed41d

Catalogue record

Date deposited: 09 Apr 2018 16:30
Last modified: 13 Mar 2019 18:41

Export record

Altmetrics

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×