Posterior predictive checks

[hodabbbb]

Hi Prof. Pfeifer and to all,

I am trying to compare two models which are estimated using different datasets (different in the sense that one of the models use an extra observable data series which is not included in the other model. Example: model1 is estimated on observable series Y -plus some othe obs- while model2 estimated using observable x and z where x+z = y - and the rest of observables are the same)

I am interested in some model fit tests and read about it the only relevant post. (link below)

viewtopic.php?f=1&t=6668

My questions are :

1. Can one run these posterior predictive checks manually by using any of the '' forecast '' commands of dynare ?
2. Does the new unstable version include any routines that allow to run any posterior predictive checks ?

many thanks in advance !

[jpfeifer]

Which exact posterior predictive check do you have in mind?

[hodabbbb]

Honestly I am not quite sure I am right.
What I have done is:
I have estimated.
I have compared data 2-nd moments to second moments from oo_.var (are these moments based on mode or posterior mean ?)
What I was thinking is based on what I read from a paper saying:

... compare model-implied statistics with those as in the data. I generate
samples of the same length as the data (after a burn-in period of 100 observations)
from the model economy using 1000 posterior draws. Table 30 reports the median of
the model-implied moments and the 90% credible intervals for raw data and Table 31
that for the cyclical component.

It seams the author is making a forecast of endogenous variables and calculating their CI as well drawing simulations.
Initially I thought I could do that with SMOOTHED variables but they turn to be exactly the same like the observed variables .#
From the documentation of dynare none of forecasts commands includes a ''burn-in'' option.
Is there a way to do a similar exercise pls?

Many thanks for the replies Professor!

[jpfeifer]

That quote indicates that you have to simulate the model for a given posterior draw and then compute the unconditional second moments. The easiest way to do this is most probably to use the

Code: Select all

posterior_function

command and within the posterior function call

Code: Select all

stoch_simul

with the respective options for simulation. An example can be found at https://github.com/DynareTeam/dynare/tree/master/tests/prior_posterior_function

[hodabbbb]

Many thanks for the guide Professor .


I am trying to figure out where the ''stoch_simul '' command is placed.
I read the documentation for the posterior function in '' 9 Dynare misc commands '' section of ''dynare.pdf'', quoting below:

posterior_function(OPTIONS) ; [Command]
% Options
function = FUNCTION_NAME
sampling_draws = INTEGER

It does not say that ''stoch_simul '' is a command WITHIN this command.

Also from the example you mentioned in the link in ''GitHub.com'' :

posterior_function(function='posterior_function_demo', sampling_draws=500);

.. the ''stoch_simul'' is not used WITHIN the ''posterior_function'' command.

So The way I understand your advise is to use the ''stoch_simul '' command AFTER the ''posterior_function'' command (not WITHIN)

. i.e

estimation(options.....)

posterior_function(function='posterior_function_demo', sampling_draws=500);

Stoch_simul(options....)

Did I get it right Professor ?
OR should I insert the ''stoch_simul'' command INSIDE the

posterior_function_demo

?
Many thanks again for your time Professor!

[jpfeifer]

Inside of the

Code: Select all

posterior_function_demo

you would need something along the lines of

Code: Select all

% set the parameters draws to the model structure
M_ = set_all_parameters(xparam1,estim_params_,M_);
% compute the steady state for the parameter draw written to M_
options_.noprint=1;
options_.drop=100;
options_.periods=300;
var_list=char('gp_obs','gy_obs');
[ys, oo_] = simult(oo_.dr.ys,oo_.dr,M_,options_,oo_);
oo_=disp_moments(ys,var_list,M_,options_,oo_);

%set second part of output cell
output_cell{1,1}=oo_.mean;


where you have to adjust

Code: Select all

options_.periods

to

Code: Select all

options_.drop

plus the number of data points you want to simulate. Moreover, you need to adjust the statistic you want to read out. In the example, it is the mean.

[hodabbbb]

Well,

I would

options_.drop=any number similar to 100
options_.periods=equal = the number of data points I want to simulate. (to replicate my observables)

for example:

options_.drop=100;
options_.periods=250;

I hope I got you right>
I still do not see where the ''stoch_simul'' command is written ?

Many thanks again

[jpfeifer]

Yes, you got that right. And no, you don't need a

Code: Select all

stoch_simul

command here, because I extracted the relevant lines of code doing the simulation and moment computation. Please be aware that this piece of code will not work if you use the

Code: Select all

loglinear

option

[hodabbbb]

Many thanks for your contribution Professor.
I am not quite sure I have done it right though I basically COPIED the command lines you mentioned INSIDE the file '' posterior_function_demo.m '' ( ''posterior_function_demo.m'' file ATTACHED)

I gave it a try by loading around 160,000 MH replications that I had done before and now triggering the ''posterior_function_demo.m'' file
by the command

posterior_function(function='posterior_function_demo', sampling_draws=500);

which comes AFTER the ''estimation(...) '' command.

Although I get some CI (confidence intervals) I do NOT get any simulated endogenous variables.
Also I get the following error:

Estimation::mcmc: Posterior (dsge) IRFs...
Estimation::mcmc: Posterior IRFs, done!
Estimation::mcmc: Total number of MH draws per chain: 160000.
Estimation::mcmc: Total number of generated MH files: 1.
Estimation::mcmc: I'll use mh-files 1 to 1.
Estimation::mcmc: In MH-file number 1 I'll start at line 32001.
Estimation::mcmc: Finally I keep 128000 draws per chain.

Index exceeds matrix dimensions.

Error in BggGKlinear (line 1182)
if exist('estim_params_', 'var') == 1

Error in dynare (line 223)
evalin('base',fname) ;

Clearly I have NOt implemented it the right way..
Could I be doing something wrong ?

[jpfeifer]

Please provide all files.

[hodabbbb]

Professor

the zipped folder is way too big (ca. 700 MB) due to the metropolis folder .
It would NOT not get uploaded.
GMAIL would not take anything greater than 25MB.

I have deleted the ''metropolis'' folder and the *_results.mat folder.

So you may be able to do some 3000 mh-replications and hopefully not get much time.Many thanks again!

[jpfeifer]

I was not able to replicate the crash when I ran 3000 draws. Rather, everything worked as expected. In

Code: Select all

oo_.posterior_function_results

I get the stored variable means from the simulation for each draw.

[hodabbbb]

Many thanks Professor.
1. It works for me as well now. I think it required that I run the Mh-replications from scratch rather than download existing ones as I did yesterday.

2. Regarding the output. I am trying to understand the output I get in

oo_.posterior_function_results

I am getting 500 cells each with 9 values inside (Probably this means 500 draws (times) the 9 observables I specified in

var_list= char(... );

Is my reading of results right pls ?

3. If so, What kind of interpretation can I make ? Does it mean that I can draw the confidence intervals myself based on these 500 draws for each observable (so I get their average -mean- and Confidence intervals ) ?

3. Don't I get simulated variables as specified in the option ?

options_.periods=250;

4. DO I get a Confidence Interval ?

I am struggling to understand : what is the output I am getting ?

[jpfeifer]

What you get in

Code: Select all

oo_.posterior_function_results

is the output provided by

Code: Select all

posterior_function_demo.m

i.e.

Code: Select all

output_cell{1,1}=oo_.mean;

This is the mean of the variables specified in

Code: Select all

var_list

in that function. If you need other output, you have to adjust that function. Simulated series for example are in

Code: Select all

ys

2. Yes, you can compute the credible set yourself using that output. Note that it's not a confidence interval, because this is a Bayesian and not a frequentist exercise.

[hodabbbb]

Hi Professor Pfeifer,

Earlier you had helped with the file '' posterior_function_demo.m ''
to simulate the endogenous variables with the estimated parameters.
It was quite helpful as I could use it finally. thank you for that.

Is there a way to obtain the second moments of HP-FILTERED variables instead ? by using the option:

options_.hpfilter=1600;

inside the '' posterior_function_demo.m '' file. It did not work in when I tried !!?? .

Many thanks in advance!