pymare.results.MetaRegressionResults

class MetaRegressionResults(estimator, dataset, fe_params, fe_cov, tau2=None)

Bases: object

Container for results generated by PyMARE meta-regression estimators.

Parameters:

• estimator (BaseEstimator)) – The estimator used to produce the results.

• dataset (Dataset)) – A Dataset instance containing the inputs to the estimator.

• fe_params (numpy.ndarray of shape (p, d)) – Fixed-effect coefficients. Must be a 2-d numpy array with shape p x d, where p is the number of predictors, and d is the number of parallel datasets (typically 1).

• fe_cov (numpy.ndarray of shape (p, p)) – The p x p covariance matrix of the fixed effects, (X'WX)^-1 (or its cluster-robust replacement). Despite the "inv_cov" key it arrives under, this is a covariance, not an inverse covariance or a precision matrix – fe_se takes sqrt(diagonal(...)) of it, which is only meaningful for a covariance.

• tau2 (None or numpy.ndarray of shape (d,) or float, optional) – A 1-d array containing the estimated tau^2 value for each parallel dataset (or a float, for a single dataset). May be omitted by fixed-effects estimators.

Warning

When an Estimator is fitted to arrays directly using the fit method, the Results object’s utility is limited. Many methods will not work.

property fe_dof

Get the degrees of freedom used for fixed-effect inference.

An array of shape (p, d), one per predictor and parallel dataset, or None when a normal reference is used instead. Two different corrections can supply them, and they are not interchangeable: with group labels, Satterthwaite degrees of freedom for the CR2 cluster-robust standard errors [1]; without them, K - P, the residual degrees of freedom of the Knapp-Hartung adjustment [2], or None under small_sample_correction="wald". The Which one produced a given array is recorded rather than inferable from it: estimator.n_groups_ is None for the model-based path and the group count for the cluster-robust one. It is not safe to read the provenance off the values – balanced groups with an intercept-only model give a constant Satterthwaite dof, so “constant” does not imply K - P.

Changed in version 0.0.11: No longer None for an estimator fitted without group labels, unless small_sample_correction="wald" was requested.

The rest of this applies to the cluster-robust case. These are not the naive of Hedges et al.[3]. That count is right only when weight is spread evenly across groups; when a predictor is unbalanced at the group level it badly overstates the information available, and the resulting test rejects far too often. See satterthwaite_dof().

Read these before the p-values. Tipton[1] established that the approximation holds its nominal level only while the degrees of freedom exceed roughly pymare.stats.MIN_DOF_FOR_SATTERTHWAITE; below that the reference distribution is outside its validated range and satterthwaite_dof() warns. A comfortable number of groups does not guarantee comfortable degrees of freedom, because these depend on how evenly each predictor is spread across groups [4][5].

References

[1] (1,2,3)

Elizabeth Tipton. Small sample adjustments for robust variance estimation with meta-regression. Psychological methods, 20(3):375–393, 2015. URL: https://doi.org/10.1037/met0000011, doi:10.1037/met0000011.

[2] (1,2)

Guido Knapp and Joachim Hartung. Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine, 22(17):2693–2710, 2003. URL: https://doi.org/10.1002/sim.1482, doi:10.1002/sim.1482.

[3]

Larry V Hedges, Elizabeth Tipton, and Matthew C Johnson. Robust variance estimation in meta-regression with dependent effect size estimates. Research synthesis methods, 1(1):39–65, 2010. URL: https://doi.org/10.1002/jrsm.5, doi:10.1002/jrsm.5.

[4]

James E. Pustejovsky and Elizabeth Tipton. Small-sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models. Journal of Business & Economic Statistics, 36(4):672–683, 2018. doi:10.1080/07350015.2016.1247004.

[5]

Guido W. Imbens and Michal Kolesar. Robust standard errors in small samples: some practical advice. Review of Economics and Statistics, 98(4):701–712, 2016. URL: https://doi.org/10.1162/REST_a_00552, doi:10.1162/REST_a_00552.

property fe_se

Get fixed-effect standard error.

get_fe_stats(alpha=0.05)

Get fixed-effect statistics.

Parameters:

alpha (float, optional) – Default = 0.05.

Returns:

A dictionary of fixed-effect statistics. The dictionary has the following keys:

est	The parameter estimate for the regressor.
se	The standard error of the estimate.
z	The z score of the estimate.
p	The two-tailed p value of the estimate.
logp	The natural logarithm of p.
ci_l/ci_u	Lower and upper bounds of the estimate.

Return type:

dict

Notes

est / se is referred to a t distribution whenever the estimator recorded degrees of freedom for it, and to a normal distribution otherwise; fe_dof says which correction supplied them. With group labels the standard errors are cluster-robust and the reference has Satterthwaite degrees of freedom [1]; without them, under the default small_sample_correction="knapp-hartung", the covariance carries the Knapp-Hartung scale factor [2] and the reference has K - P. small_sample_correction="wald" gives a normal reference, as before.

Under "knapp-hartung" a standard error of exactly zero reaches this method when the weighted residuals are all zero – the scale factor correctly reporting that a perfectly fitting dataset says nothing about how uncertain the coefficients are. The undefined branch below turns that into a NaN p-value rather than a maximally significant one.

The tail is computed in logs because p cannot express the transform’s own input: a two-tailed normal p is exactly zero from |z| = 38.5 on, and every deviate past that would come back infinite. Via logp, z grows like sqrt(2 |logp|) without bound.

Changed in version 0.0.11: p is no longer floored at numpy.finfo(float).eps. The floor read a limit of precision as one of magnitude, truncating z at norm.isf(eps / 2) = 8.21 for every estimate below 2.2e-16, however far below it fell.

References

get_heterogeneity_stats()

Get heterogeneity statistics.

Warning

This method relies on the .dataset attribute, so the original Estimator must have be fitted with fit_dataset, not fit.

Returns:

A dictionary with the associated heterogeneity statistics. The keys to this dictionary are:

Q	Cochran’s Q [6]. This measure follows a chi-squared distribution, with n - k degrees of freedom, where n is the number of independent observations and k is the number of regressors.
p(Q)	P values associated with the Cochran’s Q values.
logp(Q)	Natural logarithms of the p(Q) values, computed directly and therefore still finite where p(Q) has underflowed to zero.
I^2	The proportion of the variance in input estimates that is due to heterogeneity instead of sampling error [7]. This measure is bounded from 0 to 100.
H	The ratio of the standard deviation of the estimated overall effect size from a random-effects meta-analysis compared to the standard deviation from a fixed-effect meta-analysis [7].

Return type:

dict

References

[6]

William G Cochran. The combination of estimates from different experiments. Biometrics, 10(1):101–129, 1954.

[7] (1,2)

Julian PT Higgins and Simon G Thompson. Quantifying heterogeneity in a meta-analysis. Statistics in medicine, 21(11):1539–1558, 2002.

get_re_stats(method='QP', alpha=0.05)

Get random-effect statistics.

Warning

This method relies on the .dataset attribute, so the original Estimator must have be fitted with fit_dataset, not fit.

Parameters:

• method ({"QP"}, optional) – Method for estimating the confidence interval of the tau^2 estimate. Default = “QP” [8].

• alpha (float, optional) – Default = 0.05.

Returns:

A dictionary of random-effect statistics. The dictionary has the following keys:

tau^2	The parameter estimate for the regressor.
ci_l/ci_u	Lower and upper bounds of the tau^2 estimate.

Return type:

dict

References

[8]

Wolfgang Viechtbauer. Confidence intervals for the amount of heterogeneity in meta-analysis. Statistics in medicine, 26(1):37–52, 2007. URL: https://doi.org/10.1002/sim.2514, doi:10.1002/sim.2514.

permutation_test(n_perm=1000)

Run permutation test.

Warning

This method relies on the .dataset attribute, so the original Estimator must have be fitted with fit_dataset, not fit.

Parameters:

n_perm (int, optional) – Number of permutations to generate. The actual number used may be smaller in the event of an exact test (see below), but will never be larger. Default = 1000.

Returns:

An instance of class PermutationTestResults.

Return type:

PermutationTestResults

Notes

If the number of possible permutations is smaller than n_perm, an exact test will be conducted. Otherwise an approximate test will be conducted by randomly shuffling the outcomes n_perm times (or, for intercept-only models, by randomly flipping their signs). Note that for closed-form estimators (e.g., ‘DL’ and ‘HE’), permuted datasets are estimated in parallel. This means that one can often set very high n_perm values (e.g., 100k) with little performance degradation.

to_df(alpha=0.05)

Return a pandas DataFrame summarizing fixed effect results.

Warning

This method only works for one-dimensional results.

Warning

This method relies on the .dataset attribute, so the original Estimator must have be fitted with fit_dataset, not fit.

Parameters:

alpha (float, optional) – Default = 0.05.

Returns:

df – DataFrame summarizing fixed effect results. The DataFrame will have one row for each regressor, and the following columns:

name	Name of the regressor.
estimate	The parameter estimate for the regressor.
se	The standard error of the estimate.
z-score	The z score of the estimate.
p-value	The p value the estimate.
-log10(p)	The p value on a base-10 log scale.
ci_+	Lower and upper bounds of the estimate. There will be two columns, with names based on the alpha value. For example, if alpha = 0.05, the CI columns will be "ci_0.025" and "ci_0.975".

Return type:

pandas.DataFrame

Examples using pymare.results.MetaRegressionResults

The Basics of Running a Meta-Analysis

The Basics of Running a Meta-Analysis

Run Estimators on a simulated dataset

Run Estimators on a simulated dataset