Subgroup analysis in meta-analysis.

For centuries, Western medicine relied on an idea as fascinating as it was colourful: the theory of the four humours. According to Hippocrates and his successors, our health depended entirely on the balance between blood, phlegm, yellow bile, and black bile.

If you were prone to anger, you were classified into the “choleric” subgroup; if you were depressed, you clearly had an excess of black bile. Thus, ancient physicians divided all of humanity into four neat categories to try and explain why diseases and remedies affected each person so differently.

Today, this sounds to us like fairground quackery, but that ancient obsession with classifying everything to understand the variability of outcomes is more alive than ever.

In modern research, we no longer use leeches to balance humours, but when evaluating mountains of empirical evidence, we face the exact same question: does a treatment work equally well for everyone, or does its efficacy change drastically if we apply a subgroup analysis based on patient type, dosage, or experimental design?

The problem is that the human brain is an expert at seeing patterns where there is only noise. How do we know if that apparent discrepancy between two groups is a real difference or just an illusion caused by chance? To avoid ending up diagnosing an excess of methodological phlegm, we need an impartial referee: Cochran’s Q.

So, get comfortable, because in today’s post, we are going to dig into the guts of the data to master subgroup analysis within a meta-analysis, and discover how to separate, this time with mathematical rigor, true heterogeneity from mere mirages.

Aquí tienes la traducción de este segundo fragmento. He mantenido la metáfora extendida de la macedonia (que en inglés funciona maravillosamente para ilustrar la heterogeneidad) y ese contraste tan divertido entre el “elefante en la habitación” y las frutas.

A data fruit salad

Before diving into the problem of meta-analysis subgroups, I must confess a little secret: there is a huge elephant in the meta-analysis room that we must never lose sight of. This animal is none other than heterogeneity.

When we pool several primary studies to calculate a summary measure (the long-awaited global effect), it’s normal to find ourselves with a veritable fruit salad of data, where the studies do not yield the exact same numerical result. There is variability. After all, studies don’t come off an assembly line: they are conducted in different countries, with different populations, and by researchers who don’t always measure things the same way.

This variability can be due simply to chance or related to real differences among the populations from which the studies originate. That is why methodologists offer us two main paths to approach the problem of obtaining the global summary measure.

The first is the so-called fixed-effect model (in the singular). This approach is naivety turned into an algorithm. It assumes that each and every study is measuring the exact same true effect (as if assuming that every single piece in the fruit salad tasted only of apple). In this way, the variability of the study results is solely and exclusively due to bad luck, that is, sampling error or pure chance.

The second is the so-called random-effects model. Here enters the pragmatism that most often saves our lives. This model assumes that the true effect varies from one study to another because they come from different populations with a different real effect in each one (recognizing that in our fruit salad there are pears, grapes, and kiwis). In this case, the summary measure we calculate must account for this distribution of different effects.

Put simply and elegantly, the random-effects model accepts that the world is complex and that the efficacy of a treatment can inherently fluctuate depending on the context of each population.

So far, things are fairly straightforward. We only need to study the variability and decide which of the two models to apply to extract the global measure that summarizes the set of results from the primary studies in the meta-analysis.

But here comes the plot twist that gives meaning to this post: what if that variability is not just a quirk of chance or an inevitable natural noise? What if that fruit salad hides a compelling reason? Perhaps the ripeness level of the bananas, the use of peaches in syrup instead of fresh ones, or an excessive proportion of sour kiwis are the real culprits behind the final flavour of the results dancing from side to side.

To discover whether these variations stem from real characteristics of the underlying populations (our particular ingredients) or if they are simple tricks played by chance, we have no choice but to start separating the pieces of fruit to analyze them.

Comparing apples and oranges: subgroup analysis

So now we know that when we conduct a meta-analysis, we often run into a considerable amount of variability between studies, and that this variability isn’t always just annoying noise, but rather a variation that could have a highly useful explanation for our purposes. This is where subgroup analysis (also known as moderator analysis) shines, as it allows us to test specific hypotheses about why one type of study produces larger or smaller effects than another.

To fully understand its philosophy, we must clarify an essential baseline assumption: in subgroup analysis, we hypothesize that the studies in our meta-analysis do not come from a single general population, but rather fall into different subgroups, and we assume that each subgroup has its own true overall effect.

This might sound like what we just said about fixed-effect and random-effects models, but if we think about it for a moment, it’s a bit more complex than either of those two models on their own. Let’s take a look.

A methodological hybrid

Carrying out a subgroup analysis involves two fundamental steps. The first is to combine the studies into the different subgroups we are considering and obtain the summary measure for each one. The second is to compare those subgroup summary measures to try and figure out whether their differences are due to chance or not.

But where do these subgroups come from? As a general rule, they aren’t dictated by any mystical force; they are born from the pure and hard judgment of the researcher. By observing the characteristics of the populations from which the primary studies originate, the analyst decides where to slice with the methodological scalpel to establish the categories they suspect are causing the statistical ruckus.

Since assuming that all studies within a subgroup come from the exact same population is a bit naive, we use a random-effects model for this initial grouping.

Once the studies are grouped, we perform a sort of individual meta-analysis for each subgroup separately. In doing so, as it couldn’t be any other way, we will once again face a certain amount of internal variability within each category.

This is where our beloved tau-squared (τ²) makes its grand entrance. This is the statistical parameter that measures the dispersion of individual effects of each study within the group, thereby informing us of the variance or heterogeneity not due to chance that exists among studies in the same group.

In an ideal, worry-free world, we would calculate an independent τ² for each subgroup. In practice, however, we run into a snag: subgroups with very few studies would yield highly imprecise and unstable estimates of τ². It’s like trying to deduce someone’s personality just from the way they cough.

To solve this statistical conundrum, individual values are usually replaced by a pooled version of the variance. In other words, instead of each subgroup having its own τ², we calculate a single pooled value across all combined subgroups and assign it to each of them equally. It’s a small compromise to ensure that the calculations remain robust and don’t collapse under the weight of the smaller subgroups.

To understand why statisticians decided to dub this methodological hybrid the fixed-effects model (in the plural), we have to unpack its split personality.

On one hand, it is called effects (in the plural) because it assumes there isn’t a single real effect, but rather several true effects, one for each subgroup. On the other hand, they are called fixed because the categories we are comparing are the specific ones we are interested in, and they weren’t chosen at random.

In short: we assume there is random variability within each subgroup (as in the fixed-effect model), but real variability between the different subgroups (as in the random-effects model).

Aquí tienes la traducción del nuevo fragmento, manteniendo el tono afilado e ingenioso del blog y aplicando las pautas de estilo que incluiste en tu mensaje.

The relentless judge: cochran’s Q

Once we have calculated the individual effects of the subgroups, we still need to compare them to see if the distinctions we’ve made actually hold water. The most elegant way to assess this is to pretend that the combined effect of each subgroup is simply the observed effect size of a single, gigantic study.

By disguising our subgroups as individual mega-studies, we can hurl Cochran’s Q at them exactly as we would when hunting for heterogeneity in a standard meta-analysis.

The Q statistic functions as an omnibus test that evaluates the null hypothesis that all subgroup effect sizes are equal. If the observed Q value exceeds the expected value (based on a chi-squared distribution), the p-value will be significant. Good news! Our subgroup analysis has just proven that real differences exist between the categories, and that our division into subgroups actually makes sense.

It’s not all sunshine and rainbows

Before we get too excited and start running subgroup analyses on absolutely everything, we must proceed with caution, as this technique has some major limitations that can quickly rain on our parade.

First of all, subgroup analysis suffers from low statistical power. Since subgroups are often combined using random-effects models with their own inherent heterogeneity, precision drops and confidence intervals widen.

Because of this, finding significant differences in a subgroup analysis can be an uphill battle, which brings us to a golden rule: absence of evidence is not evidence of absence. Failing to find a difference does not prove that the treatments are equivalent.

A second issue is what we might call imaginary causality. The results of a subgroup analysis are purely observational. Even if we only include randomized controlled trials, the fact that a treatment looks better than another in our analysis could simply be because the treatment type is confounded by other methodological factors, such as using different control groups, to name just one of many possibilities.

Finally, we cannot forget that analyzing aggregated data carries the risk of ecological fallacy (or ecological bias). It is highly tempting to use aggregate information to define categories in a subgroup analysis. For instance, we might find that studies with a higher mean age show larger overall effects, whereas at the individual level (if we were to look at the actual participants), the relationship could be exactly the opposite.

Remember the lesson taught by the chocolate fallacy and never use aggregated data to try to predict or interpret associations at an individual level.

At any rate, no one should despair over these minor flaws. If we manage to dodge these traps, subgroup analysis will become one of the most valuable tools in our methodological arsenal. It will allow us to leave behind the dogmas of Hippocrates’ humours and embrace evidence-based explanations. And if all else fails… well, we’ll always have leeches.

We’re leaving…

And just like that, without even realizing it, we have dissected the noble art of classifying studies in a meta-analysis. Throughout this post, we have seen how heterogeneity can turn our flawless meta-analysis into an unpredictable fruit salad, and how subgroup analysis allows us to slice up the data to see what is truly going on.

We learned how to use the fixed-effects model (in the plural) to share that elusive variance, and we handed Cochran’s Q the judge’s gavel to pass sentence on chance, solemnly promising ourselves never again to fall into the traps of imaginary causality or ecological fallacy.

But before we say goodbye for today, let me give you a little statistical spoiler. Deep down, playing at dividing studies into neat little compartments is all well and good when we have closed categories. But what happens if the characteristic we suspect is causing all this ruckus is a continuous variable, like the exact dosage of a drug in milligrams or the publication year of the paper?

That is when our beloved subgroup analysis drops its mask and reveals its true identity: in reality, it is nothing more than a special case, a simplified little brother, of a much broader and more powerful mathematical technique called meta-regression, with which we can evaluate the impact of any variable (continuous or categorical) on the effect size of our studies. But that’s another story…