Leaderboards

Instead of comparing biobanks separately, we can pool information across all of the biobanks and perform a meta-analysis, which improves the fidelity of our absolute effect size and relative effect size benchmarks. These leaderboards represent the results from the meta-analysis of relative and absolute effect size.

More details

Performance

We meta-analysed PGS effect sizes (β) using meta-analytic mixed models. The absolute performance of PGS varied considerably between biobanks for some phenotypes.

By looking at differences between PGS effect sizes (βy - βₓ), we can partially adjust for the variability between biobanks. As for the results in the relative effect size pairwise comparisons, the confidence intervals of these differences are adjusted for the correlation between PGS. We provide the option to divide by the effect size of the chosen baseline method ((βᵧ - βₓ) / βᵧ) to display “relative” performance.

While we sometimes cannot confidently estimate the absolute performance (β) of methods across biobanks in the meta-analytical mixed model, we generally can make more accurate claims about “raw” differences in performance (βᵧ - βₓ).

Tuning

Many methods allow automatically setting suitable parameters without the use of phenotype data (we refer to this generally as automatic tuning). Alternatively, target data can be used to empirically determine hyperparameters on the basis of, for example, cross-validation (CV).

Leaderboard (relative effect size)

viewof method_y = Inputs.radio(
  ["dbslmm", "sbayesr", "lassosum", "prscs", "ldpred2", "megaprs", "pt.clump", "UKBB.EnsPRS"],
  { value: ["dbslmm", "sbayesr", "lassosum", "prscs", "ldpred2", "megaprs", "pt.clump", "UKBB.EnsPRS"],
  value: "pt.clump",
  label: "Baseline method:",
  sort: true, 
  unique: true,
  disabled: disable_method_type[method_type]}
)

// some methods only have one method type, so disable the radio button
disable_method_type = ({ "CV": ["prscs", "sbayesr", "dbslmm"], "auto": ["UKBB.EnsPRS"]})

viewof method_type = Inputs.radio(new Map([["Cross Validation (CV)", "CV"], ["Automatic", "auto"]]),
  { label: "Baseline tuning:",
  value: "auto"})
  
viewof pairwise_endpoint = Inputs.select(pairwise_endpoints, {value: "T2D", label: "Endpoint:"})

viewof use_relative = Inputs.radio(new Map([["Relative ((βᵧ - βₓ) / βᵧ)", "relative"], ["Raw (βᵧ - βₓ)", "raw"]]),
  { label: "Performance metric",
  value: "relative"})

table_columns = {
  if (use_relative === "relative") {
    return ["relative_rank", "method_y", "method_x", "method_type_x",  "ancestry", "relative_beta_with_ci"];
  } else {
    return ["rank", "method_y", "method_x", "method_type_x", "ancestry", "beta_with_ci"];
  }
}

table_header = {
  if (use_relative === "relative") {
    return {
      relative_rank: "Rank",
      method_y: "Baseline (βᵧ)",
      method_x: "Compared method (βₓ)",
      method_type_x: "Tuning method",
      ancestry: "Genetic ancestry",
      relative_beta_with_ci: "Performance (relative)"
    };
  } else {
    return {
      rank: "Rank",
      method_y: "Baseline method (βᵧ)",
      method_x: "Compared method (βₓ)",
      method_type_x: "Tuning type",
      ancestry: "Genetic ancestry",
      beta_with_ci: "Performance (raw)"
    };
  }
}

sort_key = {
  if (use_relative === "relative") {
    return "relative_rank";
  } else {
    return "rank"; 
  }
}

Inputs.table(filtered_pairwise, {
  columns: table_columns,
  header: table_header,
  sort: sort_key,
  rows: 20,
  width: "auto",
  height: "auto",
  multiple: false,
  select: false,
  layout: "fixed"  
})

import { aq, op } from '@uwdata/arquero'

// hides "undefined" return of aq.addFunction
_ = aq.addFunction('roundfloat', (x, y) => Number(x).toFixed(y));

filtered_pairwise = aq.fromJSON(pv_mrg)
  .derive({ relative_beta_diff: d => d.beta_diff / d.beta_y,
    relative_ci_high_diff: d => d.ci_high_diff / d.beta_y,
    relative_ci_low_diff: d => d.ci_low_diff / d.beta_y })
  .params({endpoint: pairwise_endpoint, method_y: method_y, method_type: method_type})
  .filter((d, $) => op.includes($.method_type, d.method_type_y ) &&
    op.includes($.method_y, d.method_y) &&
    op.includes($.endpoint, d.phenotype ))
  .orderby(aq.desc('relative_beta_diff'))
  .derive({
    relative_rank: d => op.row_number(),
    relative_beta_diff: d => aq.roundfloat(d.relative_beta_diff, 3),
    relative_ci_low_diff: d => aq.roundfloat(d.relative_ci_low_diff, 3),
    relative_ci_high_diff: d => aq.roundfloat(d.relative_ci_high_diff, 3)
  })
  .derive({
    relative_beta_with_ci: d => `${d.relative_beta_diff} (${d.relative_ci_low_diff}-${d.relative_ci_high_diff})`
})
  .orderby(aq.desc('beta_diff'))
  .derive({
    rank: d => op.row_number(),
    beta_diff: d => aq.roundfloat(d.beta_diff, 3),
    ci_low_diff: d => aq.roundfloat(d.ci_low_diff, 3),
    ci_high_diff: d => aq.roundfloat(d.ci_high_diff, 3)
  })
  .derive({
    beta_with_ci: d => `${d.beta_diff} (${d.ci_low_diff}-${d.ci_high_diff})`
})

The relative effect size leaderboard uses pruning and thresholding (pt.clump) with automatic tuning as the default baseline method.

Leaderboard (absolute effect size)

viewof methods = Inputs.checkbox(
  ["dbslmm", "sbayesr", "lassosum", "prscs", "ldpred2", "megaprs", "pt.clump", "UKBB.EnsPRS"],
  { value: ["dbslmm", "sbayesr", "lassosum", "prscs", "ldpred2", "megaprs", "pt.clump", "UKBB.EnsPRS"],
    label: "Method:",
    sort: true, 
    unique: true }
)

viewof ancestries = Inputs.checkbox(
  ["EUR", "SAS"],
  { label: "Ancestry",
    value: ["EUR", "SAS"],
    sort: true,
    unique: true
  })

viewof endpoint = Inputs.select(endpoints, {value: "T2D", label: "Endpoint:"})

Inputs.table(filtered, {
  columns: ["rank", "method", "tuning_type", "ancestry", "beta_with_ci"],
  header: {
    rank: "Rank",
    method: "Development method",
    tuning_type: "Tuning type",
    ancestry: "Genetic ancestry",
    beta_with_ci: "Beta (95% CI)" 
  },
  sort: "rank",
  rows: 20,
  width: "auto",
  height: "auto",
  multiple: false,
  select: false
})

filtered = aq.fromJSON(meta_res) 
  .params({methods: methods, ancestries: ancestries, endpoint: endpoint })
  .filter((d, $) => op.includes($.ancestries, d.ancestry) && 
    op.includes($.methods, d.method) &&
    op.includes($.endpoint, d.phenotype))
  .orderby(aq.desc('beta'))
  .derive({
    rank: d => op.row_number(),
    beta: d => aq.roundfloat(d.beta, 3),
    ci_low: d => aq.roundfloat(d.ci_low, 3),
    ci_high: d => aq.roundfloat(d.ci_high, 3)
  })
  .derive({
    beta_with_ci: d => `${d.beta} (${d.ci_low}-${d.ci_high})`
  })