microprediction (c.f. home page)
Greetings. Here are my more useful packages:
| Site | Topic |
|---|---|
| precise.microprediction.org | Online (incremental) covariance and correlation estimators. Supports both sklearn and river-ml conventions. Treats arrival and departure of variables. |
| allocation.microprediction.org | Online (incremental) portfolio methods. Classics and two novel approaches: Schur and Thurstone. Asset smoothness and trading costs. Entry and exit of assets. |
| schur.microprediction.org | Schur portfolio literature. Schur provided the unification of hierarchical and optimisation-based portfolio construction via block-matrix inversion. Now part of skfolio and allocation.microprediction.org per the above. |
| thurstone.microprediction.org | I invented the fast ability transform that makes Thurstone Class V models practical. Use this package for multi-entrant contest probabilities and inverse calibration. This picture beats my words. |
| humpday.microprediction.org | All the greatest derivative-free (black box) optimization methods in one place in pure Python and Javascript with no package dependencies. Comparison tooling, claude skill and interactive demos. |
| skaters.microprediction.org | Fast and surprisingly reliable online univariate time series algorithms providing distributional predictions. Zero dependencies, runs in Pyodide. |
Book, papers, blog
| Site | Content |
|---|---|
| microprediction/home | Papers, posts etc |
| book.microprediction.org | Microprediction — the book on splintering data science into small algorithmic agents. (Audio book too) |
| microprediction.medium.com | Blog. Portfolio construction, optimization, time series, quantitative finance etc |
Educational sites
| Site | Content |
|---|---|
| conformalprediction.net | There's a small theoretical contribution: a KL-divergence theorem explaining clearly the limitations of conformal prediction. See this demo for example. |
| firstdown.microprediction.org | I've been trying to change NFL team's first down strategy for years, with little success. |
As an aside, if you were redirected from the minor-cult article Is Facebook's Prophet the Time-Series Messiah or Just a Very Naughty Boy?, it moved here.
I'm a career quant, applied mathematician, open-source developer, entrepreneur and father of three girls. My interests include quantitative finance, collective intelligence, sports analytics and mathematics in general. GitHub decided this should be my home page but this repo used to be the client for a high velocity prediction market I wrote for Intech Investments. It collected a billion predictions.
As noted above I also wrote a book predicting that data science would splinter into little agents. I've long been a believer in engineering pipelines that anyone else can improve without asking permission, and in the eventual inversion of control between humans and machine in the "microprediction domain" (frequently repeated quantitative tasks). With the arrival of LLMs I'm convinced this applies to judgemental prediction also. It seems I have more faith in small markets than most, noting the important caveat made clear in the Indispensable Markets Hypothesis paper that markets can be indispensible yet not perfectly efficient.
The book was a meditation on the power of mini-markets and algorithmic statistical agents - a thesis that went from unlikely to almost self-evident as LLMs arrived. It predates phrases like "DeAI" and "Info Finance" (Buterin) not to mention the general explosion of interest in prediction markets ... but despite this shift in the zietgeist the ideas have a long way to go as far as seeping into general software engineering consciousness in concerned (judging by this market anyway).
- manifoldbot - A bot that uses LLMs to trade on manifold prediction markets.
- randomcov - A set of quirky correlation and covariance matrix generators (I'd love your ideas).
- embarrassingly - A speculative approach to robust optimization that sends impure objective functions to optimizers.
- pandemic - Ornstein-Uhlenbeck epidemic simulation (related paper)
- momentum - My most personally re-used mini package ... for incremental mean, var, skew, kurtosis.
- muid - Memorable Unique Identifiers ... try to figure out how that can't be an oxymoron.
- timeseries-notebooks - Lots of examples of using open source timeseries packages.
- correlationbounds - Mini package for conf bounds
- building_an_open_ai_network - Book related.
- recalibrate - Utils related to Platt scaling etc.