Rendered at 18:33:04 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
brumar 20 hours ago [-]
How do we measure and compare prediction quality in a symbolic/deterministic context that also punish for complexity? I struggled on this in my phd thesis and had to brew my own Minimum Description Length implementation because I found nothing applicable. The statisticians I interrogated were like "don't try to do model selection on symbolic models, make them probabilistic and work from there"
nsajko 18 hours ago [-]
The approach here assigns a cost to each operator, variable and constant. See docs.
verdverm 17 hours ago [-]
Are you familiar with NSGA-2? That was the one I used in my PhD work.
A lot of research in GP/SR went toward methods for balancing density and selection from the pareto front.
One of the bigger problems is growth of meaningless equation material to counteract other pressures in the algorithm, which itself leads to premature convergence on suboptimal solutions. I had some thoughts at the time about ways to penalize this during the growth phase in my own algo, but never got around to them. Mainly needs bookkeeping for contribution to the final equation output.
19 hours ago [-]
19 hours ago [-]
saithound 18 hours ago [-]
PySR itself is still alright, mostly because of previous development. But not for long. Quality control has collapsed, and e.g. the examples in the repo are now embarrassing vibeslop.
Pangram reports 100% AI on most comment blurbs in the examples directory. That in itself is not the problem, AI-generated code could still be valid. But in this case, the AI-generated examples have the characteristic shape of slop: plausible sounding technical jargon, combined with zero actual understanding of either the target domain, or even evolutionary search (loss function design). They were vibed by people who had no idea what they were doing.
I suggest taking a look at the "evolved PRNG" example [1].
First, its loss function is basically designed to reward discovering linear feedback shift registers (LFSRs) only. It deliberately gives a massively lower loss to LFSRs than most other functions.
The AI slop phrased this in terms of "certifying the period", but the period certificate used only identifies LFSRs of max period, is unsound on the rest of the search space, and gives a meaningless very high loss to most non-LFSRs. So the regression will almost always just return an LFSR.
Whoever designed the example (Codex I suppose) also knew that most LFSRs are defective, and make poor random number generators. Thus, they included a "shear" penalty to detect such defective LFSRs, but of course they slopped it together.
The 360 correlations they measure should have been added together (having just one strong correlation is bad, and having two strong correlations is twice as bad as havin one). Instead, they take the MEAN of the correlations, so a perfect correlation contributes only 1/360=0.00278 to the loss.
And indeed, the result of their example run, reported in the docs [2] has such a perfect correlation: bit 29 is copied to bit 31 every two iterations.
I would personally hold this opinion of anyone still using genetic/evolutionary inspired algorithms for symbolic regression. There are multiple deterministic algorithms that do not have the same inherent drawbacks and are generally quite good.
Before I wrote my algo, and was trying to scale out GP, I hit the limits of RNG and had to go find some with seriously huge cycle properties. (howany calls before they reached the end and started over producing the same sequence)
archermarks 2 hours ago [-]
Can you share more? I've only heard of genetic-based symbolic regression
verdverm 2 hours ago [-]
I believe I wrote the second deterministic algorithm for SR
- FFX (Fast Function eXtraction) was the year before or so.
- PGE (Prioritized Grammar Enumeration) won best paper at GECCO 2013
And took a stab at vibe coding with a tiny LLM to port it to JAX: https://github.com/verdverm/pge-jax I'll put some more effort (tokens) into this and get it added to SRBench.
Fundamentally, most of GP based SR research fights issues with the algorithm (loss of diversity, equation bloat, premature convergence). I have not seen the field really adopt a new mentality in the 13 years since, not that I have been paying close attention, but I do look at the papers in GECCO GP track from time to time.
archermarks 1 hours ago [-]
Thank you so much for the resources! These look very interesting.
verdverm 18 minutes ago [-]
you're welcome, I have a channel in my discord for this topic if you want to caht and discord is something you use
JanisErdmanis 21 hours ago [-]
Is the Julia part here compiled via JuliaC and if not what would it take to do so?
A deterministic SR engine in Rust, written for SRBench. Happy to see the field getting some front-page attention!
verdverm 14 hours ago [-]
Is your project limited in scope to physics equations? The coefficient snapping is interesting, but seems like it would not work well for problems like predator-prey (https://en.wikipedia.org/wiki/Lotka%E2%80%93Volterra_equatio...). I mainly worked with biology and life-sciences data. (real-world, separate from the synthetic benchmark data)
I introduced a new type of "constant" for well known values like this that sit between measured data and free parameters. I think I had some idea similar to your snapping, i.e. to look for places where free parameters could be replaced with const, but multiples of them was not part of that.
owls-on-wires 3 hours ago [-]
Physics and known-constant problems were definitely the focus. My ambition was to create a fast algorithm that could pull out a nice, “natural-looking” equation from (potentially noisy) data, if one existed. Model ranking and selection, coefficient snapping, noise robustness, and the search patterns were all designed to that end.
For that reason, it does terribly on the SRBench black-box track (it doesn’t like to produce long, overfit models), but (to my knowledge) it scores the highest on exact symbolic recovery, and even with noise it can often find the answer in under 10s (single-thread, no GPU).
verdverm 3 hours ago [-]
Does SRBench still run deterministic algorithms 30 times? A more useful measure than wall clock time is the number of equation evaluations and iterations it takes to get there. One of the benefits to deterministic algos is they only need to be run once, whereas you have to run the rng based algos 30+ times for statistical significance.
My algorithm came from frustration at the lack of rigor in the field. There were no standard benchmarks, results were not reproducible, don't get me started on Korns lol
verdverm 17 hours ago [-]
Is there a paper for your method? I'm always keen to see other deterministic algos for SR, it sounds interesting from what I can glean, but I don't read/write rust so may need an agent to help me decipher
I'll see about getting mine added to SRbench as well
owls-on-wires 3 hours ago [-]
Unfortunately no paper yet, but I’m considering publishing one here soon! As far as agents go, I just pointed Claude at the repo at asked it to explain, and it did a good job. It also helps that it’s not a ton of code, and a lot of it is structurally similar to patterns in training data (in this case, it can recognize a recursive beam-search).
Would highly recommended submitting a PR for SRBench! The maintainers were very responsive, despite one necessary correction mine was merged within 2 weeks.
verdverm 2 hours ago [-]
Well, you've inspired me to work on my revamp, get it benched, make some new additions, and check out your project in more detail.
If you want to chat about papers or have me do a pre-review, I'm more than happy to. Contact methods can be found in my HN profile
A lot of research in GP/SR went toward methods for balancing density and selection from the pareto front.
One of the bigger problems is growth of meaningless equation material to counteract other pressures in the algorithm, which itself leads to premature convergence on suboptimal solutions. I had some thoughts at the time about ways to penalize this during the growth phase in my own algo, but never got around to them. Mainly needs bookkeeping for contribution to the final equation output.
Pangram reports 100% AI on most comment blurbs in the examples directory. That in itself is not the problem, AI-generated code could still be valid. But in this case, the AI-generated examples have the characteristic shape of slop: plausible sounding technical jargon, combined with zero actual understanding of either the target domain, or even evolutionary search (loss function design). They were vibed by people who had no idea what they were doing.
I suggest taking a look at the "evolved PRNG" example [1].
First, its loss function is basically designed to reward discovering linear feedback shift registers (LFSRs) only. It deliberately gives a massively lower loss to LFSRs than most other functions.
The AI slop phrased this in terms of "certifying the period", but the period certificate used only identifies LFSRs of max period, is unsound on the rest of the search space, and gives a meaningless very high loss to most non-LFSRs. So the regression will almost always just return an LFSR.
Whoever designed the example (Codex I suppose) also knew that most LFSRs are defective, and make poor random number generators. Thus, they included a "shear" penalty to detect such defective LFSRs, but of course they slopped it together.
The 360 correlations they measure should have been added together (having just one strong correlation is bad, and having two strong correlations is twice as bad as havin one). Instead, they take the MEAN of the correlations, so a perfect correlation contributes only 1/360=0.00278 to the loss.
And indeed, the result of their example run, reported in the docs [2] has such a perfect correlation: bit 29 is copied to bit 31 every two iterations.
[1] https://github.com/astroautomata/PySR/blob/master/examples/p...
[2] https://pysr.ai/v2.7.0/examples/objectives#inventing-a-pseud...
I would personally hold this opinion of anyone still using genetic/evolutionary inspired algorithms for symbolic regression. There are multiple deterministic algorithms that do not have the same inherent drawbacks and are generally quite good.
Before I wrote my algo, and was trying to scale out GP, I hit the limits of RNG and had to go find some with seriously huge cycle properties. (howany calls before they reached the end and started over producing the same sequence)
- FFX (Fast Function eXtraction) was the year before or so.
- PGE (Prioritized Grammar Enumeration) won best paper at GECCO 2013
I go into the problems and causes in my paper: https://github.com/verdverm/go-pge/blob/master/pge_gecco2013...
And took a stab at vibe coding with a tiny LLM to port it to JAX: https://github.com/verdverm/pge-jax I'll put some more effort (tokens) into this and get it added to SRBench.
Fundamentally, most of GP based SR research fights issues with the algorithm (loss of diversity, equation bloat, premature convergence). I have not seen the field really adopt a new mentality in the 13 years since, not that I have been paying close attention, but I do look at the papers in GECCO GP track from time to time.
A deterministic SR engine in Rust, written for SRBench. Happy to see the field getting some front-page attention!
I introduced a new type of "constant" for well known values like this that sit between measured data and free parameters. I think I had some idea similar to your snapping, i.e. to look for places where free parameters could be replaced with const, but multiples of them was not part of that.
For that reason, it does terribly on the SRBench black-box track (it doesn’t like to produce long, overfit models), but (to my knowledge) it scores the highest on exact symbolic recovery, and even with noise it can often find the answer in under 10s (single-thread, no GPU).
My algorithm came from frustration at the lack of rigor in the field. There were no standard benchmarks, results were not reproducible, don't get me started on Korns lol
I'll see about getting mine added to SRbench as well
Would highly recommended submitting a PR for SRBench! The maintainers were very responsive, despite one necessary correction mine was merged within 2 weeks.
If you want to chat about papers or have me do a pre-review, I'm more than happy to. Contact methods can be found in my HN profile
A deterministic algorithm for Symbolic Regression, which I recently ported to JAX
(original post refreshed by HN mods if I understand correctly, maybe dang or tomhow will chime in on this)