[P] I solved BipedalWalker-v3 (~310 rating) with eigenvalues. The whole coverage matches on this publish.

hop hop hop

Perhaps you've seen my earlier publish about fixing CartPole-v1 with simply bitwise ops. I've tried to scale this strategy to more durable environments, but it surely didn't get me too far. Nevertheless, I used to be impressed by completely unrelated article – Eigenvalues as fashions. Whereas the writer is speaking about matrices of dimension 3×3 and bigger I went the opposite manner – I restricted the load matrix to be diagonal. This implies the eigenvalues are merely the vector parts themselves. To get the utmost or minimal eigenvalue we actually simply take the max or min worth from the vector. Easy.

Now we are able to outline a operate EIGEN(x) that outputs these eigenvalues:

EIGEN(x) = A + xB 

The place x is any scalar enter and A and B are diagonal matrices – our parameters.

Should you learn the "Eigenvalues as fashions" article you already know that we are able to take max of the eigenvalues to outline a convex operate and min to outline a concave one:

convex(x) = max(EIGEN(x)) concave(x) = min(EIGEN(x)) 

For the reason that concave operate is definitely a convex one with flipped signal we are able to outline the DC operate which is a distinction of two convex features and it seems it might approximate quite a lot of features. So in our case it’s truly a sum:

DC(x) = convex(x) + concave(x) 

This provides us scalar again and so long as the variety of eigenvalues is greater than 2 (3,4,…) this operate is non-linear and given sufficient eigenvalues we’ve fairly highly effective approximator! (when there are solely 2 eigenvalues then the operate collapses to only a sum of these 2 eigenvalues = linear)

We will simply lengthen it to high-dimensional inputs:

EIGEN(x1, x2, x3) = A + x1*B1 + x2*B2 + x3*B3 

Nevertheless, if EIGEN(x) stays linear, the ensuing DC(x) consists of flat planes, so probably not nice for "easy" features, so I made a small modification. I allowed the linear projection to "bend" itself by including a quadratic time period:

LINEAR(x1,x2,x3) = x1*B1 + x2*B2 + x3*B3 EIGEN(x1,x2,x3) = A + LINEAR(x1,x2,x3) + Okay * LINEAR(x1,x2,x3)^2 

The Okay listed below are coefficients that outline how a lot to "bend". This hybrid can mannequin each the sharp resolution boundaries and easy areas. For instance an image under is an ideal match I educated utilizing 4 eigenvalues showcasing the sharp resolution within the center and easy wells on the left and proper facet:

Double Effectively Potential with sharp resolution boundary

The one drawback is that the min and max ops have points with gradients – the gradient flows solely to the winner, however this may be solved through the use of softmax within the backward go (the softmax is a by-product of logsumexp which is a easy approximation of max) – the STE trick. This works fairly nicely and we maintain environment friendly min/max ops within the ahead go (inference).

Now my unfastened interpretation of the DC(x) operate we've outlined is that it represents a single neuron, however a particular one which has a number of connections to a single enter x.

So for the BipedalWalker-v3 drawback I needed to do the only factor potential. Since we’ve now "fairly highly effective" neuron, I simply assigned 4 separate neurons controlling every joint independently. I educated them immediately with PPO and by some means they’ve learnt to synchronize with none bodily hyperlink between them.
There are not any connections between the neurons. The left leg has no concept the best leg exists. The whole mannequin is simply 4 decentralized and stateless "Eigen / DC" neurons, every doing its personal factor.

I've used 6 eigenvalues for every neuron and distilled the coverage right down to 69 strains of python code which you’ll simply copy-paste and run when you’ve got gymnasium and numpy put in. The whole logic for "hopping"/"strolling" is actually right here:

import numpy as np import gymnasium as gymnasium A = np.array([ 0.167, 0.146, 0., -0.063, -0.110, 0.029, -0.114, 0.081, -0.101, -0.072, 0.094, -0.066, 0.238, -0.027, 0.019, -0.131, -0.018, 0.088, 0.046, 0.106, 0.062, 0.086, -0.134, 0.039, ]) B_GENERATOR = np.concatenate([np.linspace(-1.272, 1.491, 30), [0.0]]) B_IDX = np.array([ 0x51D9E52FCC93970, 0x8B16E9C669B3A7E, 0x8B14B3FB78A725D, 0xAC3D1745F8BDB3A, 0x9464F640CAF7989, 0x4F8EB62D4762DB2, 0x5A91E21DD052D6B, 0x4286A081D293E30, 0x6318E5797E7352C, 0x73E0C92DECF39EF, 0x6B54C4B0C882D48, 0x8ADFE73E2A5C9AE, 0x3A4C5491684AFCF, 0x8794C67A2D8B20C, 0x649AC52A2B539A9, 0x725EE779CA9314D, 0x7BD5E5321E7FBCA, 0x5BDEE431B0F4D6B, 0x4AD918359164A13, 0x62FCC6FBCC5A4EE, 0x4C97E433CE6226C, 0x4B9AB6910CF316F, 0xF79CC6A48A5AD4B, 0x3C0A848A1EF428A, 0x629CD421DE7C5D6, 0x6B9F5727DE5794B, 0x5C24677A1E8FBD3, 0x779EA879CCF212B, 0xF79DE73FCF5F9FE, 0xF323E8BDEE5B3CC, 0x639D27FA486B18B, 0x5B3DE73FDE5F96A, 0x53E2F726707BBC9, 0x93E2C4298D4392F, 0xF7BC863A6C73969, 0x5A96E8219E6318E, 0x4AD4FF2D7E74DDE, 0x6264D625E85C210, 0x5B98A7A614F7970, 0x7A60A6B59E5B14D, 0xF39C8F797E637CE, 0x731CB4799EF79C7, 0xF2A3E5B3CE8397E, 0x63D4E8A9928B96C, 0x839CB82D6C743CC, 0x7795EF29F1F2DAC, 0x67A4C43A6FF3DDE, 0x7560D8C1CA741CF, ], dtype=np.int64) Okay = np.array([ -0.037, 0.018, 0.027, -0.006, 0.021, 0.041, 0.017, -0.011, 0., 0.011, 0., 0.020, -0.025, -0.023, 0.015, 0.008, -0.012, 0., -0.096, 0., 0., 0.014, -0.039, 0., ]) def coverage(state): shifts = np.arange(0, 60, 5, dtype=np.int64) indices = (B_IDX[:, None] >> shifts) & 0x1F idx = indices.flatten().reshape(24, 24) B = B_GENERATOR[idx] LINEAR = state @ B EIGEN = A + LINEAR + (Okay * (LINEAR**2)) EIGEN = EIGEN.reshape(4, 6) DC = np.max(EIGEN, axis=1) + np.min(EIGEN, axis=1) return np.clip(DC, -1, 1) def run(): env = gymnasium.make("BipedalWalker-v3", render_mode=None) scores = [] print("Working 10 episodes...") for i in vary(10): obs, _ = env.reset() ep_rew = 0 whereas True: motion = coverage(obs) obs, r, time period, trunc, _ = env.step(motion) ep_rew += r if time period or trunc: break scores.append(ep_rew) print(f"Ep {i+1}: {ep_rew:.2f}") print("-" * 20) print(f"Avg: {np.imply(scores):.2f}") print(f"Min: {np.min(scores):.2f} Max: {np.max(scores):.2f}") env.shut() if __name__ == "__main__": run() 

This could get you common rating of about 310 which is taken into account "solved" for this atmosphere.

Whereas it's now not simply "bitwise ops" like in CartPole-v1 case I feel it shares the identical spirit.

=== EDIT ===

I simply realized you’ll be able to set all of the Okay coefficients to ZERO and it doesn’t harm the efficiency. So the "quadratic time period" and "easy" half was not mandatory in spite of everything (for this drawback), so it’s even much less strains of code 🙂

=== EDIT 2 ===

Nevertheless after second thought whether or not you’ll be able to simply drop the Okay coefficients – "quadratic time period" – I’m not 100% certain because the script I posted above has truncated and quantized weights – the unique full mannequin scored increased ~315 and above, so Okay may truly may be related for the total mannequin in spite of everything to get even higher rating and possibly it makes it extra "steady", however I haven't carried out any assessments.

=== EDIT 3 ===
Repair typos.