So if I remember correctly(don't have time to look at the code right now), the machine code only calculates and updates the screen data, and the BASIC portion just randomly places starting cells. If that's the case, you might improve it by creating an interface where you can enter the cells via joystick, then press START or something to start/stop the cell calculations. Maybe have SELECT only run one generation per press. Then the next BASIC type improvements you could make are saving/loading cell layouts.
If you want to just mess around with the generation rules, like you said you would have to modify the machine language subroutine. The Wikipedia page(I think) would have rule variations and certain patterns that would work well under the new rules.
Another improvement would let it run in different graphics mode, but that would also require messing with the machine code.
You could also remove the machine language routine and perform the cell calculations in BASIC. It would run slower, but it would be much easier modify on the fly.