Thanks for the reply.
I already bought a 1/4 AAA batterypack from here:
http://www.servohut.com/A%20kan_nimh_batteries.htm
This one has 8 cells, enough for 4 battery replacements:
KAN Ni-MH 9.6V Battery 1/4 AAA 80 mAh Eight Cell Flat P $8.50
Since the cells were already soldered together, I removed two cells from the pack, keeping them soldered together. I removed the leads from the old batteries and soldered them to the new batteries. I'm lousy with a soldering iron and I had some difficulty getting the solder to stick to the battery but I got it eventually. Tanks are good as new.
I think next time, I'll see if I can rig up some kind of solder-free connectors so that battery replacement can be done without soldering. Anyone have any good ideas for that?