You haven’t mentioned what battery you are using. But the problem with using a solar panel on the USB port is that the voltage supplied by the panel in direct sunlight will cause a transistor switch on the Mayfly to essentially “disconnect” the battery from the processor power supply and use the voltage from the panel to supply power to the entire board. But since the solar panel can’t supply an adequate current, then the board will be unstable. That’s why there’s a separate “SOLAR” input on the Mayfly. When you use it as intended, the board is powered by the battery at all times, and the charge regulator on the Mayfly will charge the battery whenever there’s sufficient sunlight. It also works best with a 6v solar panel, so unless you have lots of direct sun, your panel will likely be supplying around 4v, which isn’t going to do much, especially if you’ve got it connected to the USB jack instead.
So I would highly recommend you use a 6v solar panel, and one that is completely waterproof and rugged if you’re using it outdoors. We use panels made by Voltaic Systems (https://www.voltaicsystems.com/2-watt-panel), and you can get a handy bracket for them (https://www.voltaicsystems.com/small-bracket) and an extension cable (https://www.voltaicsystems.com/extension-with-exposed-leads). Just solder a JST connector onto the extension cable, and you’ve got a rugged and weatherproof way to charge the Mayfly. For batteries, a 2200mAh Lipo is usually sufficient and can be charged in less than an hour of sunlight if your Mayfly sleeps or doesn’t have much current draw.