Did you take the TB off or loosen the fuel line?
What I'm thinking is a very slight leak on the fuel connection to the TB.
Fuel is thicker than air, so you may not see any fuel leak but air can get in.
When you let it sit a long time the fuel drains back to the tank - replaced by the air. When you turn the key on the pump runs only a short time, then shuts off until it starts - not quite long enough to get fuel all the way back up from the tank.
Try turning the key on - wait 10 seconds, turn it off again and then try starting it. That gives the pump two shots to get the fuel up. If it starts then --.