Agorism in reality is just tax and license avoidance. You really can't derive huge amounts of income from your activities without coming under suspicion of some government agency. It's just not practical. So in most cases, especially if you decline to transact in US dollars, you're setting yourself up for a limited amount of money coming in.
Having a shitty business idea is still going to lead to failure under Agorism. Some people just don't have the knack. Lots of business ventures flopped at Porc Fest.
But yes, if your idea involves only transacting with people in the liberty movement, it's probably not going to go too well with you. Limited customers, and people are just not willing to spend much money. Mostly because they choose to not make enough money themselves to be customers of your products.
Needless to say, 95% of my income is from general public business activities. The money I make each year from within the liberty movement is a small pittance.