Hardest thing to do is to cut loose a comedy booker that represents work to you. Truthfully though, in some cases it can be the best thing you do. For me I guess it falls under that 'working without a safety net' kinda thing, because honestly, if you're looking for safety as a plan for victory in stand up comedy, ...you've already lost.
This goes for special projects and talk shows too. ;)