Would a custom Y exhaust system improve the engine performance of a V6? If yes to what extent?

A true 'Y' pipe could give you as much as 5% more horsepower/wattage/torc. The Y pipe alone will not provide a huge increase, unless the exhaust manifolds are also tuned. Providing each exhaust pulse an equal length path with low resistance is critical to reducing "backpressure" on the engine. By allowing the exhaust to fully evacuate the cylinder and exit the pipes quickly, you keep the exhaust gasses hot enough to continue burning reducing polluting emissions. At the same time you prevent an excess of exhaust fumes from remaining in the cylinder during the intake stroke and contaminating the mixture. The result is a cleaner burning engine with more thrust and higher volumetric efficiency, meaning more power and millage fro the same fuel consumption while producing less emissions.