Water pollution is becoming an increasing threat to China's sustainable development. To respond to this challenge, China has pledged to cut emissions of two major water pollutants, chemical oxygen demand (COD) and ammonia nitrogen (NH4), and has disaggregated the national target among provinces. However, the abatement potential and costs have not been thoroughly assessed. This paper aims to examine the reduction potential and associated costs of COD and NH4 in the Chinese industrial sector. A parametric directional distance function is applied to modeling joint production, in which COD and NH4 are treated simultaneously as byproducts of the production process. Using provincial data from 2003 to 2012, we find that 13.18% of COD and 13.27% of NH4 can be reduced if all provinces perform efficiently. The average abatement cost to cut one additional unit of COD and NH4 is 710 and 7,390 Yuan/kg, respectively. The abatement cost is significantly correlated with the economic development level, pollution intensity and capital-labor ratio. Our results call for a market-based instrument, such as an Emission Trading System, to assist China in achieving this environmental goal in a cost-effective way. Moreover, it will become more difficult and costly to control COD, while there still exists a 'win-win' opportunity to control NH4 emissions through efficiency improvements.