- Automatic analysis of skin abnormality is an effective way for medical experts to facilitate diagnosis procedures and improve their capabilities. Efficient and accurate methods for analysis of the skin abnormalities such as convolutional neural networks (CNNs) are typically complex. Hence, the implementation of such complex structures in portable medical instruments is not feasible due to power and resource limitations. CNNs can extract features from the skin abnormality images automatically. To reduce the burden of the network for feature extraction, which can lead to the network simplicity, proper input color channels could be selected. In this paper, a pruning framework is proposed to simplify these complex structures through the selection of most informative color channels and simplification of the network. Moreover, hardware requirements of different network structures are identified to analyze the complexity of different networks. Experimental results are conducted for segmentation of images from two publicly available datasets of both dermoscopy and non-dermoscopy images. Simulation results show that using the proposed color channel selection method, simple and efficient neural network structures can be applied for segmentation of skin abnormalities.