Brain-inspired intelligence using the cognitive dynamic system (CDS) concept is proposed to control the quality-of-service (QoS) over a long-haul fiber-optic link that is nonlinear and with non-Gaussian channel noise. Digital techniques such as digital-back-propagation (DBP) assume that the fiber optic link parameters, such as loss, dispersion, and nonlinear coefficients, are known at the receiver. However, the proposed CDS does not need to know about the fiber optic link physical parameters, and it can improve the bit error rate (BER) or enhance the data rate based on information extracted from the fiber optic link. The information extraction (Bayesian statistical modeling) using intelligent perception processing on the received data, or using the previously extracted models in the model library, is carried out to estimate the transmitted data in the receiver. Then, the BER is sent to the executive through the main feedback channel and the executive produces actions on the physical system/signal to ensure that the BER is continuously under the forward-error-correction (FEC) threshold. Therefore, the proposed CDS is an intelligent and adaptive system that can mitigate disturbance in the fiber optic link (especially in an optical network) using prediction in the perceptor and/or doing proper actions in the executive based on BER and the internal reward. A simplified CDS was implemented for nonlinear fiber optic systems based on orthogonal frequency division multiplexing (OFDM) to show how the proposed CDS can bring noticeable improvement in the system’s performance. As a result, enhancement of the data rate by 12.5% and the Q-factor improvement of 2.74 dB were achieved in comparison to the conventional system (i.e., the system without smart brain).