Unmanned Aerial Vehicle (UAV) is a booming trend in major civil and military applications such, but not limited to, transportation, delivery, and surveillance missions. In order to accomplish the mission's objective, trajectory planning must be optimally achieved. The communication link established between the UAV and the ground/aerial stations is the main factor to account for designing the trajectory. However, this link is highly affected by the shape of the topography, especially when the UAV must fly at a low altitude between mountains of variable elevations. Therefore, this paper addresses the challenge of three-dimensional trajectory optimization for low/mid-altitude flying UAVs in complex propagation environments. To tackle this challenge, we propose a system model for the trajectory using the diffraction phenomenon with Multiple Knife Edge (MKE) to model the channel between the UAV and the station when the Line of Sight (LoS) is absent. Then, we propose a joint optimization to minimize the trajectory and maximize the communication quality via the Mixed Integer Linear Programming (MILP) problem design and solution. We validate the proposed approach by using real terrain profiles in the simulations with a rough topography; where the LoS propagation aspect is barely present. Our approach is able to jointly find, when physically achievable, the UAV trajectory with the shortest path and the "best feasible" communication quality.