Recent research achievements in learning from demonstration (LfD) illustrate that the reinforcement learning is effective for the robots to improve their movement skills. The current challenge mainly remains in how to generate new robot motions automatically to perform new tasks, which have a similar preassigned performance indicator but are different from the demonstration tasks. To deal with the abovementioned issue, this article proposes a framework to represent the policy and conduct imitation learning and optimization for robot intelligent trajectory planning, based on the improved locally weighted regression (iLWR) and policy improvement with path integral by dual perturbation (PI2-DP). Besides, the reward-guided weight searching and basis function's adaptive evolving are performed alternately in two spaces, i.e., the basis function space and the weight space, to deal with the abovementioned problem. The alternate learning process constructs a sequence of two-tuples that join the demonstration task and new one together for motor skill transfer, so that the robot gradually acquires motor skill, from the task similar to demonstration to dissimilar tasks with different performance metrics. Classical via-points trajectory planning experiments are performed with the SCARA manipulator, a 10-degree of freedom (DOF) planar, and the UR robot. These results show that the proposed method is not only feasible but also effective.