A second-order topological insulator in d dimensions is an insulator which has no d-1 dimensional topological boundary states but has d-2 dimensional topological boundary states. It is an extended notion of the conventional topological insulator. Higher-order topological insulators have been investigated in square and cubic lattices. In this Letter, we generalize them to breathing kagome and pyrochlore lattices. First, we construct a second-order topological insulator on the breathing Kagome lattice. Three topological boundary states emerge at the corner of the triangle, realizing a 1/3 fractional charge at each corner. Second, we construct a third-order topological insulator on the breathing pyrochlore lattice. Four topological boundary states emerge at the corners of the tetrahedron with a 1/4 fractional charge at each corner. These higher-order topological insulators are characterized by the quantized polarization, which constitutes the bulk topological index. Finally, we study a second-order topological semimetal by stacking the breathing kagome lattice.