We introduce a Rigid-Body Fluctuating Immersed Boundary (RB-FIB) method to perform large-scale Brownian dynamics simulations of suspensions of rigid particles in fully confined domains, without any need to explicitly construct Green's functions or mobility operators. In the RB-FIB approach, discretized fluctuating Stokes equations are solved with prescribed boundary conditions in conjunction with a rigid-body immersed boundary method to discretize arbitrarily shaped colloidal particles with no-slip or active-slip prescribed on their surface. We design a specialized Split-Euler-Maruyama temporal integrator that uses a combination of random finite differences to capture the stochastic drift appearing in the overdamped Langevin equation. The RB-FIB method presented in this work only solves mobility problems in each time step using a preconditioned iterative solver and has a computational complexity that scales linearly in the number of particles and fluid grid cells. We demonstrate that the RB-FIB method correctly reproduces the Gibbs-Boltzmann equilibrium distribution and use the method to examine the time correlation functions for two spheres tightly confined in a cuboid. We model a quasi-two-dimensional colloidal crystal confined in a narrow microchannel and hydrodynamically driven across a commensurate periodic substrate potential mimicking the effect of a corrugated wall. We observe partial and full depinning of the colloidal monolayer from the substrate potential above a certain wall speed, consistent with a transition from static to kinetic friction through propagating kink solitons. Unexpectedly, we find that particles nearest to the boundaries of the domain are the first to be displaced, followed by particles in the middle of the domain.