Measurements of charged jet production as a function of centrality are presented for p-Pb collisions recorded at $\sqrt{s_{\rm NN}} = 5.02$ TeV with the ALICE detector. Centrality classes are determined via the energy deposit in neutron calorimeters at zero degree, close to the beam direction, to minimise dynamical biases of the selection. The corresponding number of participants or binary nucleon-nucleon collisions is determined based on the particle production in the Pb-going rapidity region. Jets have been reconstructed in the central rapidity region from charged particles with the anti-$k_{\rm T}$ algorithm for resolution parameters $R = 0.2$ and $R = 0.4$ in the transverse momentum range 20 to 120 GeV/$c$. The reconstructed jet momentum and yields have been corrected for detector effects and underlying-event background. In the five centrality bins considered, the charged jet production in p-Pb collisions is consistent with the production expected from binary scaling from pp collisions. The ratio of jet yields reconstructed with the two different resolution parameters is also independent of the centrality selection, demonstrating the absence of major modifications of the radial jet structure in the reported centrality classes.