There are two kinds of exotic spheres: bp spheres, which bound parallelizable manifolds, and non-bp spheres, or very exotic spheres, which do not. In the 1960s, W.-C. Hsiang showed that in each dimension where bp spheres exist, there is at least one which admits infinitely many inequivalent smooth free $S^1$-actions, and in each dimension congruent to $3$ modulo $4$, there is at least one bp sphere which admits infinitely many inequivalent smooth free $S^3$-actions. On the other hand, for each fixed prime $p$, smooth free $S^1$- and $S^3$- actions are only known to exist on finitely many very exotic spheres with nontrivial $p$-local Kervaire--Milnor invariant, all in dimension less than approximately $p^3$. In this paper, we use topological modular forms to detect smooth free $S^1$- and $S^3$-actions on infinite families of very exotic spheres with nontrivial $2$- and $3$-local Kervaire--Milnor invariants.