White matter (WM) fiber bundles change dynamically with age. These changes could be driven by alterations in axonal diameter, axonal density, and myelin content. In this study, we applied a novel fixel‐based analysis (FBA) framework to examine these changes throughout the adult lifespan. Using diffusion‐weighted images from a cohort of 293 healthy volunteers (89 males/204 females) from ages 21 to 86 years old, we performed FBA to analyze age‐related changes in microscopic fiber density (FD) and macroscopic fiber morphology (fiber cross section [FC]). Our results showed significant and widespread age‐related alterations in FD and FC across the whole brain. Interestingly, some fiber bundles such as the anterior thalamic radiation, corpus callosum, and superior longitudinal fasciculus only showed significant negative relationship with age in FD values, but not in FC. On the other hand, some segments of the cerebello‐thalamo‐cortical pathway only showed significant negative relationship with age in FC, but not in FD. Analysis at the tract‐level also showed that major fiber tract groups predominantly distributed in the frontal lobe (cingulum, forceps minor) exhibited greater vulnerability to the aging process than the others. Differences in FC and the combined measure of FD and cross section values observed between sexes were mostly driven by differences in brain sizes although male participants tended to exhibit steeper negative linear relationship with age in FD as compared to female participants. Overall, these findings provide further insights into the structural changes the brain’s WM undergoes due to the aging process.