The exact identity of castrate-resistant (CR) cells and their relation to CR prostate cancer (CRPC) is unresolved. We use single-cell gene profiling to analyze the molecular heterogeneity in basal and luminal compartments. Within the luminal compartment, we identify a subset of cells intrinsically resistant to castration with a bi-lineage gene expression pattern. We discover LY6D as a marker of CR prostate progenitors with multipotent differentiation and enriched organoid-forming capacity. Lineage tracing further reveals that LY6D⁺ CR luminal cells can produce LY6D⁻ luminal cells. In contrast, in luminal cells lacking PTEN, LY6D⁺ cells predominantly give rise to LY6D⁺ tumor cells, contributing to high-grade PIN lesions. Gene expression analyses in patients' biopsies indicate that LY6D expression correlates with early disease progression, including progression to CRPC. Our studies thus identify a subpopulation of luminal progenitors characterized by LY6D expression and intrinsic castration resistance. LY6D may serve as a prognostic maker for advanced prostate cancer.