BACKGROUND: Human homologs (FEM1A, FEM1B, FEM1C) of nematode sex determination genes are candidate genes for polycystic ovary syndrome (PCOS). We previously identified a FEM1A mutation (H500Y) in a woman with PCOS; FEM1B has been implicated in insulin secretion. METHODS: Women with and without PCOS (287 cases, 187 controls) were genotyped for H500Y and six FEM1A single nucleotide polymorphisms (SNPs), five FEM1B SNPs and five FEM1C SNPs. SNPs and haplotypes were determined and tested for association with PCOS and component phenotypes. RESULTS: No subject carried the FEM1A H500Y mutation. FEM1A SNPs rs8111933 (P = 0.001) and rs12460989 (P = 0.046) were associated with an increased likelihood of PCOS whereas FEM1A SNP rs1044386 was associated with a reduced probability of PCOS (P = 0.013). FEM1B SNP rs10152450 and a linked SNP were associated with a reduced likelihood of PCOS (P = 0.005), and lower homeostasis model assessment (HOMA) for beta-cell function (HOMA-%B, P = 0.011) and lower HOMA for insulin resistance (HOMA-IR, P = 0.018). FEM1B SNP rs12909277 was associated with lower HOMA-%B (P = 0.008) and lower HOMA-IR (P = 0.037). Haplotype associations were consistent with SNP results, and also revealed association of FEM1B haplotype TGAGG with increased HOMA-%B (P = 0.007) and HOMA-IR (P = 0.024). FEM1C variants were not associated with PCOS. CONCLUSIONS: This study presents evidence suggesting a role for FEM1A and FEM1B in the pathogenesis of PCOS. Only FEM1B variants were associated with insulin-related traits in PCOS women, consistent with prior evidence linking this gene to insulin secretion. Replication of these associations and mechanistic studies will be necessary to establish the role of these genes in PCOS. © The Author 2008. Published by Oxford University Press on behalf of the European Society of Human Reproduction and Embryology. All rights reserved.