There is an extensive collection of internet resources for mixtures on David Dowe's mixture modelling page.
The following references include papers on length-frequency analysis [2, 3, 8], and the first comprehensive monographs on mixtures [1, 6, 9]. There is an extensive bibliography in [9]. The earliest paper written on mixture estimation, [7], is fascinating to read.
[1] Everitt, B.S. and D.J. Hand (1981). Finite Mixture Distributions. Chapman and Hall, London. xi+143 pp.
[2] Macdonald, P.D.M. and T.J. Pitcher (1979). Age-groups from size-frequency data: a versatile and efficient method of analysing distribution mixtures. Journal of the Fisheries Research Board of Canada 36 , 987-1001.
[3] Macdonald, P.D.M. (1987). Analysis of length-frequency distributions. In R.C. Summerfelt and G.E. Hall [editors], Age and Growth of Fish, Iowa State University Press, Ames, Iowa. pp 371-384.
[4] Macdonald, P. D. M. (1988). Demonstration Examples for MIX 2.3. Ichthus Data Systems, Hamilton, Ontario. 13 pp. ISBN 0-9692305-1-4.
[5] Macdonald, P. D. M., and Green, P.E.J. (1988). User’s Guide to Program MIX: an interactive program for fitting mixtures of distributions. Release 2.3, January 1988. Ichthus Data Systems, Hamilton, Ontario. iv+60 pp. ISBN 0-9692305-1-6.
[6] McLachlan, G.J. and K.E. Basford (1988). Mixture Models: Inference and Applications to Clustering. Marcel Dekker, New York. xi+253 pp.
[7] Pearson, K. (1894). Contributions to the mathematical theory of evolution. Phil. Trans. Roy. Soc. A. 185, 71-110.
[8] Schnute, J., and D. Fournier (1980). A new approach to length-frequency analysis: growth structure. Canadian Journal of Fisheries and Aquatic Sciences 37 , 1337-1351.
[9] Titterington, D.M., A.F.M. Smith and U.E. Makov (1985). Statistical Analysis of Finite Mixture Distributions, Wiley, New York. x+243 pp.