A system for evaluating a change: Conducting and evaluating on-farm research to support management decisions

Virtually every dairy producer conducts on-farm research, either formally or informally. Proper planning is required to ensure that meaningful information is generated. Poorly conducted on-farm studies are, at a minimum, a waste of time and resources. In worst case scenarios, poor studies can result in faulty, and costly, decision making. The most important component of an evaluation system is a commitment from all farm personnel to follow the evaluation process. With these pieces in place, on-farm evaluation systems will produce a substantial return on a farm’s personnel investment.


On-farm research trials are a much abused but potentially valuable tool for evaluating the impact of feeding or management changes. In the dairy industry chances are, if a farm is not currently involved with on-farm research, it will be either conducting or evaluating on-farm research in the near future. Properly conducted on-farm research can be invaluable in good decision making. On-farm research allows companies to test their products in commercial production settings, determine odds of success for a product under a variety of management schemes and environments, and promote awareness of their product. For the farm enterprise, on-farm research may provide a way to evaluate practicality and efficacy of a new product in that particular farm environment without substantial financial outlay. For both the supplier and the farm enterprise, on-farm research can demonstrate the economic impact of a new product, providing a foundation for the decision making process (DeGroot, 1970 and Galligan, et al., 1991). For all the positive aspects of on-farm research, serious pitfalls may occur for both the company and the farm enterprise. Potential pitfalls range from wasted time and effort to making an incorrect decision about a product. The purpose of the following paper is to help producers make the most out of their own and other producers’ on-farm evaluations.

When to Consider Conducting On-Farm Research

A properly conducted on-farm research trial can provide valuable management information for that farm. However, the decision to conduct research on-farm should not be made lightly. Conducting a trial to obtain free product or a cash payment is generally a poor business decision. In this situation, the farm has little or no incentive to conduct a high quality evaluation of the management change, since the farm’s primary goal was attained once the product or payment was received. Conducting on-farm research will strain labor and management resources and may occasionally strain facilities. Preventing gaps in the data and/or adherence to the research protocol requires full buy-in of management and labor to the research process. Otherwise, it is nearly impossible to avoid corner-cutting under the strains of day to day farm conditions. In the end, that cash payment or free product may seem poor compensation for the effort expended.

On the other hand, conducting a study on-farm to gather information to support management decisions is an excellent investment of farm resources. A properly conducted on-farm evaluation requires clear goals, a concise and manageable protocol, and collection of data exactly as described in the protocol. The reward for this investment in management and personnel time is not just determining whether a given change will increase productivity of the cows or dairy farm, but also in increasing management control of the operation through the information collected. Information generated during the course of a properly conducted on-farm evaluation varies widely, including determination of such things as site specific response economics, weak points in the herd management or nutrition program, management and labor implications of a new product or process, and collection of additional herd data. The returns from this information will more than repay the substantial investment in farm personnel resources required to conduct on-farm research.

Conducting Meaningful Evaluations on Your Farm

Having made the decision to conduct an evaluation of a management change, the first step is to determine the goals or objectives of the trial. The goals must be both measurable and attainable. An example of a measurable goal would be “reduced somatic cell count or number of treatable mastitis cases”; a similar, but unmeasurable goal would be “reduced mastitis”. An attainable goal is merely a reasonable goal. Most producers would never expect or test for an unrealistic level of response, like a 5 lb (2.27 kg) per day gain in young heifers or a 25 lb (11.4 kg) per head per day milk response. However, through ignorance or lack of planning, it is easy to set goals which are unattainable due to the precision of our measuring ability and/or equipment. For instance, in a herd with an 8% incidence of milk fever ( or 8 cases of milk fever for every 100 calvings) a test with 50 cows to determine whether a product reduces milk fever will result in an unattainable goal. Of the 25 cows in each test group (control and test product), only 2 animals would be expected to have milk fever. To have any chance of demonstrating a meaningful response under these conditions, the test product would need to completely eliminate milk fever. Unfortunately, this is definitely not an attainable goal.

If a sponsoring company is involved, their research objectives are likely to be predetermined. However, adding sub-objectives or secondary objectives specific to your farm enterprise is completely reasonable and good business sense. You may want to evaluate economic returns of a practice or gather some baseline information to support other management decisions. For example in a comparison between two milk replacers, the original trial objective may be to compare weight gains on a weekly basis from birth to weaning. However, you may wish to evaluate how the calves will handle the post-weaning transition or the stress of dehorning and tail docking. To obtain better management information, adding an additional weighing one week post weaning would be required and the trial objectives should be modified to reflect this.

Once the goals for a trial are established, a trial protocol to meet those goals must be developed. The protocol will describe the animals to be used, duration of the testing, responses to be measured, data to be collected, and management of the animals during the trial.

The goals of a study will dictate the research animal to be used, i.e.- growing heifers, dry cows, early lactation cows. However, in on-farm trials facility and management constraints may interfere with the requirements of the evaluation. A common example would be any of the feed additives which improve milk production and reduce metabolic diseases in early lactation. Recommended feeding practices for many of these additives are to begin feeding during the prefresh period and continue feeding during early lactation. For most herds, feeding two separate prefresh groups and another two fresh groups is impossible. As a result, many on-farm tests of these products are conducted around peak lactation, rather than during the time a cow is most likely to respond to these products. While these products may increase production during early lactation, it is likely that their full economic impact will not be seen because they were tested during the wrong part of lactation. In this situation, we run the risk of making an incorrect decision because our data was not generated with the type of animal that would be receiving this product under normal management conditions. By conducting our trial improperly, we run the risk of concluding that a product or practice is not profitable when it actually is. In this situation, farm management would be better off relying on properly conducted research studies done elsewhere rather than an improperly conducted on-farm evaluation. In general, a research study, whether conducted on-farm or at a research facility should use the type of animals which would be receiving the test product or practice in a commercial setting.

As with the research animal to be used, the length of time encompassed by the trial will be determined by the trial goals. For instance, if body condition score gain is being measured, trial length must be long enough to result in a measurable condition score gain. In some situations, historic herd data might be useful in determining the proper length of time for a study. In the previous data, herd condition scores might be used to determine the average length of time required for 1 condition score gain. Another example would be a product or practice that will be evaluated for reducing displaced abomasum (DA) incidence. From herd records, we might find that most DA’s occur during the first fourteen days of lactation. Our trial would then need to cover at least the first fourteen days of lactation. A longer trial length might be used to ensure that DA’s are indeed prevented and not postponed or to evaluate other responses such as condition score change and peak production. In a trial to evaluate a product’s ability to improve fertility, historical herd data could be used to determine the time elapsed between freshening and pregnancy.

In developing a research protocol, we prefer to evaluate a number of related responses in an attempt view the big picture. For instance, in evaluating a product which should increase energy intake, milk production and composition and body condition score changes will all be monitored. If the product works as advertised, we will expect to find a combination of increased milk production and body condition scores. In addition, milk components may indicate that we have increased the supply of either fatty acids or amino acids by stimulating the ruminal fermentation as the product increased energy intake. In this situation, we might also suspect that fertility would be increased; however, due to time, space, and measuring precision considerations may make evaluation of herd fertility an unattainable goal.

One of the most important aspects in developing a protocol is the experimental design. Experimental design is the process by which a fair test is ensured. In other words, a good experimental design prevents test products and procedures from being unfairly assisted or hindered in affecting cow performance. For example, if a test product is fed to one pen of cows which is pre-peak and the control pen of cows is post-peak milk production, increased milk production will be observed with the test product regardless of the products’ efficacy. In designing an on-farm evaluation, always remember that it is just as much work to conduct a trial with poor experimental design as one with an excellent design.

A complete discussion of experimental design pitfalls would be beyond the scope of this paper. The interested reader is referred to Jones et al. (1998); however, a brief discussion of the most common evaluation tools is presented here. A very common evaluation technique is to use DHIA data in a system such as Dairy Comp 305 (Valley Ag Software, Tulare, CA) or CTAP (Texas DHIA, College Station, TX). While these systems are convenient to use, they are generally used to compare present milk production to previous results. This is a potentially dangerous situation, as changes in weather, forages, and other factors may cause production changes that are attributed to the product or procedure being evaluated. Another drawback of these software packages is the fact that no statistical evaluation is conducted. As a result, differences in milk production may be due to chance rather than the product being tested.

A statistical evaluation will show whether a difference is due to chance and measurement variations or to a repeatable treatment effect. However, statistical evaluations are vulnerable to abuse also. Using too few animals in an evaluation can lead us to conclude incorrectly that a product or procedure was ineffective in altering animal performance. As a rule of thumb, a minimum of 20 lactating cows per treatment should be used in milk production studies. For heifer growth studies, at least 25 calves per treatment are required. Evaluations of reproductive performance and fresh cow metabolic diseases require substantially larger numbers of animals, ranging from 50-100 cows per treatment, depending on the herd’s baseline performance.

Another misuse of statistical evaluations is failure to include a proper control group. A very common statistical design is to measure performance of a group or herd of cows under normal herd conditions, then apply the test product to the entire group and evaluate, and finally re-evaluate performance under the normal herd conditions. The control group in this situation consists of the same cows as the treatment group; however, control and test periods are separated by time. Therefore, this design can be rendered ineffective by changes in environmental or herd conditions during the course of the test. Some of the major changes in environmental and herd conditions can be anticipated and controlled, for instance by conducting the evaluation during a period of steady forage supplies and temperatures. However, any number of unanticipated, uncontrollable factors may still invalidate the evaluation.

Comparing two similar groups of cattle at the same time removes concerns about differences in management or environment that could obscure responses to the test product or procedure. This experimental design can be criticized because effect of the cattle group is not independent from the effect of the test product (Jones et al. 1998). However, potential differences between groups (pens) of cattle can be tested prior to beginning the evaluation. Despite its drawbacks, this design is the least likely to result in incorrect conclusions due to uneven influences of environmental or management factors on evaluation groups. This experimental design will result in the most appropriate protocol for an on-farm evaluation.

In conducting on-farm evaluations, it is critical that the protocol be evaluated for feasibility from both a personnel and equipment standpoint. Remember both regular and weekend or relief personnel must be able to perform the tasks in the protocol. Weekends require special consideration in developing an on-farm evaluation protocol, because neither evaluation nor normal farm duties end on Friday. It is not uncommon to find personnel resources stretched thinly on the weekend and the added tasks of an on-farm research study may result in both normal herd and evaluation activities being performed inadequately.

We find it helpful to write two separate protocols for evaluations. The first protocol should detail all aspects of the evaluation from housing, feeding and management of the evaluation animals to sample and data collection, to laboratory analysis of the samples and statistical analysis of the data. In this protocol, no detail is too small or insignificant. If this type of protocol is written properly, a person with no knowledge of the farm or study should be able to duplicate your on-farm study in every detail. While a research sponsor should provide a thorough research protocol, farm personnel may need to add further detail to the protocol for their specific farm. Adding detail to the existing protocol will prevent miscommunication or lack of communication from rendering a study useless. For instance, farm personnel may be well aware that certain pens or sections of the barn are less well ventilated than others or are fed at quite different times (say before and after breakfast), but a researcher not associated with the farm may not know this information. A detailed protocol will also provide all farm personnel with a means to double check procedures and processes at any time and on any day, regardless of the availability of farm management and any research personnel involved. Lastly, a detailed protocol will identify potential conflicts between evaluation and routine herd activities and prevent scheduling the same person for two tasks which occur at the same time.

The second evaluation protocol should be written simply and concisely. This protocol is best described as a task list. It should list evaluation responsibilities of all farm personnel. Individual checklists for complicated protocols or activity calendars for infrequent (monthly or weekly) activities may also be helpful.

Having completed both the detailed and simplified evaluation protocols, as well as any supplementary datasheets or logs, farm management and research personnel must inform all affected farm personnel about the trial. In the communication process it is critical to relay two pieces of information. The first is that the entire evaluation project will only be as reliable as its parts. In other words, if any part of the trial is conducted sloppily, the results will be suspect. The second, and most important, piece of information is why the farm is doing the trial. Farm personnel who do not understand the reasoning behind the extra tasks involved in on-farm evaluations are less likely to be sure the protocol is being followed. Make sure that weekend and relief personnel are included in these communications. It is also essential that all involved personnel understand that another person who might perform their job duties must know about any extra duties or record keeping. During this communication process, unanticipated conflicts between evaluation and routine farm duties may come to light. This is a further advantage of communicating the protocol prior to trial initiation.

As part of the pre-trial communication efforts, researchers and/or farm management personnel must ensure that each person involved in the evaluation has both the knowledge and the tools to complete the tasks assigned to them. A good research rule is that the easier it is to do a task well, the higher the chances that the task will be done properly. Each employee should practice and be comfortable with any new tasks associated with the evaluation. In addition, management must make sure that all required equipment is readily available and in working order. Any measurement tools, such as scales, pH meters, or thermometers, should be checked for accuracy. Back up systems should be identified where practical. Backup systems can include anything from purchasing spare thermometers to finding out which neighbors have Koster testers that can be borrowed.

Datasheets must be placed in convenient locations. Providing extra datasheets on a clipboard with an attached pen will help prevent data loss. Datasheets should be designed to be easy to fill out; they should require minimal writing and be large enough to read and write on easily.

Now that the on-farm evaluation is ready to start, management personnel must be prepared to monitor the process. Continual monitoring and quality control are necessary to make sure that the farm’s investment in starting an evaluation will pay off with usable data to support the decision making process.

One of the most common pitfalls in on-farm evaluations is that treatments get intermixed and are given to the wrong animal or group of animals. Color coding is one of the easiest and most effective ways to avoid this problem. While well-rested, minimally stressed people generally have no problem matching numbers, letters or even words with treatment regimens, even the best managed farms will experience chaotic moments. Color coding will make sure that fatigue or other activities won’t result in treatment mix ups. In addition, color coding removes concerns about employees with undocumented reading disabilities or inadequate command of English. When using color coding, be sure that color blindness is not a problem for any of the employees involved.

Continual monitoring also helps prevent animals receiving the wrong product or no product at all. Disappearance of test product should be checked frequently to be sure that the expected amounts are indeed being used.

Datasheets should be checked frequently to be sure data are being entered correctly by all personnel involved. In addition, routine datasheet checks reemphasize to farm personnel the importance of the data collection process.

All measuring equipment must be checked for proper function to ensure that data collected are reliable. One of the most critical pieces of equipment for data collection, the ballpoint pen, is often ignored. Writing implements should be kept with datasheets, preferably attached in some manner. Ballpoint pens should be used instead of pencils or felt-tip pens to prevent observations from being smudged or accidentally erased. Waterproof markers may also be good choices in some situations, but can be hard to read. Personnel involved in data collection should be familiar with expected values for the data to be collected. In this way, problems resulting from misplaced decimal points, transposed numbers or an equipment misuse or malfunction will be caught before substantial damage is done.

Evaluating On-Farm Evaluations Conducted Elsewhere

Part of a system for evaluating a change is deciding that the change needs to be considered in the first place. One reason to consider changing a feeding or management practice is that research conducted on another farm suggests that a management change will be beneficial. Any on-farm research presented to support a product or practice should have been conducted with the same considerations as discussed above. In addition, look for a complete description of the trial conditions including; ration components, forage analyses, performance of both the herd and test cows, feeding and other management practices. As for your own on-farm studies, look for consistent performance responses. It is not uncommon to observe milk production increases that occur at the expense of body weight reserves.

The more similar the test farm’s management and animal performance is to your own herd, the more likely your responses will mirror those of the other farm. In fact, if test conditions are very similar to your own herd, monitoring herd production with CTAP, Dairy Comp 305, or a similar program may be adequate to evaluate the success of a change. If the test animals’ performance (either growth or lactation) is either substantially higher or lower than your herd’s, a successful on-farm test may not guarantee a successful response on your farm. In either case, conducting your own on-farm test may be required to determine whether this change will be profitable under your herd conditions.


Developing a system to evaluate a change in feeding or management practices requires proper planning to ensure that meaningful information is generated. Clear, measurable and attainable goals are the first step in developing an evaluation system. The evaluation goals will determine many of the testing conditions required. Ultimately, the goals will determine the best type of evaluation, either on-farm research or herd performance monitoring. The most important component of an evaluation system is a commitment from all farm personnel to follow the evaluation process. With these pieces in place, on-farm evaluation systems will produce a substantial return on a farm’s personnel investment.


De Groot, M.H. 1970. Optimal statistical decisions. McGraw-Hill, New York, NY.

Jones, L.R., N. St-Pierre, J. Siciliano-Jones, and R.D. Muller. 1998. Interpretation and design of non-regulatory on-farm feeding trials. J. Dairy Sci. 81(Suppl. 1):307abstr.

Galligan, D.T., W. Chalupa, and C.F. Ramberg. 1991. Application of type I and type II errors in dairy farms management decision making. J. Dairy Sci. 74:902.


Lawrence R. Jones

Lawrence R. Jones
4 articles

Vice-President, FARME Institute

Dr. Jones served as General Manager of Dairy Development International, an 850 cow research-oriented dairy facility in Homer, New York.

From 1989 until 1996 Dr. Jones served as an Assistant Professor in Dairy Management at Cornell University with a 60% extension and 40% research appointment. Following that appointment he served as a special projects consultant for both national and international agricultural companies.

Dr. Jones received his Masters Degree in Dairy Science and Dairy Biochemistry and his Ph. D. in Animal Sciences/Dairy Herd Management, both from the University of Illinois.

Read more about Lawrence R. Jones here

Read more »

F.A.R.M.E. Institute

F.A.R.M.E. Institute

FARME Institute's goal is to provide top quality, client-oriented, independent and confidential research and product development in ruminant nutrition.

Read more »