Objective: To compare documentation, referral practices, and appropriateness of telephone advice provided by pediatric residents with that provided by nurses utilizing computerized protocols. Methods: The frequency of documentation of patient information and the frequency of recommended dispositions were compared for 100 calls managed by each group. In addition, three faculty pediatricians rated each call assessing completeness of recorded history and advice and appropriateness of the advice. Results: The only difference found in documentation of patient information was that the nurses recorded the patient's phone number more often than residents (88% vs. 66%, p < 0.01). Nurses were more likely to recommend higher intensity levels of service than residents (Mann-Whitney test, p < 0.01). Inter-rater agreement among the physician evaluators was low (kappa = 0.04-0.26). When the three evaluators did reach a consensus, the advice given was usually judged to have been appropriate. Evaluators agreed with one another more often in assessment of appropriateness of advice for nurse calls than for resident calls (51% vs. 23%, 95% CI of the difference: 15-42%). Conclusions: We demonstrated a tendency for the nurses in our sample to recommend higher intensity follow-up services than residents with little difference in documentation practices. The low level of inter-rater agreement in our sample illustrates the difficulty of evaluating telephone encounters based on written documentation alone. Higher inter-evaluator agreement was achieved for nurse calls and may provide an indirect indicator of quality. Implications for practice: Practitioners must be able to evaluate the efficacy of computerized telephone triage systems as well as telephone management provided by office personnel. This study highlights the problems encountered in performing that evaluation and the need for more efficient evaluation systems.