Similar appearances, frequent contact, and occlusion make current and past appearance cues necessary for reliable pig association. External re-identification (ReID) requires an additional forward pass for appearance extraction, while the extent and use of stored appearance history affect accuracy and computational cost. We propose ORAT (Observation- and Relation-based Affinity Tracker) for online pig tracking. Bounding boxes and identity embeddings are generated from the same queries in one detector forward pass, avoiding separate cropping and ReID inference. Each track maintains cumulative state and bounded memory of accepted recent observations. Relations between current detections and stored observations are recomputed each frame using appearance similarity and elapsed time. With identical D-FINE-X detections on the PigTrack test, base ORAT achieved a Higher Order Tracking Accuracy (HOTA) of 88.1 and server full-pipeline throughput of 48.6 frames per second (FPS). HOTA jointly evaluates detection and association. On private Jochiwon23 videos, ORAT outperformed DiffMOT in association metrics under outer-test evaluation separated from internal model selection. We applied the same architecture to human tracking by training separately on SportsMOT and DanceTrack. In component experiments across five training seeds, adding recent memory increased mean validation HOTA on PigTrack and SportsMOT.