In the physical world, cause and effect are inseparable: ambient conditions trigger humans to perform actions, thereby driving status changes of objects. We use these perceived causal relationships to consistently infer actions and effects over time from video, even when they are hidden.
Perceptual causality is the perception of causal relationships from observation. We provide a framework for the unsupervised learning of this perceptual causal structure from video.