Human babies have a natural desire to interact with new toys and objects, through which they learn how the world around them works, e.g., that glass shatters when dropped, but a rubber ball does not. When their predictions are proven incorrect, such as when a glass does not shatter after a fall, they feel surprised. This, in turn, impacts their subsequent decisions and makes them reconsider their beliefs, e.g., they may continue dropping the glass until they realize it does not shatter because it falls on a carpet. Similarly, human adults and other species react differently to new and surprising events compared to familiar and expected ones, possibly due to the vital importance of these events in a continuously changing world with sparse resources. The influence of novelty and surprise on the brain and behavior has been a prominent topic in neuroscience and psychology. However, quantifying surprise and novelty and their contribution to various brain functions remain unresolved and disputed. In this thesis, I take a mathematical approach to study (i) definitions of surprise and novelty as well as (ii) their computational roles in the brain. I first present an exhaustive analysis of 18 mathematical definitions of surprise, investigating their similarities, differences, and conditions that make them indistinguishable. I classify these definitions into different categories and propose a unified framework for systematic comparison of different approaches to quantifying surprise. Within this framework, I propose a formalism that distinguishes novelty from surprise. I use this mathematical distinction to construct a Reinforcement Learning (RL) model of human behavior that describes surprise as the modulator of the learning speed ('learning from the unexpected') and novelty as the drive of goal-directed exploration ('seeking the new'). I test this model against behavioral and electroencephalogram (EEG) data of human participants and show that both surprise and novelty are crucial determinants of human behavior in volatile environments with sparse rewards. Then, I ask whether these results generalize to stochastic environments where novelty-driven exploration has proven suboptimal. To answer this question, I compare models of exploration driven by novelty and different surprise definitions in stochastic environments. Testing these models against the behavioral data of human participants shows that human exploration closely aligns with novelty-driven models, even when they are not optimal. This establishes novelty as a dominant drive of human goal-directed exploration. This thesis offers a comprehensive comparison of various computational models and definitions of surprise and novelty, from both mathematical and experimental points of view. Our theoretical findings allow fresh insights into previous research and lay a foundation for future theoretical and experimental studies. Moreover, our computational modeling of experimental data expands our understanding of the computational roles of surprise and novelty in learning and exploration.