Information Theory is a branch of mathematics that has many applications in the field of computer science. Such as in cryptography, machine learning and project management.
Information theory is the study of how information can be efficiently stored, transmitted, processed.
Information is what allows one mind to influence another. It is the data that is transmitted from one mind to another during a communication process.
- Bits: measure of surprise, the more surprising an event is, the more information it contains.
I(x)is the information contained in the event
p(x)is the probability of the event
log₂is the logarithm base 2.
To discover the information (
I) contained in the event
x, we need to know the probability of the event
x and apply it to the
Let's assume the probability to snow in 3 different brazilian cities: ❄️ ❄️ 🇧🇷
|Rio de Janeiro||0.001%||
Notice that the more surprising an event is, the more information it contains.
No one would be surprised if it snowed in São Joaquim, but if it snowed in São Paulo or Rio de Janeiro, it would be a big surprise, would be an event with a lot of information.
Amount of disorder in a system.
As things grow, the entropy grows together.
More information available = more disorder.
- Entropy is a measure of chaos. The more entropy, the more chaotic the system.
"The increase of disorder or entropy is what distinguishes the past from the future, giving a direction to time."
Entropy is a natural law of the universe. It is a measure of disorder.
Do your fast-growing company is more chaotic now than it was when you started? Yes, it is.
Why? Because it is growing.
What is the solution? Try to keep the chaos under control as eliminating it is impossible.
- Entropy is an important concept in information theory. Probably the most important concept.
H(X)is the entropy of the random variable
p(xᵢ)is the probability of the event
log₂is the logarithm base 2.
To discover the entropy (H) of the random variable
X, we need to know the probability of each event
xᵢ and apply it to the formula.
You are a data scientist building a decision tree to predict which ad you should show to an user based on its profile. You have 3 possible ads to show to the user:
You want to reduce the entropy of the system, so you need to know how much entropy you have now.
- The entropy of the system is 1.1446 bits.
- You can reduce the entropy by showing the ad with the highest probability.
- Because it is the least surprising ad. Probably the user has already seen it many times.
- Machine Learning
- Project Management
- Data Compression
- Encryption is the process of encoding information.
- Decryption is the process of decoding information.
- The more information you have, the more difficult it is to decode the information.
- Good encryption algorithms should have a high entropy.
- UUIDs usually have around 122 bits of entropy.
- Information gain.
- Information bottleneck.
- Mutual information.
- Decision trees.
- Risk management.
- Minimum Description Length.