A Hitchhiker's Guide to the Age of Big Data

A Hitchhiker's Guide to the Age of Big Data

Don’t Panic

Introduction

There is, or will soon be, a problem in this society: most people produce data most of the time, are governed by data at the same time, and yet know almost nothing about it. Humanity has produced a large number of articles in response, but most of them begin with combinations of mathematical symbols, which is strange, because the things that are ignorant are not those symbols.

The purpose of this article is to give you everything you need to enter the age of big data, even if you have never heard of calculus. It may look long, and it certainly contains some false or at least imprecise statements, but it surpasses more advanced and old-fashioned academic works in two extremely important ways.

First, it is completely free. Second, at the beginning, in large friendly letters, it says “Don’t Panic”. Its content also lets readers skip difficult formulas and directly obtain the high-level picture. I would call this approach math-free.

Read more

Risk of Rain 2 Engineer Exponential Build Optimization

In Risk of Rain 2, a character’s DPS can be decomposed into the product of several factors, such as base damage, attack speed, and critical strike chance. As a result, linearly stacking items that increase attack speed, critical chance, and similar stats can produce exponential DPS growth. Similarly, survivability can be decomposed into total health multiplied by armor. Because item resources are limited, marginal returns are huge, and the costs of improving different stats are inconsistent, it is worth computing the optimal build in advance under an item-count constraint so that DPS and effective health are maximized. This matters most for the Engineer, because turrets deal damage in a pure stationary-output style.

Read more

k-Means Clustering: A Convergence Proof and Two Variants

This article first proves the convergence of the k-means algorithm in a relatively concise and clear way compared with many existing materials online. The second half discusses two variants produced by combining k-means with ideas from GMM and RPCL. This article is not intended as introductory reading for someone encountering k-means for the first time.

Read more
Cliff Walking: Sarsa vs. Q-Learning in Python

Cliff Walking: Sarsa vs. Q-Learning in Python

This article trains an agent in Python to move from the start to the goal without falling off the cliff. It implements and compares the Sarsa and Q-learning algorithms in reinforcement learning, and discusses the factors that shape the final solution.

Read more

Logistic Regression in 20 Lines of Python

Logistic regression is a classification algorithm widely used in industry. Its structure is simple, with the following main advantages and disadvantages:

pros

  1. Training and inference are both fast
  2. Easy to implement
  3. Low memory usage
  4. Good interpretability

cons

  1. Because it cannot fit nonlinear relationships, it places higher demands on feature engineering.
  2. It is relatively sensitive to multicollinearity
Read more