LKL stats & predictions

Addition Resources for Basketball Enthusiasts

• Player Profiles: Get to know your favorite athletes better through detailed profiles highlighting their careers and achievements.
• Historical Highlights: Relive iconic moments from past seasons with our collection of memorable highlights from LKL history.yaochaojiang/2018_04_20_Self-supervised_learning<|file_sep|>/README.md # Self-supervised learning Self-supervised learning (SSL) is one kind of unsupervised learning which learns representations from unlabeled data by constructing pretext tasks. The term self-supervised learning was first used by Richard Szeliski et al., where he presented an algorithm which learned visual representations by predicting color channels from grayscale images. ## Motivation Traditional supervised learning algorithms are data-hungry methods which rely heavily on labeled data for training. However human-annotated labels are expensive or even impossible to obtain for large-scale datasets. Unsupervised learning methods such as PCA/ICA can discover some simple patterns like principal directions but cannot learn useful features which can be applied to downstream tasks. Self-supervised learning aims at training models on unlabeled data using constructed pretext tasks which can learn useful features. ## Current status SSL has been applied widely in computer vision tasks such as image classification (see [Supervised pretraining](https://arxiv.org/pdf/1801.06146.pdf)), object detection ([SSL for Object Detection](https://arxiv.org/pdf/1909.11157.pdf)), segmentation ([SSL for Semantic Segmentation](https://arxiv.org/pdf/1805.07984.pdf)), tracking ([SSL for Visual Tracking](https://arxiv.org/pdf/1906.02222.pdf)) etc. In natural language processing (NLP), [BERT](https://arxiv.org/pdf/1810.04805.pdf) has achieved state-of-the-art results on many NLP tasks by pretraining on masked language modeling task (which is also called self-supervised learning). In speech recognition task (which can be considered as sequence-to-sequence problem), [wav2vec](https://arxiv.org/pdf/1909.12459.pdf) learns representations by predicting future audio frames. ## Methods ### Reconstruction-based methods Reconstruction-based methods are based on autoencoder structure. It learns representations by reconstructing inputs. The pretext task here is predicting inputs given their corrupted version.  #### Autoencoder Autoencoder consists of an encoder function $phi$ which projects inputs into latent space $mathbf{z}$, and a decoder function $psi$ which reconstructs inputs from latent space $mathbf{z}$. The objective function here is minimizing reconstruction loss between inputs $mathbf{x}$  and reconstructed inputs $hat{mathbf{x}}$: $$ mathcal{L} = mathbb{E}_{mathbf{x}}[lVert mathbf{x} - hat{mathbf{x}}rVert^2] $$ where $hat{mathbf{x}} = psi(phi(mathbf{x}))$ Autoencoders can be trained using backpropagation algorithm. #### Denoising autoencoder Denoising autoencoder (DAE) is similar to autoencoder except that it predicts clean inputs given their noisy version. In this way DAE forces encoder $phi$ to learn robust representations.  The objective function here is minimizing reconstruction loss between clean inputs $mathbf{x}$  and reconstructed inputs $hat{mathbf{x}}$: $$ mathcal{L} = mathbb{E}_{tilde{mathbf{x}}sim q(tilde{mathbf{x}}|mathbf{x})}[lVert mathbf{x} - hat{mathbf{x}}rVert^2] $$ where $tilde{mathbf{x}}$ denotes corrupted inputs generated by $q(tilde{mathbf{x}}|mathbf{x})$, i.e., noise model. $hat{mathbf{x}} = psi(phi(tilde{mathbf{x}}))$ #### Variational autoencoder Variational autoencoders (VAEs) add an extra constraint which forces latent space $phi(mathbf{x})$ to follow Gaussian distribution. This constraint helps prevent overfitting. The objective function here consists of two parts: 1) Minimize reconstruction loss between inputs $mathbf{x}$  and reconstructed inputs $hat{mathbf{x}}$: $$ mathcal{L}_1 = mathbb{E}_{q_{phi}(mathbf{z}|mathbf{x})}[log p_{theta}(mathbf{x}|mathbf{z})] $$ where $q_{phi}(cdot)$ denotes encoder function which approximates posterior $p(cdot|cdot)$, and $p_{theta}(cdot|cdot)$ denotes decoder function which approximates likelihood $p(cdot|cdot)$. 2) Minimize Kullback–Leibler divergence between encoder output $q_{phi}(cdot)$ and prior $p_{theta}(cdot)$: $$ begin{align} begin{split} mathcal{L}_2 &= D_{KL}(q_{phi}(cdot|cdot)||p_{theta}(cdot))\ &= int q_{phi}(cdot|cdot)logdfrac{q_{phi}(cdot|cdot)}{p_{theta}(cdot)}dcdot end{split} end{align} $$ The total loss function is: $$ L = max_theta,min_phi (mathcal{L}_1 - beta times mathcal{L}_2) $$ where $beta$ controls trade-off between reconstruction loss $mathcal{L}_1$ and KL divergence loss $beta times mathcal{L}_2$. #### Generative adversarial network Generative adversarial networks (GANs) consists of two neural networks: generator network $G$ which generates fake samples, and discriminator network $D$ which distinguishes fake samples from real samples. Training GANs involves training generator network $G$ to fool discriminator network $D$, i.e., maximizing probability that discriminator network $D$ mistakenly classify fake samples generated by generator network $G$ as real samples. Training discriminator network $D$ involves training discriminator network $D$ to correctly classify real samples as real samples, and fake samples generated by generator network $G$ as fake samples. The objective functions here are: $$ V(G,D) = min_D max_G V(D,G) = min_D max_G (log D(x) + log(1-D(G(z)))) $$ where $x$ denotes real samples, and $G(z)$ denotes fake samples generated by generator network $G$. ### Contrastive methods Contrastive methods aim at learning representations by contrasting positive examples against negative examples. Positive examples are similar pairs while negative examples are dissimilar pairs. Contrastive methods have been applied widely in representation learning problems such as image retrieval task, where contrastive loss is used as objective function.  Here we use Siamese neural network structure which has two identical subnetworks sharing weights. Each subnetwork takes one example as input (positive or negative example). Each subnetwork projects input into latent space. Latent vectors from two subnetworks are fed into distance metric function (e.g., Euclidean distance function) to calculate similarity between two examples. The objective function here is contrastive loss defined below: $$ L(x_1,x_2,y) = (1-y)dfrac{1}{2}(d_W)^2 + ydfrac{1}{2}{max(0,m-d_W)}^2 $$ where $(x_1,x_2)$ denotes two input examples, $d_W=lVert f(x_1,W)-f(x_2,W)rVert^2$, where $f(x,W)$ denotes projection output, $m > 0$ denotes margin parameter, and $y=0$ if $(x_1,x_2)$ is positive pair while $y=1$ if $(x_1,x_2)$ is negative pair. ### Prediction-based methods Prediction-based methods aim at learning representations by predicting one part from another part. It can be considered as self-supervised learning version of supervised learning problem where labels are generated automatically by splitting inputs into two parts: input part (which will be fed into model) and label part (which will be predicted). By doing so prediction-based methods can learn useful representations without manual annotations.  Here we use Masked Language Modeling (MLM) task proposed by BERT as example. Masked Language Modeling task aims at predicting original tokens given masked tokens. It masks 15% tokens randomly selected from each sequence using [BERT masking strategy](https://github.com/google-research/bert/blob/master/machine_comprehension/prepro_utils.py#L370), and then predicts original tokens given masked tokens using BERT model structure. ## References * [Supervised pretraining](https://arxiv.org/pdf/1801.06146.pdf) * [SSL for Object Detection](https://arxiv.org/pdf/1909.11157.pdf) * [SSL for Semantic Segmentation](https://arxiv.org/pdf/1805.07984.pdf) * [SSL for Visual Tracking](https://arxiv.org/pdf/1906.02222.pdf) * [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/pdf/1810.04805.pdf) * [wav2vec: Unsupervised Pre-training for Speech Recognition](https://arxiv.org/pdf/1909.12459.pdf) * [Auto-Encoding Variational Bayes](https://arxiv.org/pdf/1312.6114.pdf) * [Improving Variational Inference with Inverse Autoregressive Flow](https://arxiv.org/pdf/1606.04934.pdf) * [A Neural Algorithm of Artistic Style](https://arxiv.org/pdf/1508.06576.pdf) * [Generative Adversarial Nets](https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf) * [Siamese Neural Networks For One Shot Image Recognition](http://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf) <|repo_name|>yaochaojiang/2018_04_20_Self-supervised_learning<|file_sep|>/requirements.txt tensorflow-gpu==1.13.* numpy