Visualizing the stock market structure of sklearn
Visualizing the stock market structure
https://scikit-learn.org/stable/auto_examples/applications/plot_stock_market.html#stock-market
此例使用了集中非监督学习技术， 从历史报价波动中提取股票市场结构。
分析的目标量值为股票日变化，包括 开市报价，和闭市报价。
报价 === quotes
This example employs several unsupervised learning techniques to extract the stock market structure from variations in historical quotes.
The quantity that we use is the daily variation in quote price: quotes that are linked tend to cofluctuate during a day.
Learning a graph structure ---- 使用逆协方差矩阵
使用稀疏的逆协方差矩阵估计，来找出报价之间的条件相关性。
逆协方差矩阵描绘了一个图， 图中元素是 两只股票的 关联关系 -- 相关关系。
股票代码之间的连接，有助于表示股票价格的波动。例如 一个股票当天跌了， 另外一个相关度最高的股票，大概率也是下跌的情况。
Symbol === 股票代码。
We use sparse inverse covariance estimation to find which quotes are correlated conditionally on the others.
Specifically, sparse inverse covariance gives us a graph, that is a list of connection.
For each symbol, the symbols that it is connected too are those useful to explain its fluctuations.
逆协方差矩阵
https://www.quora.com/What-is-the-inverse-covariance-matrix-What-is-its-statistical-meaning#:~:text=The%20inverse%20covariance%20matrix%2C%20commonly%20referred%20to%20as,variable%20j%20by%20reading%20off%20the%20%28i%2Cj%29-th%20index.
协方差矩阵，往往带有噪声。
例如 x -》 y -》 z 这样计算xz之间的相关关系，其中包括了 xy 和 yz的相关度。
逆协方差矩阵，屏蔽这些噪音， 只计算偏相关关系。
xy 和 yz， 并不计算 xz之间的相关性。
Another view to add to the previous answers:
# Covariance is a measure of how much two variables move in the same direction (i.e. vary together).
ISSUE: Covariance between two variables also captures the effects of the others. For example, strong covariance between variable 1 and variable 2 with a third variable 3 will induce a covariance between variable 1 and variable 2.
In other words, the correlation matrix captures a lot of noise.
# Therefore, in order to gain in interpretability, one can derive the inverse covariance matrix (also called precision matrix).
It gives covariation of two variables while conditioning on the potential influence of the other ones involved in the analysis. In other words, it removes the effect of other variables. The precision matrix thus allows to obtain direct covariation between two variables by capturing partial correlations. It gives the conditional independent covariation between two variables. Say differently, variable 1 and variable 2 are not connected if the covariance can be explain by a third variable 3.
The precision matrix makes the interactions between variables more interpretative and more robust to the confounds.
# Sources: Varoquaux et al, 2010, smith et al, 2011, Varoquaux and Craddock, 2013.
偏相关性存在于逆协方差矩阵。
So here's another perspective, to add to Charles H Martin and Vladimir Novakovski's answer.
The inverse covariance matrix, commonly referred to as the precision matrix displays information about the partial correlations of variables.
With the covariance matrix Σ
, one observes the unconditional correlation between a variable i, to a variable j by reading off the (i,j)-th index. It may be the case that the two variables are correlated, but do not directly depend on each other, and another variable k explains their correlation. If we displayed this information on a conditional independence graph, it would look like:
i−k−j
So for example: if k is the event that it rains, i is the event that your lawn is wet and j is the event that your driveway is wet, then you will notice that i and j are heavily correlated, but once you condition on k - they are pretty uncorrelated. (If you don't believe that, for sake of argument - say you hose the lawn pretty sporadically, and wash your car on the driveway whenever you arbitrarily remember to, and those are the only other ways your lawn / driveway gets wet).A partial correlation describes the correlation between variable i and j, once you condition on all other variables. If i and j are conditionally independent, such as in the example, then the (i,j)-th element of your precision matrix
Σ−1i,j
will equal zero. Also, if your data follows a multivariate normal then the converse is true, a zero element implies conditional independence. Deriving information about conditional independence is really helpful in understanding how your covariates relate to one another. Short of drawing a full causal graph, this is probably the best summary of covariate relations that you can hope to extract.
https://scikit-learn.org/stable/auto_examples/covariance/plot_sparse_cov.html#sphx-glr-auto-examples-covariance-plot-sparse-cov-py
其与协方差矩阵同等重要。
To estimate a probabilistic model (e.g. a Gaussian model), estimating the precision matrix, that is the inverse covariance matrix, is as important as estimating the covariance matrix. Indeed a Gaussian model is parametrized by the precision matrix.
Clustering
使用紧密度传播聚类方法，来发现报价行为相似的股票。
紧密度传播不强制聚类的大小，自动选择聚类的中心。
协方差方法探索的是 变量之间的条件关系， 而聚类方法则表示有相似影响。
We use clustering to group together quotes that behave similarly. Here, amongst the various clustering techniques available in the scikit-learn, we use Affinity Propagation as it does not enforce equal-size clusters, and it can choose automatically the number of clusters from the data.
Note that this gives us a different indication than the graph, as the graph reflects conditional relations between variables, while the clustering reflects marginal properties: variables clustered together can be considered as having a similar impact at the level of the full stock market.
Embedding in 2D space
为了可视化， 需要将不同的股票映射到二维空间。
利用流行学习（非线性降维方法），将股票行为降解到二维空间。
For visualization purposes, we need to lay out the different symbols on a 2D canvas. For this we use Manifold learning techniques to retrieve 2D embedding.
Visualization
图中每个节点表示一个股票。
边表示 股票之间的相关程度 -- 粗细表示， 来自于 稀疏逆协方差矩阵。
节点的颜色， 表示聚类结果。
The output of the 3 models are combined in a 2D graph where nodes represents the stocks and edges the:
cluster labels are used to define the color of the nodes
the sparse covariance model is used to display the strength of the edges
the 2D embedding is used to position the nodes in the plan
This example has a fair amount of visualization-related code, as visualization is crucial here to display the graph. One of the challenge is to position the labels minimizing overlap. For this we use an heuristic based on the direction of the nearest neighbor along each axis.
Out:
Fetching quote history for 'AAPL' Fetching quote history for 'AIG' Fetching quote history for 'AMZN' Fetching quote history for 'AXP' Fetching quote history for 'BA' Fetching quote history for 'BAC' Fetching quote history for 'CAJ' Fetching quote history for 'CAT' Fetching quote history for 'CL' Fetching quote history for 'CMCSA' Fetching quote history for 'COP' Fetching quote history for 'CSCO' Fetching quote history for 'CVC' Fetching quote history for 'CVS' Fetching quote history for 'CVX' Fetching quote history for 'DD' Fetching quote history for 'DELL' Fetching quote history for 'F' Fetching quote history for 'GD' Fetching quote history for 'GE' Fetching quote history for 'GS' Fetching quote history for 'GSK' Fetching quote history for 'HD' Fetching quote history for 'HMC' Fetching quote history for 'HPQ' Fetching quote history for 'IBM' Fetching quote history for 'JPM' Fetching quote history for 'K' Fetching quote history for 'KMB' Fetching quote history for 'KO' Fetching quote history for 'MAR' Fetching quote history for 'MCD' Fetching quote history for 'MMM' Fetching quote history for 'MSFT' Fetching quote history for 'NAV' Fetching quote history for 'NOC' Fetching quote history for 'NVS' Fetching quote history for 'PEP' Fetching quote history for 'PFE' Fetching quote history for 'PG' Fetching quote history for 'R' Fetching quote history for 'RTN' Fetching quote history for 'SAP' Fetching quote history for 'SNE' Fetching quote history for 'SNY' Fetching quote history for 'TM' Fetching quote history for 'TOT' Fetching quote history for 'TWX' Fetching quote history for 'TXN' Fetching quote history for 'UN' Fetching quote history for 'VLO' Fetching quote history for 'WFC' Fetching quote history for 'WMT' Fetching quote history for 'XOM' Fetching quote history for 'XRX' Fetching quote history for 'YHOO' /home/circleci/miniconda/envs/testenv/lib/python3.9/site-packages/numpy/core/_methods.py:202: RuntimeWarning: invalid value encountered in subtract x = asanyarray(arr - arrmean) Cluster 1: Apple, Amazon, Yahoo Cluster 2: Comcast, Cablevision, Time Warner Cluster 3: ConocoPhillips, Chevron, Total, Valero Energy, Exxon Cluster 4: Cisco, Dell, HP, IBM, Microsoft, SAP, Texas Instruments Cluster 5: Boeing, General Dynamics, Northrop Grumman, Raytheon Cluster 6: AIG, American express, Bank of America, Caterpillar, CVS, DuPont de Nemours, Ford, General Electrics, Goldman Sachs, Home Depot, JPMorgan Chase, Marriott, 3M, Ryder, Wells Fargo, Wal-Mart Cluster 7: McDonald's Cluster 8: GlaxoSmithKline, Novartis, Pfizer, Sanofi-Aventis, Unilever Cluster 9: Kellogg, Coca Cola, Pepsi Cluster 10: Colgate-Palmolive, Kimberly-Clark, Procter Gamble Cluster 11: Canon, Honda, Navistar, Sony, Toyota, Xerox
Code
相比原始代码，添加了打印，分析每个过程的含义。
# Author: Gael Varoquaux gael.varoquaux@normalesup.org # License: BSD 3 clause import sys import numpy as np import matplotlib.pyplot as plt from matplotlib.collections import LineCollection import pandas as pd from sklearn import cluster, covariance, manifold print(__doc__) # ############################################################################# # Retrieve the data from Internet # The data is from 2003 - 2008. This is reasonably calm: (not too long ago so # that we get high-tech firms, and before the 2008 crash). This kind of # historical data can be obtained for from APIs like the quandl.com and # alphavantage.co ones. symbol_dict = { 'TOT': 'Total', 'XOM': 'Exxon', 'CVX': 'Chevron', 'COP': 'ConocoPhillips', 'VLO': 'Valero Energy', 'MSFT': 'Microsoft', 'IBM': 'IBM', 'TWX': 'Time Warner', 'CMCSA': 'Comcast', 'CVC': 'Cablevision', 'YHOO': 'Yahoo', 'DELL': 'Dell', 'HPQ': 'HP', 'AMZN': 'Amazon', 'TM': 'Toyota', 'CAJ': 'Canon', 'SNE': 'Sony', 'F': 'Ford', 'HMC': 'Honda', 'NAV': 'Navistar', 'NOC': 'Northrop Grumman', 'BA': 'Boeing', 'KO': 'Coca Cola', 'MMM': '3M', 'MCD': 'McDonald\'s', 'PEP': 'Pepsi', 'K': 'Kellogg', 'UN': 'Unilever', 'MAR': 'Marriott', 'PG': 'Procter Gamble', 'CL': 'Colgate-Palmolive', 'GE': 'General Electrics', 'WFC': 'Wells Fargo', 'JPM': 'JPMorgan Chase', 'AIG': 'AIG', 'AXP': 'American express', 'BAC': 'Bank of America', 'GS': 'Goldman Sachs', 'AAPL': 'Apple', 'SAP': 'SAP', 'CSCO': 'Cisco', 'TXN': 'Texas Instruments', 'XRX': 'Xerox', 'WMT': 'Wal-Mart', 'HD': 'Home Depot', 'GSK': 'GlaxoSmithKline', 'PFE': 'Pfizer', 'SNY': 'Sanofi-Aventis', 'NVS': 'Novartis', 'KMB': 'Kimberly-Clark', 'R': 'Ryder', 'GD': 'General Dynamics', 'RTN': 'Raytheon', 'CVS': 'CVS', 'CAT': 'Caterpillar', 'DD': 'DuPont de Nemours'} print("-------------sorted(symbol_dict.items())-------------------") print(sorted(symbol_dict.items())) symbols, names = np.array(sorted(symbol_dict.items())).T print("-------------symbols-------------------") print(symbols) print("-------------names-------------------") print(names) quotes = [] for symbol in symbols: print('Fetching quote history for %r' % symbol, file=sys.stderr) url = ('https://raw.githubusercontent.com/scikit-learn/examples-data/' 'master/financial-data/{}.csv') quotes.append(pd.read_csv(url.format(symbol))) close_prices = np.vstack([q['close'] for q in quotes]) open_prices = np.vstack([q['open'] for q in quotes]) # The daily variations of the quotes are what carry most information variation = close_prices - open_prices # ############################################################################# # Learn a graphical structure from the correlations edge_model = covariance.GraphicalLassoCV() # standardize the time series: using correlations rather than covariance # is more efficient for structure recovery X = variation.copy().T X /= X.std(axis=0) edge_model.fit(X) print("-------- edge_model.covariance_ -----------") print(edge_model.covariance_) print("-------- edge_model.precision_ -----------") print(edge_model.precision_) # ############################################################################# # Cluster using affinity propagation _, labels = cluster.affinity_propagation(edge_model.covariance_, random_state=0) print("-------- labels -----------") print(labels) n_labels = labels.max() for i in range(n_labels + 1): print('Cluster %i: %s' % ((i + 1), ', '.join(names[labels == i]))) # ############################################################################# # Find a low-dimension embedding for visualization: find the best position of # the nodes (the stocks) on a 2D plane # We use a dense eigen_solver to achieve reproducibility (arpack is # initiated with random vectors that we don't control). In addition, we # use a large number of neighbors to capture the large-scale structure. node_position_model = manifold.LocallyLinearEmbedding( n_components=2, eigen_solver='dense', n_neighbors=6) embedding = node_position_model.fit_transform(X.T).T print("-------- X.shape -----------") print(X.shape) print("-------- X.T.shape -----------") print(X.T.shape) print("-------- X.T -----------") print(X.T) print("-------- embedding.shape -----------") print(embedding.shape) print("-------- embedding -----------") print(embedding) # ############################################################################# # Visualization plt.figure(1, facecolor='w', figsize=(10, 8)) plt.clf() ax = plt.axes([0., 0., 1., 1.]) #plt.axis('off') # Display a graph of the partial correlations partial_correlations = edge_model.precision_.copy() print("-------- np.diag(partial_correlations) -----------") print(np.diag(partial_correlations)) print("-------- np.sqrt(np.diag(partial_correlations)) -----------") print(np.sqrt(np.diag(partial_correlations))) d = 1 / np.sqrt(np.diag(partial_correlations)) print("-------- d -----------") print(d) partial_correlations *= d print("-------- partial_correlations -----------") print(partial_correlations) print("-------- d[:, np.newaxis] -----------") print(d[:, np.newaxis]) partial_correlations *= d[:, np.newaxis] print("-------- partial_correlations -----------") print(partial_correlations) print("-------- np.triu(partial_correlations, k=1) -----------") print(np.triu(partial_correlations, k=1)) print("-------- np.abs(np.triu(partial_correlations, k=1)) -----------") print(np.abs(np.triu(partial_correlations, k=1))) non_zero = (np.abs(np.triu(partial_correlations, k=1)) > 0.02) print("-------- non_zero -----------") print(non_zero) # Plot the nodes using the coordinates of our embedding plt.scatter(embedding[0], embedding[1], s=100 * d ** 2, c=labels, cmap=plt.cm.nipy_spectral) # Plot the edges start_idx, end_idx = np.where(non_zero) print("-------- start_idx -----------") print(start_idx) print("-------- end_idx -----------") print(end_idx) # a sequence of (*line0*, *line1*, *line2*), where:: # linen = (x0, y0), (x1, y1), ... (xm, ym) segments = [[embedding[:, start], embedding[:, stop]] for start, stop in zip(start_idx, end_idx)] print("-------- segments -----------") print(segments) values = np.abs(partial_correlations[non_zero]) print("-------- values -----------") print(values) print("-------- values.max() -----------") print(values.max()) lc = LineCollection(segments, zorder=0, cmap=plt.cm.hot_r, norm=plt.Normalize(0, .7 * values.max())) lc.set_array(values) lc.set_linewidths(15 * values) ax.add_collection(lc) # Add a label to each node. The challenge here is that we want to # position the labels to avoid overlap with other labels for index, (name, label, (x, y)) in enumerate( zip(names, labels, embedding.T)): dx = x - embedding[0] dx[index] = 1 dy = y - embedding[1] dy[index] = 1 this_dx = dx[np.argmin(np.abs(dy))] this_dy = dy[np.argmin(np.abs(dx))] if this_dx > 0: horizontalalignment = 'left' x = x + .002 else: horizontalalignment = 'right' x = x - .002 if this_dy > 0: verticalalignment = 'bottom' y = y + .002 else: verticalalignment = 'top' y = y - .002 plt.text(x, y, name, size=10, horizontalalignment=horizontalalignment, verticalalignment=verticalalignment, bbox=dict(facecolor='w', edgecolor=plt.cm.nipy_spectral(label / float(n_labels)), alpha=.6)) plt.xlim(embedding[0].min() - .15 * embedding[0].ptp(), embedding[0].max() + .10 * embedding[0].ptp(),) plt.ylim(embedding[1].min() - .03 * embedding[1].ptp(), embedding[1].max() + .03 * embedding[1].ptp()) plt.show()
