Global Financial Data has calculated a 100-share index that is more comprehensive than existing long-term indices of the United States stock market. GFD has collected historical data on U.S. shares from 1791 until 2017 and is using this database to calculate a more representative index of U.S. shares than is currently available.
The GFD-100 Index is superior to existing indices for three reasons. First, the index is capitalization weighted from its beginning in 1791 until 2019. Second, the selection of stocks is based upon all shares that were traded on U.S. exchanges and over-the-counter. The selection of shares is not limited to the New York Stock Exchange (NYSE), but includes shares from regional exchanges and over-the-counter markets. Third, the index includes finance stocks throughout its history. Finance shares were excluded from the S&P 500 Composite until July 1, 1976 when the composition of the S&P 500 was changed from 425 Industrials, 60 Utilities and 15 Rail shares (adopted in March 1957) to 400 Industrials, 40 Utilities, 20 Transportation and 40 Finance stocks.
Few people realize that bank and insurance stocks were excluded from any major United States stock index until 1976. Although bank and finance stocks mostly traded over-the-counter before the 1970s, finance stocks represented between 10% and 20% of the overall capitalization of the stock market before 1976. Bank and insurance companies were excluded because consistent price quotes were not available and finance stocks were not as liquid as the shares traded on exchanges. The exclusion of over-the-counter shares was not limited to banks and insurance companies. Standard Oil and its subsidiaries also traded over-the-counter and were excluded from stock market indices until they moved onto the New York Stock Exchange. Nevertheless, people did invest in bank and insurance stocks, and in the Standard Oil companies, and their exclusion creates biases in the historical calculations of stock market indices.
Existing calculations of long-term stock market returns in the United States are based upon four primary sources: Smith and Cole (1803-1862), Macaulay (1857-1871) Cowles (1871-1928) and Standard and Poor’s (1928-2017). Unfortunately, these indices have major flaws in them that create biases that researchers have tolerated until now because no one has ever collected historical data on U.S. share prices, corporate actions (dividends and splits) and shares outstanding so accurate price and return indices could be calculated. The current S&P Composite includes data from the Cowles Indices from 1871 until 1927, the 90-share daily S&P Index from January 1928 until February 1957, the 500-share composite that excludes finance stocks until July 1976 and the all-sector 500-share index since July 1, 1976.
Global Financial Data has collected data on all major United States exchanges going back to 1791 as well as data on over-the-counter shares since 1865. The data includes not only share price data, but information on corporate actions (dividends and splits) as well as shares outstanding. These data sources enable us to calculate cap-weighted price and return indices for the United States that accurately reflect what an investment in the 100 largest stocks each year would have produced for investors.
GFD set up criteria for determining which stocks are included or excluded from its indices. GFD has followed these rules for inclusion: 1) there had to be at least 9 observations per year, 2) Dividend data have to be available for each stock in order that both price and return indices can be calculated, and 3) There had to be share outstanding information available so the stock could be included in a capitalization-weighted index.
The index includes all shares that met these criteria from 1791 to 1824, the top 50 shares by capitalization are used from 1825 to 1850 and the top 100 shares by capitalization from 1851 to 2017. To choose which companies to include, shares were weighted by capitalization during the first month of each year and included if they were among the 100 largest shares in the United States. It was assumed that the stocks were held for the rest of the year, and in January of the next year, the same selection methodology was used to choose which shares to hold for the coming year. Each year, the list of the 100 largest companies was recalculated and a new list of stocks was introduced. For continuity purposes, if a stock missed a year, i.e. the stock was in the top 100 in 1914 and 1916, but not in 1915, the stock was included in the index in 1915 even though this put over 100 stocks in the index.
A decade-by-decade comparison of the return to stocks in the GFD-100, bonds in GFD’s U.S. Bond Index, bills in GFD’s US Bill Index and the equity-risk premium is provided below.
Although the overall returns between 1870 and 2017 do not differ significantly between the GFD-100 and Cowles/S&P Composite, there are several advantages in using the GFD-100 as the benchmark for long-term historical data for the United States stock market rather than the Cowles/S&P Composite.
1. The GFD-100 uses shares that traded on all United States exchanges and over-the-counter from 1791 until 2017. The Cowles/S&P Composite is limited to the New York Stock Exchange before 1972. Finance companies that traded OTC and companies such as Standard Oil which traded OTC for several decades before moving to the NYSE are included in the GFD-100, but excluded from Cowles/S&P.
2. The GFD-100 uses shares from all sectors, including finance, from 1791 until 2019. The Cowles/S&P Composite only includes finance shares beginning in 1976 and ignores the finance sector before 1976.
3. The GFD-100 includes accurate data on dividends from 1791 to 2019. The Cowles/S&P Composite only calculated dividends from 1871 until 2017. There is no inclusion of dividends before 1871 in the Cowles/S&P Composite because no data on dividends were collected. Consequently, existing indices are missing 80 years of dividend data.
4. The GFD-100 is capitalization weighted from 1791 until 2019. The Cowles/S&P is cap-weighted from 1871 until 2019 and includes no capitalization weighting before 1871.
5. The Smith and Cole bank indices that cover the period from 1802 until 1845 suffer from survivorship bias. Banks were chosen for the indices based upon their longevity, not on their size or liquidity. The GFD-100 components are chosen based upon their market capitalization. The largest companies are chosen during each January and are “held” in the portfolio for the rest of the year when a new portfolio is organized for the coming year.
6. The Smith and Cole indices are based upon a very limited population of six to eighteen companies per year from 1802 until 1862. The GFD-100 includes 50 companies from 1825 until 1850 and 100 companies from 1851 using a broader population of shares.
7. The GFD-100 uses a consistent methodology from 1791 until 2019. The Cole and Smith/Macaulay/Cowles/S&P Index use different methodologies. The data that are used to put together the composite are collected from four different sources and chain-linked together in an uncertain pattern. During the periods of time when different indices exist, choosing different indices generates different rates of return.
Overall, the GFD-100 provides a superior benchmark stock index. Because of its greater accuracy, we would encourage financial historians to use the GFD-100 for their analysis of long-term trends in the stock market in the United States.
Decade | Stock Price | Stock Return | Dividends | Bonds | Cash | Equity Premium |
---|---|---|---|---|---|---|
1792-1799 | -3.7 | 2.67 | 6.61 | -0.15 | 5.75 | -2.91 |
1800-1809 | 1.35 | 8.01 | 6.57 | 6.03 | 4.96 | 2.91 |
1810-1819 | -4.35 | 1.59 | 6.21 | 7.42 | 4.99 | -3.24 |
1820-1829 | 0.53 | 5.82 | 5.26 | 5.01 | 3.66 | 2.08 |
1830-1839 | -0.95 | 5 | 6.01 | 0.44 | 4.51 | 0.47 |
1840-1849 | -0.8 | 6.12 | 6.98 | 6.97 | 4.94 | 1.12 |
1850-1859 | -0.58 | 6.65 | 7.27 | 4.67 | 5.01 | 1.56 |
1860-1869 | 4.62 | 12.45 | 7.48 | 9.28 | 4.97 | 7.13 |
1870-1879 | 2.04 | 8.97 | 6.79 | 5.52 | 3.82 | 4.96 |
1880-1889 | 1.19 | 6.27 | 5.02 | 4.16 | 3.01 | 7.98 |
1890-1899 | 4.05 | 9.45 | 5.19 | 4.57 | 2.13 | 2.81 |
1900-1909 | 6.25 | 11.23 | 4.69 | 0.76 | 3.01 | 7.98 |
1910-1919 | -0.69 | 5.5 | 6.23 | 2.22 | 2.62 | 2.81 |
1920-1929 | 8.72 | 14.44 | 5.26 | 5.53 | 3.5 | 10.57 |
1930-1939 | -3.48 | 0.5 | 4.12 | 6.32 | 0.27 | 0.23 |
1940-1949 | 3.43 | 8.21 | 4.62 | 2.25 | 0.58 | 7.59 |
1950-1959 | 13.67 | 18.3 | 4.07 | 0.70 | 2.16 | 15.80 |
1960-1969 | 3.82 | 6.83 | 2.90 | 1.28 | 4.4 | 2.33 |
1970-1979 | 2.14 | 6.12 | 3.90 | 4.09 | 6.71 | -0.55 |
1980-1989 | 12.94 | 18.14 | 4.60 | 15.54 | 7.96 | 9.43 |
1990-1999 | 19.78 | 22.65 | 2.40 | 7.20 | 4.52 | 17.35 |
2000-2009 | -1.66 | 0.16 | 1.85 | 5.42 | 2.25 | -2.04 |
2010-2017 | 7.67 | 10.06 | 2.22 | 3.67 | 0.21 | 9.83 |
Average | 3.30 | 8.48 | 5.06 | 4.73 | 3.74 | 4.60 |
The Wilshire 5000 is the broadest index in the United States covering every stock that is listed on the New York Stock Exchange, NASDAQ and the NYSE AMEX. When the Index was launched back in 1974, the index included about 5000 stocks, hence the name. The Wilshire 5000 was never limited to 5000 stocks, but includes every stock listed on the three exchanges regardless of the total number of companies.
Wilshire also created the Wilshire 4500, which includes all of the stocks that are listed on the three U.S. exchanges excluding the stocks in the S&P 500 as well as the Wilshire 2500, which includes the 2500 largest companies. Wilshire’s Large Cap Index includes the top 750 stocks by capitalization while the Small Cap Index includes the 1750 stocks after the first 750. The Micro-Cap Index includes all stocks included in the Wilshire 5000 that are not included in the Wilshire 2500.
Growth in the Wilshire 5000
As the stock market boomed in the 1990s, the Wilshire 5000 grew in size and reached its maximum membership on July 31, 1998 when 7,562 companies were included in the index. Thence, membership has shrunk, declining back to 5000 on December 29, 2005. Since then, the Wilshire 5000 has included less than 5000 companies. The Wilshire 5000 Full Cap Index calculates the value of all of the shares listed in the United States in billions of dollars. The index was at 28,552.5 on February 15, 2019 meaning that the value of all the common stocks listed in the United States was $28,552.5 billion. The index reached $30 trillion in September 2018 before the current sell off. The index has data going back to December 1970 when the total capitalization of the US Stock market was $830 billion, though this sank to a low of $550 billion in September 1974. GFD has used its own calculations of all the stocks listed in the United States to extend the Wilshire 5000 back to 1792. So to answer the question, how many stocks are in the Wilshire 5000? On June 30, 2018, there were 3,486 companies. This means that the number of companies that are included in the Wilshire 5000 has shrunk by 54% Since July 1998. Figure 1 illustrates the performance of the Wilshire 5000 since 1970. The index showed strong growth between 1975 and 1999, but slower growth since then.Large Caps vs. Small Caps
It is also instructive to compare the performance of the S&P 500 Total Return Index with the Wilshire 5000 Total Market Index, since both are total return indices. This comparison provides us with the best comparison of the performance of large cap and small cap stocks that is available. If the S&P 500 stocks (large caps) outperform the rest of the market, then the relative index rises, if the rest of the stocks (small caps) outperform, it falls. As Figure 2. Illustrates, the S&P 500 peaked relative to the rest of the market in 1975, declined until 1983, then began a stead rise relative to the rest of the market until 2001. Between 2001 and 2014, small caps outperformed the S&P 500, but during the past 5 years, the S&P 500 has been gaining strength relative to the rest of the market.The Wilshire/Russell 3000?
The number of stocks that are listed in the United States continues to decline. Because of the regulatory cost of listing on public exchanges, and the willingness of venture capitalists to invest in private companies, fewer and fewer companies are going public, and the number of publicly-traded companies is declining. The Russell 3000 may soon face the fate of the Wilshire 5000, being forced to include fewer companies than its name suggests. At some point in the near future, there may be fewer than 3000 companies listed in the United States and at that point, the Wilshire 5000 and the Russell 3000 will include the same number of companies. When that happens, we will let you know.Country | Begins | Ends | Companies | Max Market Cap | Year |
---|---|---|---|---|---|
Argentina | 1865 | 1985 | 69 | 749.0 | 1929 |
Australia | 1825 | 1985 | 201 | 12,550.0 | 1981 |
Austria | 1856 | 1932 | 6 | 27.0 | 1906 |
Belgium | 1845 | 1985 | 8 | 21.0 | 1875 |
Bolivia | 1825 | 1985 | 8 | 100.0 | 1929 |
Brazil | 1825 | 1984 | 60 | 2,563.0 | 1984 |
Canada | 1825 | 1985 | 100 | 19,278.0 | 1976 |
Chile | 1852 | 1969 | 46 | 815.0 | 1929 |
China | 1882 | 1930 | 7 | 38.0 | 1925 |
Colombia | 1825 | 1962 | 21 | 33.0 | 1920 |
Costa Rica | 1886 | 1932 | 1 | 1.3 | 1890 |
Cuba | 1838 | 1961 | 23 | 135.0 | 1926 |
Denmark | 1853 | 1984 | 4 | 40.0 | 1937 |
Ecuador | 1924 | 1975 | 1 | 2.3 | 1944 |
Egypt | 1856 | 1969 | 23 | 918.0 | 1929 |
El Salvador | 1887 | 1985 | 4 | 5.4 | 1978 |
Ethiopia | 1908 | 1924 | 1 | 0.2 | 1920 |
France | 1801 | 1985 | 53 | 881.0 | 1890 |
Germany | 1835 | 1985 | 17 | 7,647.0 | 1985 |
Ghana | 1869 | 1985 | 21 | 175.0 | 1936 |
Greece | 1839 | 1930 | 3 | 15.0 | 1911 |
Guatemala | 1923 | 1963 | 2 | 10.0 | 1956 |
Guyana | 1845 | 1921 | 1 | 0.1 | 1895 |
Hong Kong | 1865 | 1985 | 6 | 272.0 | 1969 |
India | 1792 | 1985 | 156 | 530.0 | 1865 |
Indonesia | 1891 | 1981 | 40 | 157.0 | 1980 |
Ireland | 1792 | 1985 | 87 | 402.0 | 1877 |
Italy | 1848 | 1985 | 19 | 78.0 | 1927 |
Jamaica | 1907 | 1985 | 6 | 3.0 | 1967 |
Japan | 1907 | 1985 | 44 | 68.0 | 1913 |
Kenya | 1907 | 1970 | 7 | 31.0 | 1966 |
Malaysia | 1889 | 1985 | 212 | 3,373.0 | 1981 |
Mauritius | 1854 | 1915 | 5 | 2.0 | 1878 |
Mexico | 1824 | 1985 | 61 | 2,112.0 | 1989 |
Mozambique | 1895 | 1975 | 2 | 3.5 | 1928 |
Myanmar | 1890 | 1977 | 6 | 95.0 | 1937 |
Netherlands | 1845 | 1985 | 25 | 19,106.0 | 1985 |
New Zealand | 1862 | 1980 | 32 | 219.0 | 1974 |
Nicaragua | 1863 | 1891 | 2 | 3.3 | 1864 |
Nigeria | 1887 | 1976 | 34 | 53.0 | 1965 |
Paraguay | 1889 | 1965 | 3 | 10.0 | 1965 |
Peru | 1825 | 1975 | 13 | 317.0 | 1968 |
Philippines | 1889 | 1985 | 10 | 954.0 | 1980 |
Portugal | 1855 | 1981 | 8 | 13.0 | 1922 |
Romania | 1870 | 1940 | 3 | 10.0 | 1914 |
Russia | 1865 | 1932 | 22 | 80.0 | 1917 |
Singapore | 1895 | 1977 | 5 | 6.0 | 1969 |
South Africa | 1833 | 1985 | 379 | 54,785.0 | 1980 |
Spain | 1845 | 1985 | 21 | 90.0 | 1913 |
Sri Lanka | 1842 | 1984 | 57 | 151.0 | 1927 |
Sweden | 1853 | 1985 | 21 | 334.0 | 1929 |
Switzerland | 1872 | 1930 | 1 | 0.5 | 1928 |
Thailand | 1919 | 1970 | 2 | 11.5 | 1964 |
Trinidad | 1865 | 1985 | 8 | 741.0 | 1980 |
Turkey | 1856 | 1930 | 11 | 38.0 | 1929 |
United States | 1821 | 1985 | 198 | 4,584.0 | 1926 |
Uruguay | 1873 | 1971 | 10 | 30.0 | 1928 |
Venezuela | 1852 | 1971 | 15 | 305.0 | 1948 |
Zimbabwe | 1893 | 1985 | 43 | 108.0 | 1985 |