GFD's 100-share Index
Existing calculations of long-term stock market returns in the United States are based upon four primary sources: Smith and Cole (1803-1862), Macaulay (1857-1871) Cowles (1871-1928) and Standard and Poor’s (1928-2017). Unfortunately, these indices have major flaws in them that create biases that researchers have tolerated until now because no one has ever collected historical data on U.S. share prices, corporate actions (dividends and splits) and shares outstanding so accurate price and return indices could be calculated. The current S&P Composite includes data from the Cowles Indices from 1871 until 1927, the 90-share daily S&P Index from January 1928 until February 1957, the 500-share composite that excludes finance stocks until July 1976 and the all-sector 500-share index since July 1, 1976. Global Financial Data has collected data on all major United States exchanges going back to 1791 as well as data on over-the-counter shares since 1865. The data includes not only share price data, but information on corporate actions (dividends and splits) as well as shares outstanding. These data sources enable us to calculate cap-weighted price and return indices for the United States that accurately reflect what an investment in the 100 largest stocks each year would have produced for investors. GFD set up criteria for determining which stocks are included or excluded from its indices. GFD has followed these rules for inclusion: 1) There have to be at least 9 observations per year for each stock and there have to be at least two price changes during each year to warrant inclusion. Otherwise, the stock was excluded due to illiquidity. 2) Dividend data have to be available for each stock in order that both price and return indices can be calculated. A stock may not have paid a dividend during a particular year, but to be included, we had to know the company had passed on paying a dividend during that year. Any shares for which we were unable to find dividend information was excluded. 3) There had to be share outstanding information available so the stock could be included in a capitalization-weighted index 4) Only non-assessable shares were included. A stock was excluded if it imposed assessments on shareholders. Shares that were liquidating assets and were paying liquidating dividends were excluded as well. The index includes all shares that met these criteria from 1791 to 1824, the top 50 shares by capitalization are used from 1825 to1850 and the top 100 shares by capitalization from 1851 to 2017. Although GFD plans to calculate indices that include more than 100 shares in the future, we decided that because the index was capitalization-weighted, a 100-share index would track a 500-share index very closely. The graph below compares the performance of the S&P 100 (green line) and the S&P 500 (black line) between 1976 and 2018. The performance of the two is very similar. The S&P 500 did outperform the S&P 100 by 0.5% per annum, but the returns are close enough to show that a strong correlation exists between the top 100 shares and the top 500 shares. In 2017, the top 100 shares in the S&P 500 represented about 60% of the total capitalization of the S&P 500.
To choose which companies to include, shares were weighted by capitalization during the first month of each year and included if they were among the 100 largest shares in the United States. It was assumed that the stocks were held for the rest of the year, and in January of the next year, the same selection methodology was used to choose which shares to hold for the coming year. Each year, the list of the 100 largest companies was recalculated and a new list of stocks was introduced. For continuity purposes, if a stock missed a year, i.e. the stock was in the top 100 in 1914 and 1916, but not in 1915, the stock was included in the index in 1915 even though this put over 100 stocks in the index. Initially, there were less than 100 shares in the index because of a lack of companies that met the criteria outlined above. There were only 3 companies in 1791, 7 in 1800, 18 in 1810, 25 in 1820 and 50 in 1825. The index included 50 members through 1850 and beginning in 1851, there were 100 shares in the index. The chart below shows the sectoral composition of the GFD-100 index from 1800 until 2017 to cover the period before the introduction of the Cowles indices in 1871. However, the Smith and Cole indices have some major biases in them. The indices use a very limited number of companies, excluding, for example, the first and second Bank of the United States, which represented from 50% to 80% of the overall capitalization of the United States stock market. Consequently, Smith and Cole exclude over half of the U.S. stock market in their indices.
As can be seen, finance stocks (banks and insurance companies) represented over 95% of the capitalization until 1825. After 1825, the transportation sector grew, first with canals and then with railroads. By the 1840s, transports represented over half of the total capitalization, growing to 80% by the 1870s, then shrinking as industrials began to dominate the U.S. stock market. By 1900, industrial stocks represented about 40% of total capitalization with Energy stocks, primarily Standard Oil and its subsidiaries, representing a growing portion of total capitalization. It should be noted that Standard Oil of New Jersey and California (ExxonMobil and Chevron) weren’t included in the Cowles indices until 1918 when Standard Oil listed on the NYSE. By 1918, Standard Oil had grown tenfold in size to become the largest company in the world, yet was not included in any indices because it traded over-the-counter. And as we stated previously, no finance stocks were included in either the Cowles indices or the S&P indices until 1976. Therefore, we feel that the composition of stocks in the GFD indices are more representative of the stocks that were available to investors in the United States than any of the existing indices of United States stocks. To show how the GFD-100 index differs from existing indices of U.S. stocks we will compare the components and performance of the GFD-100 index and other indices.
Survivorship Bias in the Smith and Cole Indices
Currently, no index for U.S. stocks exists prior to 1802 when the Smith and Cole indices begin. Therefore, the GFD-100 index covers data that is currently unavailable from existing indices. The graph below compares the Cole and Smith indices (USCFALLM) with the GFD-100 index (GFUS100MPM) from 1802 until 1845 when Smith and Cole’s Bank and Insurance index was discontinued. As can be seen, the Smith and Cole index outperforms the GFD-100 index; however, in analyzing how the Smith and Cole indices were put together, we discovered that the superior performance was caused by the survivorship bias that Smith and Cole built into their indices.
In Walter B. Smith and Arthur H. Cole, Fluctuations in American Business, 1790-1860, Cambridge: Harvard Univ. Press, 1935, they provided calculations of stock market indices for Banks (1802-1820), Banks and Insurance (1815-1845) and Railroad (1834-1862) shares which have been used to cover the period before the introduction of the Cowles indices in 1871. However, the Smith and Cole indices have some major biases in them. The indices use a very limited number of companies, excluding, for example, the first and second Bank of the United States, which represented from 50% to 80% of the overall capitalization of the United States stock market. Consequently, Smith and Cole exclude over half of the U.S. stock market in their indices. For the dataset of bank stocks from 1802 until 1820, Smith and Cole collected data on 21 bank stocks, 9 insurance shares, 5 bridge and turnpike shares and 3 miscellaneous companies. Smith and Cole based their index on 7 bank stocks which existed during the entire 19-year period from 1802 until 1820, creating a survivorship bias. Banks such as the first Bank of the United States were excluded because they did not survive until 1820. Since Smith and Cole collected no data on shares outstanding, the index was calculated as a geometric mean of the underlying shares. The index is not cap-weighted even though it included only 7 banks. Because Smith and Cole collected no data on dividends, no return index was calculated. Scholars who have attempted to calculate total returns have calculated a dividend yield based upon the yield on government stocks. Jeremy Siegel in his book Stocks for the Long Run, uses a 6.4% dividend between 1802 and 1870. Smith and Cole collected data on 17 bank stocks, 5 canal stocks, 3 gas-light stocks, 19 insurance stocks, 27 rail, 6 miscellaneous and 17 Boston bank shares between 1815 and 1845. From these shares, they chose 6 bank shares and 1 insurance company for the Bank and Insurance index. Again, only stocks that survived the 30-year stretch between 1815 and 1845 were included, once again creating a survivorship bias. Finally, Smith and Cole calculated a railroad index that went from 1834 to 1862. Smith and Cole removed shares that either changed little or were erratic in their behavior. They broke up the data into 8 shares from1834 to 1853 and in several cases, because the data series were incomplete, they “spliced” together some of the series even though they admitted this was a statistical device of “questionable value.” Smith and Cole calculated two rail indices of 8 and 10 shares from 1834 to 1853 and a composite of 18 shares from 1853 to 1862. Finally, they calculated a chain-linked index of railway shares that covered 1834 to 1862 as well as some regional rail indices. A railway index of 25 shares calculated by Frederick R. Macaulay in The Movements of Interest Rates, Bond Yields and Stock Prices in the United States Since 1856, New York: National Bureau of Economic Research, 1938 was used to fill in the gap between 1862 and 1870. The performance of the rail indices is provided below. Although the GFD-100 and Smith and Cole Rail index take divergent paths, they both end up at the same levels over the 28-year period. The GFD Transports outperform the Smith and Cole Railway Index, especially in the 1850s, but the general pattern of their performance at any point in time is similar. The difference lies in the rate of return.
It is unfortunate that the Smith and Cole indices have been the sole source of returns for the United States before 1870. The problem with the Smith and Cole indices are several: 1) they use a limited number of stocks, 2) their selection process creates a survivorship bias that increases returns, 3) calculations are based upon the geometric mean of the shares rather than the capitalization of each share, 4) railway shares are spliced together to provide consistent coverage, and 5) no data on dividends was collected so no total returns could be calculated. As was discussed above, because of the survivorship bias that is inherent in the Cole and Smith indices, one would expect that their indices would outperform the other indices that lack a survivorship bias. The survivorship bias is particularly notable in the exclusion of the second Bank of the United States. After the bank lost its charter from the Federal government in 1836, Stephen Girard took over the bank and privatized it. The bank collapsed during the depression that followed the Panic of 1837, and Bank of the United States stock declined in value from 123 in January 1839 to 3.5 by December 1841. The bank’s capitalization fell from $42 million to $1 million. The GFD-100 index lost almost half of its value as a result. The chart below compares the Smith and Cole/Macaulay index with the GFD-100 index from 1802 until 1871. As can be seen, the survivorship bias enables the Smith and Cole/Macaulay index to outperform the GFD-100 during the entire period. From 1845 on, the Smith and Cole/Macaulay index includes no finance stocks and outperforms the GFD-100 even more. The combination of survivorship bias and exclusion of bank and insurance shares after 1845 causes the GFD-100 to underperform the Smith and Cole/Macaulay index. Because the GFD-100 includes more stocks and is cap-weighted, it provides a better representation of the return shareholders would have received before the introduction of the Cowles indices in 1871.
Limitations of the Cowles Indices
In the 1930s, Cowles created cap-weighted indices for the United States going back to 1871. The Commercial and Financial Chronicle published annual summaries of monthly high and low prices for stocks. The Cowles Commission collected data on shares outstanding and dividends in order to calculate cap-weighted price and total return indices for the United States. The Cowles indices included 44 shares in 1871, 75 in 1880, 88 in 1890, 102 in 1900, 126 in 1910, 243 in 1920 and 450 in 1930. The GFD-100 index included about 100 shares in each of those years. Although the Cowles indices are cap-weighted and include dividends, there are drawbacks to the Cowles indices. In calculating the indices, prices were based upon the average of the high and low prices for each month, not on the closing price of shares in each month. Although this factor affects the volatility of the indices, it should have a minimal impact on the long-term returns of the index. More importantly, the selection of stocks was limited to shares traded on the New York Stock Exchange. This means that bank and insurance shares as well as Standard Oil and other non-NYSE companies are completely excluded from the Cowles indices. Even though Standard Oil began trading in 1882, neither Standard Oil of New Jersey (ExxonMobil) nor of California (Chevron) was included in the Cowles indices until 1918 when they joined the NYSE. At the time, Standard Oil was the largest corporation in the world. The impact of excluding Standard Oil, banks and insurance companies is illustrated below.
Standard Oil (XOM – blue) clearly outperformed the other indices with finance stocks (GFUSBNKINSMPM – green) coming in second. Transports (GFUSTRANSMPM – black) did outperform the Cowles Indices (_SPXD – purple), but the Cowles indices provided the lowest return of the indices in the graph. The graph below looks at the sectoral composition of the Cowles/S&P Broad Composite index from 1871 to 1957 when it was discontinued in favor of the S&P 500. This sector chart can be compared with the sector chart of the GFD-100 provided above. As can be seen, the Cowles/S&P index is missing the finance sector and gives a lower weighting to the Energy sector than the GFD-100 sector chart. This absence is made up by the larger role of transportation (railroad) shares in the index until the 1920s.
The GFD-100 and the S&P 90/500
The S&P 90-share index was introduced on January 1, 1928. It is used as the basis for the S&P Composite through March 1, 1957 when the S&P 500 replaced the S&P 90. Both the GFD-100 and the S&P 90 use a similar number of shares between 1928 and 1957 and as the graph below shows, the two indices track each other very well. The S&P 500 was introduced on March 1, 1957 and excluded finance stocks until July 1, 1976 when 40 finance shares were included in the S&P 500. On April 6, 1988 exact sectoral balances were dropped and the number of industrial, utility, transportation and finance stocks was based purely upon market cap, not on fixed ratios for different sectors. The chart below compares the performance of the GFD-100 and the S&P 90/500 index between 1918 when Standard Oil and Chevron listed on the NYSE and 1976 when finance stocks were added to the index. The correlation between the two indices is strong with the GFD-100 providing a 5.1% annual return between 1918 and 1976 and the S&P Composite providing a 4.7% return during the same period. Although we found reasons for the difference in the performance of the GFD-100 and the Smith and Cole/Cowles indices between 1802 and 1918, the similarity in composition between the GFD-100 and the S&P Indices has generated quite similar returns between 1918 and 1976.
Total Returns
Until now, we have only looked at the return to the price indices for the GFD-100 and the S&P Composite. Of course, shareholders receive dividends in addition to capital gains and historically, shareholders have received a higher return from dividends than from capital gains. Until now, no one has ever measured the dividends shareholders received before 1871. GFD’s calculations show that before 1871, U.S. shareholders on average received no capital gains. This means that 100% of shareholder return came from dividends. GFD has collected historical data on the dividends that were paid by thousands of companies in order that we could accurately calculate the dividends that shareholders received. A comparison of the yields on the S&P Cowles Composite and the GFD-100 is provided below. You can generally observe that the dividend yield declined between 1791 and 1820, rose from 1820 until 1870, then declined for the rest of the 1800s. The dividend yield shows similar behavior in the GFD-100 and the S&P Composite between 1871 and 2017. The dividends for the S&P Composite appear to be more volatile than the yields on the GFD-100 between 1871 and 1980. The reason for this lies in the different methodology used to calculate the dividend yield. S&P/Cowles collected data on annual dividends through 1935 and quarterly thereafter. The S&P dividend yield was calculated by dividing the annual dividends that were “paid” to shareholders by the price of the S&P Composite. Consequently, a large decline or rise in the S&P Composite index caused a spike in the dividend yield. Data for the GFD-100 dividend yield is calculated on an ongoing basis from month to month when the dividends are “paid” to the component companies on the ex-date, reducing the volatility in the dividend yield. Nevertheless, the dividend yield of the GFD-100 and the S&P Cowles Composite follow each other fairly closely. With this dividend data, we can calculate the total return to U.S. stocks over the entire period covered. No estimates based upon the yield on government bonds is needed. The total return from the GFD-100 and the S&P Composite are illustrated below. As can be seen, over the long-term, the differences between the two indices is small. The Smith/Cole/Cowles data outperform the GFD-100 in the early 1800s because of the survivorship bias inherent in the Smith and Cole data. However, the exclusion of over-the-counter shares (Standard Oil and finance stocks) by Cowles between 1871 and 1918 allows the GFD-100 to “catch up” with the Composite data. The two indices then track each other closely from 1918 until 1976 when S&P introduced bank and insurance shares into the S&P 500 Composite.
We can provide actual data on the total returns to the two indices in order that the two sets of indices can be compared directly. The table below shows the returns of the GFD-100 and the S&P Composite between 1802 and 2017. Annual capital gains for the GFD data and the S&P data are provided in the first two columns. Total Returns including reinvested dividends are provided in the next two columns and the dividends that were paid are compared in the last two columns. The returns are very similar over the 215 years for which both indices have data with capital gains of 3.57% per annum for the GFD-100 and 3.46% for the Cowles/S&P Composite. After dividends are added and total returns are calculated, similar numbers result. The GFD-100 provided a total return of 8.75% per annum and the Smith/Cole/Cowles/S&P Composite provided an 8.38% total return between 1802 and 2017. The dividends we interpolated between 1802 to 1870 was calculated as the yield on government bonds plus 1% yielding an annual dividend of 5.76%. If the dividend yield of 6.42% that we actually calculated between 1802 and 1870 were used, the total return to the S&P Composite would equal 8.67% between 1802 and 2017 which is close to the GFD-100 annual return of 8.75%. The period from 1802 until 2017 is also divided into a pre-Cowles period from 1802 until 1870 when Smith and Cole/Macaulay were the only source for indices, the period from 1871 to 1918 when Standard Oil was excluded from the index, and 1918 until 1976 when S&P calculated the index, but excluded finance shares.
Table 1 Comparison of Returns to GFD-100 and Cowles/S&P Composite
| SPX | GFD | SPX | ||||
|---|---|---|---|---|---|---|
| Period | GFD Nominal | Nominal | GFD TR | SPX TR | Dividends | Dividends |
| 1802-1870 | -0.03 | 1.33 | 6.39 | 7.17 | 6.42 | 5.76 |
| 1871-1918 | 2.15 | 0.93 | 7.75 | 5.88 | 5.48 | 4.9 |
| 1918-1976 | 5.2 | 4.82 | 9.63 | 9.9 | 4.21 | 4.84 |
| 1802-1976 | 2.19 | 2.36 | 7.79 | 7.67 | 5.48 | 5.19 |
| 1802-2017 | 3.57 | 3.46 | 8.75 | 8.38 | 5 | 4.76 |
| 1791-2017 | 3.32 | 8.54 | 5.05 |
A decade-by-decade comparison of the return to stocks in the GFD-100, bonds in GFD’s U.S. Bond Index, bills in GFD’s US Bill Index and the equity-risk premium is provided below.
Decade-by-Decade Returns to Stocks, Bonds and Bills in the United States
| Decade | Stock Price | Stock Return | Dividends | Bonds | Cash | Equity Premium |
|---|---|---|---|---|---|---|
| 1792-1799 | -3.7 | 2.67 | 6.61 | -0.15 | 5.75 | -2.91 |
| 1800-1809 | 1.35 | 8.01 | 6.57 | 6.03 | 4.96 | 2.91 |
| 1810-1819 | -4.35 | 1.59 | 6.21 | 7.42 | 4.99 | -3.24 |
| 1820-1829 | 0.53 | 5.82 | 5.26 | 5.01 | 3.66 | 2.08 |
| 1830-1839 | -0.95 | 5 | 6.01 | 0.44 | 4.51 | 0.47 |
| 1840-1849 | -0.8 | 6.12 | 6.98 | 6.97 | 4.94 | 1.12 |
| 1850-1859 | -0.58 | 6.65 | 7.27 | 4.67 | 5.01 | 1.56 |
| 1860-1869 | 4.62 | 12.45 | 7.48 | 9.28 | 4.97 | 7.13 |
| 1870-1879 | 2.04 | 8.97 | 6.79 | 5.52 | 3.82 | 4.96 |
| 1880-1889 | 1.19 | 6.27 | 5.02 | 4.16 | 3.01 | 3.16 |
| 1890-1899 | 4.05 | 9.45 | 5.19 | 4.57 | 2.13 | 7.17 |
| 1900-1909 | 6.25 | 11.23 | 4.69 | 0.76 | 3.01 | 7.98 |
| 1910-1919 | -0.69 | 5.5 | 6.23 | 2.22 | 2.62 | 2.81 |
| 1920-1929 | 8.72 | 14.44 | 5.26 | 5.53 | 3.5 | 10.57 |
| 1930-1939 | -3.48 | 0.5 | 4.12 | 6.32 | 0.27 | 0.23 |
| 1940-1949 | 3.43 | 8.21 | 4.62 | 2.25 | 0.58 | 7.59 |
| 1950-1959 | 13.67 | 18.3 | 4.07 | 0.70 | 2.16 | 15.80 |
| 1960-1969 | 3.82 | 6.83 | 2.90 | 1.28 | 4.4 | 2.33 |
| 1970-1979 | 2.14 | 6.12 | 3.90 | 4.09 | 6.71 | -0.55 |
| 1980-1989 | 12.94 | 18.14 | 4.60 | 15.54 | 7.96 | 9.43 |
| 1990-1999 | 19.78 | 22.65 | 2.40 | 7.20 | 4.52 | 17.35 |
| 2000-2009 | -1.66 | 0.16 | 1.85 | 5.42 | 2.25 | -2.04 |
| 2010-2017 | 7.67 | 10.06 | 2.22 | 3.67 | 0.21 | 9.83 |
| Average | 3.30 | 8.48 | 5.06 | 4.73 | 3.74 | 4.60 |
Conclusion
Although the overall returns between 1802 and 2017 do not differ significantly between the GFD-100 and Cowles/S&P Composite, there are several advantages in using the GFD-100 as the benchmark for long-term historical data for the United States stock market rather than the Cowles/S&P Composite. 1. The GFD-100 uses shares that traded on all United States exchanges and over-the-counter from 1791 until 2017. The Cowles/S&P Composite is limited to the New York Stock Exchange before 1972. Finance companies that traded OTC and companies such as Standard Oil which traded OTC for several decades before moving to the NYSE are included in the GFD-100, but excluded from Cowles/S&P. 2. The GFD-100 uses shares from all sectors, including finance, from 1791 until 2017. The Cowles/S&P Composite only includes finance shares beginning in 1976 and ignores the finance sector before 1976. 3. The GFD-100 includes accurate data on dividends from 1791 to 2017. The Cowles/S&P Composite only calculated dividends from 1871 until 2017. There is no inclusion of dividends before 1871 in the Cowles/S&P Composite because no data on dividends were collected. Consequently, existing indices are missing 80 years of dividend data. 4. The GFD-100 is capitalization weighted from 1791 until 2017. The Cowles/S&P is cap-weighted from 1871 until 2017 and includes no capitalization weighting before 1871. 5. The Smith and Cole bank indices that cover the period from 1802 until 1845 suffer from survivorship bias. Banks were chosen for the indices based upon their longevity, not on their size or liquidity. The GFD-100 components are chosen based upon their market capitalization. The largest companies are chosen during each January and are “held” in the portfolio for the rest of the year when a new portfolio is organized for the coming year. 6. The Smith and Cole indices are based upon a very limited population of six to eighteen companies per year from 1802 until 1862. The GFD-100 includes 50 companies from 1825 until 1850 and 100 companies from 1851 using a broader population of shares. 7. The GFD-100 uses a consistent methodology from 1791 until 2017. The Cole and Smith/Macaulay/Cowles/S&P Index uses different methodologies. The data that are used to put together the composite are collected from four different sources and chain-linked together in an uncertain pattern. During the periods of time when different indices exist, choosing different indices generates different rates of return. Overall, the GFD-100 provides a superior benchmark stock index. Because of its greater accuracy, we would encourage financial historians to use the GFD-100 for their analysis of long-term trends in the stock market in the United States.
