Because of the breadth of data that Global Financial Data provides in its UK and US Stock Database, providing data from 1601 to 2018, GFD runs into problems of coverage that no other database encounters.  Today, stocks trade thousands or even millions of shares on a daily basis. Data on stock prices, shares outstanding and corporate actions are kept in exacting detail by the exchanges and by data providers and calculating indices poses few problems because computers can use their algorithms to spin out thousands of results on a daily basis.  Unfortunately, this was not always the case. In the 1800s, stock volume was often extremely low and many stocks did not trade over the course of an entire month.  The low volume day for the New York Stock Exchange was March 16, 1830 when only 31 shares traded on the entire exchange. Up until the 1870s, the New York Times was able to list every trade on the NYSE in each issue as did the London Times for stocks traded in London.  Today thousands of shares trade every nanosecond. To calculate price and return indices, you need three things, price data, corporate actions and shares outstanding. Global Financial Data has done everything it can to obtain as complete a record of prices, corporate actions and shares outstanding as is possible, but the further back you go in time, the more difficult it is to obtain this information.  

Sources for the Data

For the United Kingdom, the Course of the Exchange kept data on prices, shares outstanding and dividends from 1811 until the present.  Bradshaw’s Railway Manual was first published in 1848, The Investors Monthly Manual, published by The Economist, provided larger coverage of the London Stock Exchange than the Course of the Exchange and was published from 1864 until 1930, and the Stock Exchange Yearbook was first published in 1875.  In 1930, The Economist noted in The Investors Monthly Manual that it was ceasing publication because data on London stocks was readily available, especially since the Complete Stock Exchange Investment List was published on a monthly basis, but unfortunately, only a handful of libraries kept copies of the Investment List while dozens of libraries  kept copies of The Investors Monthly Manual. Nevertheless, between these books, continuous coverage of dividends and shares outstanding are available for the United Kingdom from 1811 to date. Unfortunately, because the United States hosted several regional exchanges and bank and insurance companies were listed over-the-counter, no comparable books exist for the United States in the 1800s. Poor introduced the American Railroad Journal in 1832 and the magazine began providing regular tables on the railroads in 1851.  Poor began publishing his Railroad Manuals in 1868 and continued publication until the company was bought out by Standard Statistics in 1941. William Buck Dana first published the Commercial and Financial Chronicle in 1865.  The goal of Dana was to make it an American version of The Economist. The weekly journal was preceded by Hunt’s Merchants Magazine and Commercial Review which was introduced in 1839, but stopped publication during the civil war. Our record of the U.S. stock market after the Civil War wouldn’t exist without Dana’s weekly journal.  The Investor’s Supplement, a monthly summary of the markets, was introduced in 1875 by Dana, and the Bank and Quotation Record replaced The Investor’s Supplement in 1928.  Each of these expanded the CFC’s stock coverage. The Manual of Statistics began publication in 1879, but the real growth in coverage came with the Moody’s Manual of Corporation Securities which covered thousands of industrial, transport and finance stocks beginning in 1900.  Although Poor’s Railroad Manuals and the Investor’s Supplements covered the primary industrial stocks that were listed on the New York Stock Exchange, Moody’s Manual became the first to cover the full spectrum of industrial stocks that were listed in the United States. Despite the introduction of all of these sources, there was no American equivalent to the British Course of the Exchange or the Investor’s Monthly Manual that provided complete coverage of share prices, dividends and shares outstanding in the United States during the 1800s.  Consequently, GFD has had to piece together these three important variables from an array of different sources.  The New York Stock Exchange was generally well covered by these publications and Joseph G. Martin’s A Century of Finance provided fairly complete coverage of the Boston Stock Exchange from 1798 to 1898, but other exchanges were less well covered.  

Corporate Actions Explained

The provision of cash dividends, stock dividends, stock splits and rights has changed dramatically over time and few people are aware of how GFD differentiates between these corporate actions and how they impact the total return to investors. For cash dividends, GFD has divided them into five types: Regular Cash Dividends, Extra Cash Dividends, Optional Cash Dividends, Special Cash Dividends and Liquidating Cash Dividends.  A regular cash dividend is paid, usually on a recurring basis, over the course of the year. Historically, both bonds and stocks received payments twice a year, but in the late 1800s, companies began making stock payments quarterly, which is the norm today. An Extra Cash Dividend is a payment that is made to shareholders as the result of higher profits; however, the company is under no obligation to continue the extra cash dividend in the future whereas it is expected that if the company raises its regular cash dividend, that increase will be perpetuated in the future.  Nevertheless, extra dividends tended to become regular payments and were usually repeated several years in a row at the same amount. Because these dividends achieved some degree of regularity, they are included in our calculations of the dividend yield of the company.  An optional cash dividend can be paid in either cash or shares at the discretion of the shareholder. A Special Cash Dividend was designed to be a one-time dividend that occurred because of special circumstances and was not going to be repeated.  For example, in 2004, Microsoft had too much cash.  What a problem! Microsoft did not see any good investment opportunities, so Microsoft paid a $3 dividend to its shareholders and let them decide how to invest the money. Similarly, in 1993, General Dynamics sold a division of the company and not seeing any good investment opportunities for the cash, distributed the money to shareholders in three special dividends of $20, $18 and $12.  During World War I, American corporations were encouraged to support the war effort and did so by paying dividends not in cash, but in Liberty Bonds or cash for Liberty Bonds.  These were treated as a special dividend since they were not provided on a regular basis. Of course, payment can be in kind as well. When Prohibition ended, the National Chemical Company distributed the whiskey they had put in storage in 1918 as a special dividend to shareholders. When the Dutch East India Company paid dividends to its shareholders in the 1600s, the company often provided payment in the goods they brought back to the Netherlands from Indonesia and other parts of Asia.  Payment was made in mace, paper, nutmeg, cloves and other goods. During the 1600s, there was a struggle for control of the Dutch East India Company between merchants who owned shares and wanted to be paid in kind so they could sell the spices at a profit, and the passive investors who just wanted cash.  In the long run, the passive investors won the battle of ownership and today dividends in kind are rare. A final type of cash dividend is the liquidating dividend which is paid when the company has decided to shut down its operations.  As the company sells off its assets, it distributes the proceeds to shareholders in the form of a liquidating dividend. The opposite of a cash dividend is an assessment.  Almost no companies assess shareholders anymore because assessments were very unpopular.  The goal of owning stock is to receive money from the company, not to pay it.  Assessments were popular for English railroads in the 1840s and for mining and oil stocks in the United States in the 1800s.  The idea behind the assessment was that shareholders could fund the company on a payment plan.  Building a railroad or mining for gold required a lot of capital, and the company would only ask for money when they were ready to extend invest new capital.  An English Railroad share might have a par value of £100 and the company would ask for £10 when the shares were first issued.  When those funds were used up, the railroad could ask for £10 more until the £100 was completely paid up.  The par value set a limit on how much the company could demand from its shareholders and in most cases, the company never asked for the full par value of £100.  Instead, at some point in time, the company would do a reverse split and 5 shares of £20 became 1 share of £100 and no more calls could be made on shareholders. American mining stocks enticed investors by having a low par value of $1 or $5.  The mining company would ask for ten cents or twenty cents at a time as the company dug the mine deeper and deeper into the ground.  The hope was that eventually, the company would strike gold and the investors would make a large profit, though that was the exception rather than the rule. What could go wrong with payment on the installment plan as assessment funding was known is illustrated by Overend, Gurney and Co. which issued £50 Par shares at £15 in 1865 assuring investors that they had no plans to assess them the remaining £35. However, the company went bankrupt in 1866 and the shares plunged to zero.  The company still had outstanding liabilities and the creditors demanded that the company’s shareholders, who had lost everything in the company, pay an additional £35 to cover the losses of the debtors.  This added insult to injury because not only had investors lost everything they had put in the company, but they had to pay out additional money to the company which they would never get back. Lawsuits followed, but the courts said the shareholders had to pay.  

Stock Splits and Dividends

It is also important to understand how companies differentiated between stock splits and stock dividends before World War II.  Today, companies rarely differentiate between a 5:4 split and a 25% stock dividend, but 100 years ago this was important.  Originally, in the United States, shares were issued at $100.  The company would issue 100,000 shares and the capital of the company would be equal to $10 million.  If the company paid a 100% stock dividend, this not only doubled the number of outstanding shares, but doubled the capital of the company.  In the 1800s, many railroads would pay the capital required to increase the value of the company into the company’s treasury rather than pay the money out to shareholders. On the other hand, when the company split the stock, they would replace one $100 par share with two $50 par shares or 5 $20 par shares.  The number of shares outstanding would double, but the capital in the company stayed the same.  A company could have both a stock split and stock dividend occur simultaneously, for example a 4:1 split and a 50% stock dividend which converted into 6 new shares for investors. By the 1920s, these accounting differences were seen as superfluous and companies began issuing “no par” shares. Companies can also provide stock distributions to shareholders.  A stock dividend gives shareholders additional shares in the company itself while a stock distribution gives investors shares in another company, usually one that is spun off from the parent.  For example, in 2007, Altria spun off its Kraft Foods division, providing each shareholder with 0.692 share Kraft Foods Inc. and in 2008 Philip Morris spun off Philip Morris International, the foreign division of Philip Morris to Altria shareholders. The other way that shares were distributed to investors in the 1800s and early 1900s was through a rights distribution.  If a company wanted to raise additional capital, they can either sell shares to the general public, or they can sell shares to existing shareholders.  Selling shares to the general public enables the company to raise more capital, but it reduces existing shareholders’ share of ownership in the company.  Distributing rights to existing shareholders allows them to maintain their existing share of ownership in the company, and if they don’t want to increase their ownership, they can sell their shares to someone who wants to obtain ownership in the company.  Today, ownership is so diversified that rights distributions are rarely used because few investors are concerned about reductions in their share of ownership in the company, but in the 1800s and early 1900s they were quite popular. The value of the rights depended upon how much shareholders were required to pay for a new share and how many shares they were allowed to buy.  For example, a stock might be trading at $100.  The company could issue the right to buy 1/3 share at $50. In effect, this would be the same as a 33% stock dividend. Using the rights formula, this would give the value of the right at $12.50.  On the other hand, if the price to buy a share was $80, the value of the right would be $5.  The value of the right increased as the number of shares issued increased and the amount shareholders had to pay for new shares decreased.  If a shareholder sold his right, it could be viewed as an effective cash dividend.  

Descriptions of Dividends in the GFDatabase

Although GFD has been able to obtain historical data for prices, shares outstanding and dividends, the data are not always complete.  Price data might be available from one source, dividends from a second and shares outstanding from a third. We might have share prices, but lack complete data on dividends or shares outstanding. We need to know when there are gaps in the data so GFD and subscribers can know when to include or exclude data from individual stocks for their index. GFD has put together a system to aid subscribers in determining the coverage of stocks in the 1800s when data was often unavailable or missing.  With this information, subscribers can accurately choose the stocks that will create indices for their analysis of the stock market. For price data, there can be long stretches when no data are available, even in the 1900s. For a stock traded on the New York Stock Exchange, this is rarely an issue, but GFD covers a dozen regional exchanges and over-the-counter transactions for which coverage can be spotty. There can be gaps of a few years or even a decade for smaller stocks that were listed over the counter.  To help subscribers differentiate between the liquid and illiquid stocks, we have calculated the percentage of months in which each stock has data. The situation with dividends is more complicated. Today, some stocks pay dividends and some don’t, but in the 1800s investors bought stocks to receive dividends, not to receive capital gains. Most companies tried to pay dividends whenever possible.  Bonds and preferred shares were the investments of choice for most people, and common stocks mainly attracted speculators.  But how do you know whether a stock did not pay a dividend, or whether the dividend data is simply unavailable?  We divide stocks into several categories to help subscribers solve this dilemma. Dividend Payers are stocks that paid at least one dividend during its life.  Our research shows that we were able to find all the dividends that were paid and if there are any periods when no dividend was paid, no dividends are included in the stock’s record.  To the best of our knowledge, the dividend record for these stocks is complete, though there may be periods when no dividend was paid by the company. Non-Dividend Payers are stocks that never paid a dividend between the time the stock listed and the time the stock delisted, and we have verified that this is the case.  Many mining companies and railroads in the 1800s fell in this category. Incomplete refers to stocks for which we were able to find some dividend data, but for which we know there are other periods when the company probably paid a dividend, but we have been unable to determine when these dividends occurred, or how much the dividends were for.  In those cases, we have indicated in parentheses the time period where the dividend payments are known so the subscriber can exclude the stock from their analysis during the unknown periods. If a stock traded between 1880 and 1920 and the description says Incomplete (1900-1920) you know that you can rely upon the dividend data from 1900 to 1920, but any dividends prior to 1900 are unknown.  The stock can be used for calculating a price index from 1880 to 1900, but not a return index. Unavailable means that we were unable to find any information on dividends for this stock.  The company probably paid a dividend, but we were unable to find any information on dividends that were paid.  This stock can be used for calculating a price index, but should be excluded from any total return calculations. Fixed Dividend refers to a company that was owned by another company and paid a fixed dividend to shareholders year after year. Railroads and utilities typically fell into this category. In this case, the common stock is similar to preferred shares and consequently, should be excluded from any common stock index calculations because the stock behaved more like a preferred share or a bond than a common stock. Non-Common Stocks are securities that did not allow shareholders to participate in the profits of the company. This included preferred shares, bonds, warrants, units, scrip and rights. These stocks should be excluded from any common stock indices.  

Conclusion

Global Financial Data faces issues that no other data provider must address because of the unique coverage that we provide.  Global stock markets have changed dramatically over the past four centuries, and data issues that might have seemed commonplace 200 years ago are no longer an issue.  For this reason, it is necessary that we educate our subscribers so they understand how corporate actions have changed over time and how they should treat dividends when the record of those dividends has been lost or is unavailable. Should you have any questions, we hope you will feel free to contact us.

Global Financial Data has revised its sector and industry classification system and has added comparison with the GICS and SIC classification system to help users understand the differences between the three systems.

Global Financial Data has collected extensive data on stocks from the United States and the United Kingdom covering over 400 years.  With this, GFD plans to generate indices that cover the history of the stock market from the incorporation of the Dutch East India Company in 1602 to the current market in 2018.  GFD will provide a general index, sector indices and size indices.

One question which the creation of size indices creates is how many components should be in the large cap, midcap and small cap indices.  Where should large cap, midcap and small cap begin and end? Currently, each index company treats large cap, midcap and small cap indices differently. Let’s look at how different index companies treat market capitalization.

 

Large Cap, Midcap, Small Cap

Standard and Poor’s has three size indices for the United States with 500 shares in the large cap index, 400 in the midcap and 600 in the small cap. The 500-share index was introduced in 1957 and was meant to represent all of the actively-traded stocks on the New York Stock Exchange.  Stocks on the American Stocks Exchange were mostly excluded.  As the American market grew and shares began trading actively on NASDAQ, S&P saw a need for a new index. The 400-share Midcap was introduced in 1981, and the Small Cap Index was introduced in 1994.  The proper weights for the three size indices was not calculated when the indices were introduced, so the S&P 500 Composite represents a disproportionate amount of the total market capitalization as is illustrated below.

Index	Market Cap	Percent
Dow Jones Industrials	7,105,746	23.82
S&P 500	26,517,561	88.89
S&P MidCap 400	1,935,311	6.49
S&P SmallCap 600	939,132	3.15
S&P SuperComposite 1,500	29,384,488	98.50
US Total Market Cap	29,832,836

The idea for a small cap index was introduced by Russell in 1987 and the data was extended back to 1978. Russell has 1000 stocks in their large cap index and 2000 in their small cap index. However, this creates an even greater imblance for the large cap stocks since the Russell 1000 represents about 92% of the total market cap in the United States and the Russell 2000 represents about 8%. Realizing that this allocation creates such a large imbalance Russell now calculates an index for the top 50 (39% of the market cap of the 3000), the top 200 (65% of the market cap) bottom 2500 (19% of the marke cap), and so forth.

Morningstar and MSCI have more balanced approaches to the size categories. Morningstar refers to the top 70% of stocks as large cap stocks, the next 20% as midcaps and the bottom 10% as small caps.  MSCI divides the US stock market into 300 Large Cap stocks, 450 Midcap Stocks, 1750 Small Cap Stocks and the remaining stocks (around 1000) as Micro-cap stocks.  By our calculations, this would give about 70% to the Large Cap 300, 16% to the Midcap 450, 13% to the Small Cap 1750 and 1% to the Micro-Cap 1000.

By contrast, the FTSE UK indices have 100 stocks in the Large cap Index, 250 stocks in the Midcap and the next 370 stocks in the Small Cap.  By our calculations this gives about 79% of total stock market cap to the FTSE-100, 16% to the FTSE-250 and 5% to the remaining stocks.
Of all of these allocations, the Morningstar approach of 70% Large Cap, 20% Midcap and 10% Small Cap seems the most logical, but how do you put it into practice?

 

Taylor’s Golden Rule

When the S&P 500 was introduced in 1957, the 500 stocks represented over 90% of the stocks listed on the New York Stock Exchange, but by 2000 there were over 5000 stocks listed on the NYSE, AMEX and NASDAQ.  In 2018 the number of stocks listed on the NYSE and NASDAQ had shrunk to around 3500.  At the present rate of decline, in a few years there may not even be 3000 stocks to make up the Russell 3000.

The problem with creating long-term indices is that the number of stocks that listed on the exchanges and over-the-counter grew dramatically over time and the number of stocks in the large cap, midcap and small cap groups vary accordingly.  During most of the 1800s, there weren’t even 500 stocks listed on all of the exchanges in the United States.  So how do you determine how to allocate stocks to the large cap, midcap and small cap categories if the number of stocks in existence is constantly changing?

The best solution is to determine a ratio of the number of stocks in each category.  Our research has shown that a golden rule possibly exists for the different groups by size.  Our rule is the 10-20-30-40 rule.  The top 10% of the stocks by number are Blue Chips, the next 20% are large caps, the next 30% are midcaps and the bottom 40% are small caps.  A stock index that included 1000 stocks would include 100 Blue Chips, 200 Large Caps, 300 Midcaps and 400 Small Cap Stocks.  The Blue Chips and Large Caps can be combined to create a single Large Cap group.

 

Allocations of the Sectors

To test our theory we calculated the market caps that would result from this combination for historical indices for the United States and the United Kingdom.  For the United States, we chose 1000 stocks from 1894 to 2016, 500 stocks from 1875 to 1893 and 200 stocks from 1830 to 1874.  For the United Kingdom we chose 250 stocks from 1864 to 1985.

After we summed the market cap for each group, we divided the market cap of each group by the market cap of all the stocks in the population to determine what the ratios of the different groups were.  The results of our analysis are provided below.  What is interesting is how closely the percentages for the Large cap, Midcap and Small cap compare over the 300 years of data that is used from the United States and the United Kingdom.

United States			United Kingdom
	Average	St.Dev		Average	St.Dev
Blue Chips	55.27	13.15		60.39	10.49
Large Cap	22.67	5.29		18.85	5.01
BC + LC	77.94	8.25		79.24	5.64
Mid Cap	13.73	4.72		12.48	3.34
Small Cap	8.35	3.57		8.28	2.35
PC of Total	93.85	3.47		94.33	13.44

The result is five size indices which can provide useful information to investors.  The Blue Chip stocks provide about 60% of the total market cap, the Large Caps 20%, the Midcaps 12% and the Small Caps 8%.  As the number of stocks changes over time, or the number of stocks varies from one exchange to the other, these ratios can be used to create useful indices that provide contrasting behavior between the different size indices.

GFD’s size indices will be based upon Taylor’s Golden Rule, and we recommend that other index creators begin using this rule as well.

If you had to choose between investing in one of the two companies, Dow Chemical or du Pont 100 years ago on September 1, 1917, which company would have provided the better investment?  We crunched the numbers and found that the winner was Dow Chemical, though not by much. If you had invested $1 in du Pont 100 years ago you would have ended up with $870 on September 1, 2017, and if you included reinvested dividends, you would have ended up with $63,309, giving you a 7% annual return through capital appreciation, an annual 4.38% dividend yield and a total annual return of 11.69%. On the other hand, if you had invested $1 in Dow Chemical 100 years ago, you would have ended up with $3410 in 2017 and $149,755 if you include dividends.  This comes out to an annual capital appreciation of 8.47%, an annual dividend yield of 3.85% and a total annual return of 12.66%. How does this compare with the S&P Composite?  If you had invested $1 in the S&P 500 on September 1, 1917, you would have ended up with $290, and if you include dividends, $15,143 for an annual capital appreciation of 5.83% and 10.1% after including dividends.  Obviously, both stocks outperformed the S&P Composite over the past 100 years. The chart below compares the performance of each company over the past 100 years.