HANOVER CAPITAL MORTGAGE HOLDINGS, INC.

200 Metroplex Drive, Suite 100

Edison, New Jersey 08817

Dear Fellow Shareholder:

      You are cordially invited to attend the 2006 Annual Meeting of Shareholders of Hanover Capital Mortgage Holdings, Inc. to be held at 11:00 a.m., local time, on Friday, May 19, 2006, at the American Stock Exchange, 86 Trinity Place, New York, New York, 10006.

      The attached notice of annual meeting and proxy statement describe the formal agenda for the Annual Meeting. Your Board of Directors and management will also present a report on our operations and will be happy to respond to questions properly brought before the meeting.

      I hope that you will be able to attend the Annual Meeting in person. Even if you plan to attend, I urge you to sign, date and return the enclosed proxy card as soon as possible. This will assist us in achieving a quorum at the Annual Meeting and will ensure that your vote will be counted even if you are unable to attend the Annual Meeting. You will still be able to vote in person at the Annual Meeting if you return the enclosed proxy card. Your vote is important regardless of how many shares you own.

      On behalf of our Board of Directors, thank you for your continued interest and support. We look forward to meeting and speaking with those of our shareholders who are able to attend the Annual Meeting.

IMPORTANT

YOUR VOTE IS IMPORTANT REGARDLESS OF THE NUMBER OF SHARES YOU OWN. PLEASE COMPLETE, SIGN, DATE AND MAIL THE ENCLOSED PROXY CARD AT YOUR EARLIEST CONVENIENCE. NO POSTAGE IS REQUIRED IF MAILED IN THE UNITED STATES. IF YOU HAVE ANY QUESTIONS OR NEED ASSISTANCE, PLEASE CALL US AT (732) 548-0101.

HANOVER CAPITAL MORTGAGE HOLDINGS, INC.

200 Metroplex Drive, Suite 100

Edison, New Jersey 08817

NOTICE OF 2006 ANNUAL MEETING OF SHAREHOLDERS

To be held Friday, May 19, 2006

To Our Shareholders:

      The 2006 Annual Meeting of Shareholders of Hanover Capital Mortgage Holdings, Inc., a Maryland corporation, will be held on Friday, May 19, 2006, at 11:00 a.m., local time, at the American Stock Exchange, 86 Trinity Place, New York, New York, 10006, for the following purposes:

      The Board of Directors has fixed the close of business on March 31, 2006 as the record date for the meeting. All shareholders of record on that date are entitled to notice of and to vote at the meeting and any adjournments or postponements thereof.

      Your proxy is enclosed. Your vote is important regardless of the number of shares you own. To secure the largest possible representation and avoid the additional expense to Hanover of further solicitation, mark your preferences, sign, date and return the enclosed proxy card in the enclosed postage-paid envelope at your earliest convenience, even if you plan to attend the Annual Meeting in person. The giving of this proxy will not affect your right to vote in person in the event you do attend the meeting.

Edison, New Jersey

This Proxy Statement is first being sent to shareholders on or about April 21, 2006.

TABLE OF CONTENTS

HANOVER CAPITAL MORTGAGE HOLDINGS, INC.

PROXY STATEMENT

      We are sending you this proxy statement in connection with the solicitation by our Board of Directors of proxies to be voted at our 2006 Annual Meeting of Shareholders and at any adjournments or postponements thereof. The Annual Meeting will be held on Friday, May 19, 2006, at the time and place set forth in the notice of the meeting. This proxy statement, accompanying proxy card and our Annual Report on Form 10-K for the year ended December 31, 2005, as filed with the Securities and Exchange Commission (SEC), form your meeting package. We are sending this package on or about April 21, 2006 to record holders of our common stock as of March 31, 2006, the record date for the Annual Meeting. The terms “we”, “us”, “our” and “Hanover” refer to Hanover Capital Mortgage Holdings, Inc.

      If you properly execute, date and return the enclosed proxy card it will be voted in the manner you direct. If you do not specify instructions with respect to any particular matter to be acted upon, then the shares represented by your proxy will be voted:

      The persons named as proxies may also vote on any other matter to properly come before the Annual Meeting.

      Any person giving the enclosed form of proxy has the power to revoke it by voting in person at the Annual Meeting, by giving written notice of revocation to our Secretary at any time before the proxy is exercised or by duly executing and dating a subsequent proxy relating to the same shares of our common stock and delivering it to our Secretary at or before the Annual Meeting.

      The presence at the meeting in person or by proxy of shareholders entitled to cast a majority of all votes entitled to be cast at the meeting will constitute a quorum for the transaction of business. The election of the nominees for Director will be decided by plurality vote. To approve Proposal #2, at least a majority of the votes cast at the meeting must be voted in favor of the proposal. Abstentions and broker non-votes are not considered votes cast, and will not affect the vote. If your shares of our common stock are held by a broker and you do not submit your proxy card, your brokerage firm may choose to vote for you or leave your shares unvoted.

      We will bear the cost of the solicitation of proxies. We expect that the solicitation will be made primarily by mail, but our regular employees, directors or representatives (none of whom will receive any extra compensation for their activities) may also solicit proxies by telephone, in person and by other means, and may arrange for brokerage houses and other custodians, nominees and fiduciaries to send proxies and proxy materials to their principals at our expense.

      Our principal executive offices are located at 200 Metroplex Drive, Suite 100, Edison, New Jersey 08817, telephone number (732) 548-0101.

RECORD DATE AND VOTING SECURITIES

      Only shareholders of record at the close of business on March 31, 2006, the record date for the Annual Meeting, are entitled to notice of and to vote at the meeting. On that date, we had outstanding and entitled to vote 8,492,237 shares of common stock, par value $.01 per share. Our shares of common stock have no cumulative voting rights. Each outstanding share of our common stock entitles the record holder to cast one vote for each director to be elected and one vote on Proposal #2.

PROPOSAL #1 — ELECTION OF DIRECTORS

      Our Board of Directors is divided into three classes, with each class as nearly equal in number as possible. One class is elected each year for a term of three years. Any Director who was appointed by the Board of Directors to fill a vacancy holds office until the next annual meeting of shareholders, at which time the shareholders elect a Director to hold office for the balance of the term then remaining. It is proposed that the Director nominees listed below, whose terms expire at this meeting, be elected to serve for a term of three years and until their successors are duly elected and qualified. None of the non-executive Directors are related to any of our executive officers.

      The terms of three directors expire at the annual meeting. The Board has nominated two of these directors, John A. Burchett and John A. Clymer, to be re-elected at the Annual Meeting as Directors for terms that will expire in 2009. The third incumbent director with a term expiring at the annual meeting, Saiyid T. Naqvi, has indicated that his other current responsibilities are such that he will not be able to continue with the commitment required to be a director of Hanover. Therefore, Mr. Naqvi has decided not to stand for re-election.

      All of the Director nominees advised us that they are available and willing to serve if elected. If any of the Director nominees becomes unavailable for election, which we do not anticipate, then the persons named in the accompanying proxy will vote for such substitutes as the Board of Directors may recommend.

      The election of nominees for Director will be decided by plurality vote. Your Board of Directors recommends that you vote “FOR” Mr. Burchett and Mr. Clymer. Proxies solicited by the Board of Directors will be voted “FOR” the election of the Director nominees named below, all of whom are now members of the Board of Directors, unless shareholders specify a contrary vote.

Nominees for Terms Expiring 2009:

      The following are the continuing members of our Board of Directors, whose terms of service are indicated below:

Directors Whose Terms Expire in 2008:

Directors Whose Terms Expire in 2007:

Our Executive Officers

      We have listed below our executive officers that are not included in the tables above.

PROPOSAL #2 — RATIFICATION OF APPOINTMENT OF INDEPENDENT ACCOUNTANTS

      The Audit Committee selected and appointed Grant Thornton LLP to act as our independent accountants for the year ending December 31, 2006. In recognition of the important role of the independent accountants, their selection is being submitted to the shareholders for review and ratification.

      An affirmative vote of the holders of at least a majority of the shares of common stock voting on this proposal is required for its adoption. Your Board of Directors recommends ratification of Grant Thornton LLP as our independent accountants for 2006. Proxies solicited by the Board of Directors will be voted “FOR” this proposal unless shareholders specify a contrary vote. In the event the shareholders do not ratify the appointment, the appointment will be reconsidered by our Audit Committee.

      A representative of Grant Thornton LLP is expected to be present at the Annual Meeting and to be available to respond to appropriate questions. The representative will have an opportunity to make a statement if he or she so desires.

SECURITY OWNERSHIP OF CERTAIN BENEFICIAL OWNERS AND MANAGEMENT

      The following table sets forth information regarding the ownership of our common stock as of March 31, 2006 by:

Compliance with Section 16(a) of the Exchange Act

      Section 16(a) of the Exchange Act requires our executive officers and directors, and persons who own more than ten percent of a registered class of our equity securities, to file reports of ownership and changes in ownership on Forms 3, 4 and 5 with the Securities and Exchange Commission (the “SEC”). Officers, directors and greater than ten percent stockholders are required by SEC regulation to furnish the Company with copies of all Forms 3, 4 and 5 they file.

      Based solely on our review of the copies of such forms we have received, we believe that all of our executive officers, directors and greater then ten percent stockholders complied with all filing requirements applicable to them with respect to events or transactions during fiscal 2005, except that Douglas Jacobs was late in filing his Form 3 in connection with his initial appointment to the Board in 2003; and each of John Clymer, John Rees, James Stone and James Strickler were required to file Form 4A’s, which made certain adjustments and amendments to issuances of shares and options granted to such individuals during 2004 and 2005.

GOVERNANCE OF THE COMPANY

The Board of Directors and its Members

      During 2005, the Board of Directors held four in-person meetings and eight meetings by telephone conference. The Board of Directors also took actions by unanimous written consent. In 2005, each Director attended at least 75% of the aggregate of:

      Directors who are also officers of Hanover do not receive additional compensation for their services as Directors. In 2005, each non-employee Director received the following compensation:

      Board members are expected to attend the Company’s annual meeting of shareholders, which is held in conjunction with one of the Board’s regularly scheduled meetings. Accordingly, all members of the Board are generally present for the annual meeting. All members of the Board at the time of the Company’s 2005 annual meeting of shareholders attended that meeting.

      Pursuant to our stock option plans, non-employee Directors also receive a non-qualified stock option to purchase 2,000 shares of our common stock on the date he or she is first elected a member of the Board. Thereafter, non-employee Directors are entitled to receive a grant of a non-qualified stock option to purchase an additional 2,000 shares as of the date of each subsequent meeting of our shareholders at which he or she is re-elected to the Board. The purchase price per share for each option will be equal to the fair market value on the date the option is granted. Options to non-employee Directors are fully vested and immediately exercisable on the date of grant. In 2005, Mr. James F. Stone received options to purchase 2,000 shares of our common stock upon his re-election to the Board in 2005.

Role of the Committees of the Board of Directors

      The standing committees of the Board of Directors are the Audit Committee, the Nominating and Corporate Governance Committee and the Compensation Committee.

      Audit Committee. The Audit Committee was established by the Board of Directors for the purpose of overseeing the accounting and financial reporting processes and audits of the financial statements of the Company. The Audit Committee is comprised of four independent Directors, John N. Rees, John A. Clymer, James F. Stone and Douglas L. Jacobs, each of whom the Board has determined is independent within the meaning of SEC regulations and the American Stock Exchange listing requirements. The Board of Directors has determined that Mr. Rees is an “audit committee financial expert” as defined under Item 401(h) of Regulation S-K adopted by the SEC. Each member of the Audit Committee meets the requirements for financial literacy of the American Stock Exchange. The Audit Committee held nine meetings during 2005.

      Nominating and Corporate Governance Committee. Our Nominating and Corporate Governance Committee is comprised of three Directors, James F. Stone, Joseph J. Freeman and Saiyid T. Naqvi, each of whom meets the requirements for independence of the American Stock Exchange. The Nominating and Corporate Governance Committee recommends to the Board persons to be nominated as Directors or to fill vacancies on the Board. To fulfill this role, the Committee reviews the composition of the full Board to

determine the qualifications and areas of expertise needed to further enhance the composition of the Board and works with management in attracting candidates with those qualifications. Among other things, when assessing a candidate’s qualifications, the Committee considers:

      In addition, Directors are expected to be able to exercise their best business judgment when acting on behalf of the Company and its shareholders while relying on the honesty and integrity of the Company’s senior management and its outside advisors and auditors. The Committee considers all of these qualities when determining whether or not to recommend a candidate for Director.

      The Nominating and Corporate Governance Committee will consider nominees recommended by our shareholders. Recommendations must be delivered to our Secretary at our principal executive offices in accordance with the requirements set forth in Section 1.12 of our By-Laws. The Nominating and Corporate Governance Committee held one meeting during 2005.

      The Nominating and Corporate Governance Committee operates under a written charter, a copy of which is available on our website at www.hanovercapitalholdings.com. Please note that information on our website is not incorporated by reference in this proxy statement.

      Compensation Committee. Our Compensation Committee is comprised of four Directors, John A. Clymer, Joseph J. Freeman, Saiyid T. Naqvi and Douglas L. Jacobs, each of whom meets the requirements of independence of the American Stock Exchange. The Compensation Committee oversees and approves executive compensation policies and plans, including incentive and stock-based plans, which seek to enhance our profitability and value. Our Compensation Committee oversees these policies on behalf of the Board of Directors. The Compensation Committee held four meetings during 2005.

Codes of Conduct and Ethics

      The Board has adopted a code of business conduct and ethics that applies to all officers and employees and a code of ethics for principal executive and senior financial officers. The purpose of these codes is to deter wrongdoing and to promote:

      The “Code of Business Conduct and Ethics” and the “Code of Ethics for Principal Executive Senior Financial Officers” can be found under “Code of Conduct” and “Code of Ethics,” respectively, under the Corporate Governance section of our website at www.hanovercapitalholdings.com. Please note that information on our website is not incorporated by reference in this proxy statement.

AUDIT COMMITTEE REPORT

      The Audit Committee is comprised of four independent Directors, each of whom meets the requirements for independence and financial literacy of the American Stock Exchange. The Audit Committee operates pursuant to a written charter adopted by the Board of Directors. The current charter of the Audit Committee is attached to this proxy statement as “Appendix A” and is available on our website at www.hanovercapitalholdings.com. Please note that information on our website is not incorporated by reference in this proxy statement.

      Management is responsible for internal controls and the financial reporting process. The independent accountants are responsible for performing an independent audit of Hanover’s consolidated financial statements in accordance with auditing standards generally accepted in the United States of America and for issuing a report thereon. The Audit Committee’s responsibility is to oversee the accounting and financial reporting processes and audits of the financial statements of the Company.

      In this context, the Audit Committee has met and held discussions with management and Grant Thornton LLP, Hanover’s independent accountants for the year ended December 31, 2005. Management represented to the Audit Committee that Hanover’s consolidated financial statements were prepared in accordance with accounting principles generally accepted in the United States of America, and the Audit Committee has reviewed and discussed the audited consolidated financial statements with management and the independent accountants. The Audit Committee discussed with Grant Thornton LLP the matters required to be discussed by Statement on Auditing Standards No. 61 (Communication with Audit Committees). These matters included a discussion of Grant Thornton LLP’s judgments about the quality (not just the acceptability) of our accounting principles as applied to financial reporting.

      Grant Thornton LLP also provided the Audit Committee with the written disclosures and letter required by Independence Standards Board Standard No. 1 (Independence Discussions with Audit Committees), and the Audit Committee discussed with Grant Thornton LLP its independence.

      Based upon the reviews and discussions described above, the Audit Committee recommended to the Board of Directors that Hanover’s audited consolidated financial statements be included in its Annual Report on Form 10-K for the year ended December 31, 2005, for filing with the Securities and Exchange Commission. The Audit Committee also recommended the selection of Grant Thornton LLP as Hanover’s independent accountants for 2006.

DISCLOSURE OF FEES CHARGED BY PRINCIPAL ACCOUNTANTS

      The following table presents expenses incurred for professional services rendered by Grant Thornton LLP, the Company’s principal accountant, for audit services, audit-related services, tax services and all other services in 2005 and 2004.

Audit Committee Pre-Approval of Audit and Permissible Non-Audit Services of Independent Accountants

      The policy of the Audit Committee is to pre-approve all audit and permissible non-audit services to be performed by the independent accountants during the fiscal year.

INDEPENDENT PUBLIC ACCOUNTANTS

      On February 18, 2004, Deloitte & Touche LLP, our former independent accountants (“D&T”), advised us that it would not stand for reelection as our independent accountants following the completion of its audit of our December 31, 2003 financial statements. On April 13, 2004, we engaged Grant Thornton LLP as our new independent accountants. The engagement of Grant Thornton LLP was approved by our Audit Committee and subsequently by the shareholders at the Annual Meeting of Shareholders on May 20, 2004.

      D&T’s report on our financial statements for the fiscal years ended December 31, 2003 and 2002 did not contain an adverse opinion or a disclaimer of opinion and was not qualified or modified as to uncertainty, audit scope or accounting principles. However, D&T’s opinion dated April 13, 2004 contained an explanatory paragraph relating to the restatement discussed in note 21 to our 2003 financial statements contained in our Annual Report on Form 10-K for the year ended December 31, 2003. During the last two fiscal years and the subsequent interim period through April 20, 2004, there were no disagreements (as defined in Item 304(a)(1)(iv) of Regulation S-K) with D&T on any matter of accounting principles or practices, financial statement disclosure or auditing scope or procedure, which disagreements if not resolved to the satisfaction of D&T, would have caused D&T to make reference to the subject matter of the disagreements in connection with its report on the financial statements for such years, except as follows:

      (1) In the course of D&T’s audit of our financial statements for the year ended December 31, 2003, D&T advised us on March 25, 2004 that it believed that the accounting treatment relating to Amendment No. 1 to the Contribution Agreement, dated July 1, 2002 (the “Contribution Agreement”), between us and four of our executive officers, should be revised.

      At that time, D&T advised us that it believed that the distributions of shares of our common stock to the executives and forgiveness of certain of the executives’ indebtedness, both pursuant to the Contribution Agreement, should have been accounted for as compensation expense in the second quarter of 2003 and, accordingly, as a reduction of our net earnings. D&T also advised us that it believed that notes receivable from executives related to this agreement should be treated as a reduction of Shareholders’ Equity. We had previously accounted for this transaction as an adjustment to additional paid-in capital in our financial statements included in our quarterly reports for the second and third quarters of 2003 and had reported the notes receivable as assets in our financial statements.

      Our discussions with D&T in the course of our review of the accounting treatment were deemed by D&T to constitute a disagreement (as that term is used in Item 304(a)(1)(iv) of Regulation S-K). We revised the accounting treatment for this transaction and accordingly the accounting matter was resolved to the satisfaction of D&T. Our Annual Report on Form 10-K for the year ended December 31, 2003 reflected our revised accounting treatment.

      (2) On April 13, 2004, D&T reported to our Audit Committee that the design of our financial infrastructure created an environment that could adversely affect our ability to produce timely financial statements. D&T reported that the design did not include the appropriate preventative and detective controls to identify misstatements of accounting information as well as appropriate procedures for appropriately assessing and applying accounting principles. D&T reported that it considered such matters to be a material weakness, as defined by the standards established by the American Institute of Certified Public Accountants, involving the design of our internal controls over financial reporting and their operation. D&T advised our Audit Committee that it believed the matter constituted a reportable event (as that term is used in Item 304(a)(1)(v) of Regulation S-K). We had a difference of opinion with D&T as to whether the weakness reported by D&T constituted a material weakness in the design of our

internal controls over financial reporting. We do not believe that the design of our financial infrastructure lacks appropriate preventative and detective controls to identify misstatements of accounting information as well as appropriate procedures for appropriately assessing and applying accounting principles.

      The need for the restatements of our financial statements described above arose solely out of the change in accounting treatment with respect to the Contribution Agreement. We have taken steps that we believe will enhance and improve our internal controls and procedures, including the hiring of replacements in the positions of general counsel and controller and the addition of a new senior vice president of operations. We have also increased and upgraded the capabilities of our accounting system, in order to address the issues raised by D&T.

      We have authorized D&T to respond fully to the inquiries of our new independent auditors concerning the subject matter of the foregoing disagreement and difference of opinion.

      During the fiscal years ended December 31, 2002 and 2003 and the subsequent interim period through the date of the engagement of Grant Thornton LLP, we did not consult with Grant Thornton LLP regarding the application of accounting principles to a specified transaction, either completed or proposed, the type of audit opinion that might be rendered on our consolidated financial statements, or any matter that was the subject of disagreement or a reportable event as set forth in Items 304(a)(1)(iv) and (v) of Regulation S-K.

EXECUTIVE COMPENSATION

Summary Compensation Table

      The following table contains information about the compensation of our chief executive officer and four other most highly compensated executive officers serving at the end of 2005 for services rendered to us in 2005, 2004 and 2003.

Aggregated Option/ SAR Exercises in Last Fiscal Year and Fiscal Year-End Option/ SAR Values

      The following table shows 2005 stock option exercises and the value of unexercised options for those officers named in the Summary Compensation Table. In the case of exercised options, “value realized” is considered to be the difference between the exercise price and the market price on the date of exercise. In the case of unexercised options, value is considered to be the difference between the exercise price and the market price at the end of 2005. An “In-the-Money” option is an option for which the exercise price is less than $6.68, the closing market price of our common stock on the last trading day of 2005.

Equity Compensation Plan Information

Employment Agreements

      Effective July 1, 2002, we entered into Amended and Restated Employment Agreements with each of Mr. Burchett, Ms. Mizerak, Mr. Ostendorf and Ms. Tavares. The Amended and Restated Employment Agreements provide for an initial annual base salary of $331,770 for Mr. Burchett and $248,827 for each of Ms. Mizerak, Mr. Ostendorf and Ms. Tavares. These base salaries may be increased annually at the discretion of the Compensation Committee for cost-of-living adjustments and merit increases, among other things. Each of these employment agreements has a five-year term and will automatically renew for successive one-year terms thereafter until the officer or we terminate the agreement. Each of Mr. Burchett, Ms. Mizerak, Mr. Ostendorf and Ms. Tavares is eligible to participate in the Bonus Incentive Compensation Plan, the 1997 Executive and Non-Employee Director Stock Option Plan and the 1999 Equity Incentive Plan. Mr. Burchett is also entitled to $2 million in term life insurance, and Ms. Mizerak, Mr. Ostendorf and Ms. Tavares are each entitled to $1.5 million in term life insurance. In addition, these officers are entitled to a car allowance, club dues and disability insurance.

      Our employment agreements with Messrs. Burchett and Ostendorf, Ms. Mizerak and Ms. Tavares contain a provision prohibiting competition with us for a certain period following his or her termination for “good cause.” Good cause means:

      Under these employment agreements, each employee is entitled to receive his or her base salary in effect at the date of termination until the later of one year from the termination date or the end of the employment term if the employee:

      Within 90 days after a change of control of Hanover, if any of Mr. Burchett, Ms. Tavares, Ms. Mizerak or Mr. Ostendorf is terminated without good cause, or resigns without a pending termination for good cause, then he or she will be entitled to receive his or her base salary then in effect until the later of two years from the date of termination or to the end of the term of the employment agreement.

Change in Control

      On March 30, 2000, the Board of Directors adopted policies to take effect in the event a single person, entity or a group of persons and/or entities acting in concert acquire control of us. If there is a change of control, the Chief Executive Officer may:

      In addition, pursuant to our 1999 Equity Incentive Plan and 1997 Executive and Non-Employee Director Stock Option Plan, our Compensation Committee can take certain actions prior to a change in control, including:

      Our Stockholder Protection Rights Agreement became effective on April 28, 2000 and provides that the holder of a Right, upon the exercise of the Right, is entitled to purchase from us one one-hundredth of a share of Participating Preferred Stock at an exercise price of $17.00, subject to adjustment. The Stockholder Protection Rights Agreement provides that upon the separation time, which is when there is a public announcement by a person to acquire beneficial ownership of 10% or more of our common stock, the Rights will become exercisable and entitle each holder of a Right, other than Rights that are owned by the acquiring person, the right to receive shares of common stock having a market value of two times the exercise price of the Right. Our Board may amend the Agreement anytime prior to the separation time in any respect. On June 10, 2002 we amended our Stockholder Protection Rights Agreement to change the ownership limit applicable to Mr. Burchett from 18% to 20%. A copy of the Agreement was filed with the SEC on Form 8-A on April 24, 2000, and a copy of the amendment was filed with the SEC on Form 8-K on July 16, 2002. A copy of the Agreement as amended is available from us free of charge.

Compensation Committee Report on Executive Compensation

      The members of the Compensation Committee are John A. Clymer, Joseph J. Freeman, Saiyid T. Naqvi and Douglas L. Jacobs; each is an independent Director. In this report, the term “we” refers to members of the Committee.

      We are responsible to the Board of Directors and, ultimately, Hanover’s shareholders, for:

      Our principal objectives are to oversee the development and implementation by the Company of compensation policies that will:

      To this end, we employ compensation policies and plans which combine (1) competitive base salaries with (2) bonuses and stock-based compensation which emphasize the relationship between executive compensation and corporate performance.

      We regularly review the base salaries payable to executive officers. The base salary payable to the four original principals and the senior executive officers, Mr. Burchett, Ms. Tavares, Ms. Mizerak, and Mr. Ostendorf, is determined in part by the terms of their respective employment agreements, which requires their base salary to be adjusted annually to reflect, at a minimum, the cost of living. These agreements expire on July 1, 2007. The base salary payable pursuant to their employment agreements was last increased to reflect cost-of-living adjustments in July 2005. The base salary payable to other executive officers is determined by looking at factors such as performance, salary levels in the industry, and certain financial performance targets.

      In 1997 Hanover adopted a Bonus Incentive Compensation Plan to provide annual bonuses for eligible participants, which include executive officers, directors, employees, independent contractors and consultants of Hanover and its subsidiaries. The amount of awards available under this plan is based upon a percentage of Hanover’s annual net income, taking into account the effect on earnings per share (“EPS”). Bonuses may be paid one-half in cash and, subject to ownership limits, one-half in shares of Hanover’s common stock. By linking the available bonus pool to Hanover’s net income, the Bonus Incentive Compensation Plan provides additional incentive for eligible recipients to work towards Hanover’s

continued growth and success. Based on our 2005 performance, since our EPS was not at the targeted level, no funds were accrued to pay bonuses to our executive officers.

      Hanover adopted its two stock-based incentive plans in order to attract, motivate and retain qualified personnel. We believe that stock-based compensation provides additional incentive to contribute to the success of Hanover, since the value of such compensation is directly related to the market value of Hanover’s common stock.

      1997 Executive and Non-Employee Director Stock Option Plan. Under this Plan we have authority to oversee and approve the granting of incentive stock options, non-qualified stock options, restricted stock, performance shares, stock appreciation rights and other stock-based awards. We review the terms and restrictions applicable to any such awards, and assist in the determination of the participants eligible to receive them. Eligible persons under the Plan are executive officers, Directors and employees of Hanover or its subsidiaries and other persons expected to provide significant services to Hanover. Incentive stock options may only be granted to the officers and key employees of Hanover and its subsidiaries. To date, all options granted under this Plan have been granted at an exercise price equal to the fair market value on the date of grant. Under the terms of this Plan, Mr. James F. Stone was awarded options to purchase 2,000 shares of Hanover’s common stock upon his re-election to the Board in 2005. Additionally, under the terms of the Plan, we issued a total of 22,000 shares of restricted common stock to certain of our employees in 2005.

      Subject to anti-dilution provisions for stock splits, stock dividends and similar events, this Plan authorizes the grant of options to purchase, and awards of, an aggregate of up to 325,333 shares of Hanover’s common stock. If an option granted under this Plan expires or terminates, or an award is forfeited, the shares subject to any unexercised portion of such option or award will again become available for the issuance of further options or awards under this Plan. As of December 31, 2005, we had 28,009 shares of common stock remaining available for issuance of further options or awards under this Plan.

      1999 Equity Incentive Plan. This Plan authorizes us to grant non-qualified stock options or restricted stock to executive officers, key employees, Directors, agents, advisors and consultants of Hanover and its subsidiaries. To date, all options granted under the 1999 Equity Incentive Plan have been granted at an exercise price equal to the fair market value on the date of grant. No stock options were awarded under the 1999 Equity Incentive Plan during 2005.

      Subject to anti-dilution provisions for stock splits, stock dividends and similar events, the 1999 Equity Incentive Plan authorizes the grant of options to purchase, and awards of, an aggregate of up to 550,710 shares of Hanover’s common stock. If an option granted under the 1999 Equity Incentive Plan expires or terminates, or an award is forfeited, the shares subject to any unexercised portion of such option or award will again become available for the issuance of further options or awards under the 1999 Equity Incentive Plan.

      No eligible participant can be granted options exercisable into, or awards of, more than 50,000 shares of Hanover’s common stock in any year. As of December 31, 2005, we had 14,167 options for shares of common stock remaining available for issuance under this Plan.

      In connection with its initial public offering in 1997, Hanover entered into a Contribution Agreement with Mr. Burchett, Ms. Mizerak, Mr. Ostendorf and Ms. Tavares which provided that they were entitled to receive an aggregate of up to 216,667 shares of Hanover’s common stock, and to have certain indebtedness to Hanover forgiven, if Hanover met performance targets based on the initial offering price over five annual performance periods, the last of which ended on September 30, 2002. In addition, options exercisable into an aggregate of 80,160 shares were granted to these four executives, with vesting conditioned on the same performance targets

and periods. None of the targets were met within the first four periods, so none of the shares were issued, none of the options vested and none of the loans were forgiven. However, on July 1, 2002, we modified the performance period and target amounts. As a result, the shares could be issued, the options could vest and the loans could be forgiven, in annual performance periods beginning July 1, 2003 and ending on July 1, 2007, if Hanover meets new performance targets based on the average closing price of its common stock for the twenty consecutive trading days proceeding July 1 for each of these years.

      On July 1, 2003, the first performance target was met and, as a result, one-third, or 26,720, of the 80,160 options vested, $583,333 in loans was forgiven and 72,222 shares of common stock were granted. On July 1, 2004, the second performance target was met and, as a result, another one-third, or 26,720, of the 80,160 options vested, another $583,333 in loans was forgiven and another 72,222 shares of common stock were granted.

      On July 1, 2005, the conditions of the 1997 Contribution Agreement, as amended, were fulfilled. As a result, the remaining obligations of the principals of $583,333 of outstanding loans to the Company were forgiven and recognized as compensation expense in the year ended December 31, 2005. In addition, a total of 72,222 shares of common stock were earned and subsequently issued to them. The shares were valued at the closing price of June 30, 2005, and the amount of approximately $761,000 has been recognized as compensation expense in the year ended December 31, 2005. As of December 31, 2005, there are no remaining obligations under the 1997 Contribution Agreement.

      In 2005, the most highly compensated officer was John A. Burchett, our Chief Executive Officer, President and Chairman of the Board. For 2005, Mr. Burchett received a base salary of $359,498, and a bonus of $157,218, based on 2004 performance and paid in 2005, which included a grant of 2,357 shares of common stock in lieu of cash. Mr. Burchett’s base salary was increased to its present level in July 2005 to reflect cost-of-living adjustments.

      We determined Mr. Burchett’s bonus for 2004, paid in 2005, based on a number of factors. These factors included: (1) Mr. Burchett’s company-wide responsibilities and leadership, in addition to his day-to-day contributions to Hanover’s operations; (2) Mr. Burchett’s efforts to identify corporate opportunities, develop and implement long-term corporate strategies and adapt those strategies to changing economic environments; (3) Hanover’s return-on-equity and income during 2004 and over the prior five years; and (4) Hanover’s stock price performance and dividends paid during 2004 and over the prior five years. As noted above, no bonus monies were accrued in 2005 to pay bonuses to the executive officers in 2006.

      We believe that the compensation policies described above align the interests of Hanover’s management and its shareholders by attracting highly qualified executives and motivating them to increase long- and short-term corporate success. We will continue to monitor the effectiveness of Hanover’s compensation programs to ensure that they meet Hanover’s current and future needs.

COMPENSATION COMMITTEE INTERLOCKS AND INSIDER PARTICIPATION

      During 2005, Messrs. Clymer, Freeman, Naqvi and Jacobs served on our Compensation Committee. None of these Directors has ever been an officer or employee of Hanover or our subsidiaries.

      None of our directors or other executive officers served as a director or executive officer of another corporation that has a director or executive officer serving on our Board of Directors.

CERTAIN RELATIONSHIPS AND RELATED TRANSACTIONS

      During 2005, Frank Siermine, a brother of Joyce S. Mizerak, a Director, Secretary and Senior Managing Director, was employed as a vice president for Hanover Capital Partners Ltd. and received an aggregate salary of $93,730. Hanover believes that this transaction and relationship during 2005 were on terms that were reasonable and in the best interests of Hanover.

      In connection with our initial public offering, we entered into a Contribution Agreement with Mr. Burchett, Ms. Mizerak, Mr. Ostendorf and Ms. Tavares which provided that they were entitled to receive up to an aggregate of 216,667 shares of our common stock, called “earn-out shares,” and to have up to an aggregate of $1,750,000 of indebtedness to us forgiven, if we met performance targets based on the initial offering price over five annual performance periods, the last of which would have been September 30, 2002. In addition, options exercisable into an aggregate of 80,160 shares of our common stock were granted to these four executives, with vesting conditioned on the same performance targets and periods. None of the targets were met within the first four periods, so none of the shares were issued, none of the options vested, and none of the loans were forgiven.

      On July 1, 2002, we modified the performance periods and target amounts applicable to the earn-out shares, loans and options. As a result, up to 216,667 shares of our common stock could be issued, options exercisable for up to 80,160 shares of our common stock could vest and up to $1,750,000 in loans could be forgiven, on annual measuring dates ending on July 1, 2007, if we meet certain performance targets.

      The performance targets are based on the return on our common stock. One-third of the earn-out shares will be granted, one-third of the options will vest and one-third of the loans will be forgiven on any measuring date through which the return on a share of stock is at least equal to a 15% annualized return on $8.9345, which was the average of the daily market price for the twenty consecutive trading days prior to July 1, 2002.

      The “return on a share of our common stock” is determined by adding:

      In determining whether the earn-out has vested, appropriate adjustments will be made for stock splits, recapitalizations, stock dividends and transactions having similar effects.

      During 2003, the first performance target was met and, as a result, one-third, or 26,720, of the 80,160 options vested with the four executives, $583,333 in loans were forgiven and 72,222 shares of common stock were granted in the aggregate. During 2004, the second performance target was met and, as a result, another one-third, or 26,720, of the 80,160 options vested with the four executives, $583,333 in loans were forgiven and 72,222 shares of common stock were granted in the aggregate. During 2005, the conditions of the 1997 Contribution Agreement, as amended, were fulfilled. As a result, the remaining obligations of the

principals of $583,333 of outstanding loans to the Company were forgiven and recognized as compensation expense in the year ended December 31, 2005. In addition, a total of 72,222 shares of common stock were earned and subsequently issued to them. The shares were valued at the closing price of June 30, 2005 and the amount of approximately $761,000 has been recognized as compensation expense in the year ended December 31, 2005. As of December 31, 2005, there are no remaining obligations under the 1997 Contribution Agreement.

      In connection with our initial public offering, we agreed to lend up to an aggregate of $1,750,000 to Mr. Burchett, Ms. Mizerak, Mr. Ostendorf and Ms. Tavares to enable them to pay their personal income taxes on the gains they recognized upon contributing Hanover Capital Partners Ltd. preferred stock to us for shares of our common stock. An aggregate of $1,750,000 was borrowed pursuant to this agreement. These loans originally matured in September of 2002, but on July 1, 2002, the maturity was extended to July 1, 2007. No payment of principal on the loans is due before maturity unless the borrower is terminated for “good cause” under his or her employment agreement with us, in which case the loan will become immediately due and payable. Interest, however, is payable on a quarterly basis in arrears. The loans to Mr. Burchett, Ms. Mizerak, Mr. Ostendorf and Ms. Tavares are secured by an aggregate of 116,667 of their shares of our common stock, but are otherwise nonrecourse. As additional consideration to Mr. Burchett, Ms. Mizerak, Mr. Ostendorf and Ms. Tavares for their contribution of the preferred stock of Hanover Capital Partners Ltd. to us, the outstanding balance of the loans will be forgiven to the extent that we meet established performance targets. During 2005, the conditions of the 1997 Contribution Agreement, as amended, were fulfilled. As a result, the remaining obligations of the principals of $583,333 of outstanding loans to the Company were forgiven and recognized as compensation expense in the year ended December 31, 2005. In addition, a total of 72,222 shares of common stock were earned and subsequently sold to them. The shares were valued at the closing price of June 30, 2005 and the amount of approximately $761,000 has been recognized as compensation expense in the year ended December 31, 2005. As of December 31, 2005, there are no remaining obligations under the 1997 Contribution Agreement.

PERFORMANCE GRAPH

      The graph below compares the cumulative total shareholder return on our common stock to the cumulative total return on the S&P Composite-500 Stock Index and an index average of our peer group, over the past five years.

      Our peer group is composed of the following publicly-traded companies: Capstead Mortgage Corporation, DYNEX Capital, Inc., Impac Mortgage Holdings, Inc., IndyMac Bancorp, Inc., Redwood Trust, Inc. and Thornburg Mortgage Asset Corporation.

      The returns reflect stock price appreciation for our common stock and for each of the comparative indices. The graph assumes that $100 was invested in our common stock and in each of the indices on December 31, 2000 and assumes the reinvestment of any dividends. The graph lines connect fiscal year-end dates and do not reflect price fluctuations between those dates. The total return performance shown on the graph is not necessarily indicative of future total return performance of our common stock.

SHAREHOLDER PROPOSALS

      Any shareholder who intends to present a proposal at our 2007 annual meeting must deliver the proposal to our principal executive offices at 200 Metroplex Drive, Suite 100, Edison, New Jersey 08817 no later than December 22, 2006 if the proposal is submitted for inclusion in our proxy materials for that meeting pursuant to Rule 14a-8 under the Securities Exchange Act of 1934.

      Any shareholder proposal for our 2007 annual meeting that is submitted outside the processes of Rule 14a-8 will be considered “untimely” pursuant to our bylaws if we receive it before February 18, 2007 or after March 20, 2007. An untimely proposal may be excluded from consideration at our 2007 annual meeting and, if such proposals are brought before the meeting, proxies solicited by the Board of Directors for our 2007 annual meeting may confer discretionary authority to vote on any such untimely proposal without express direction from shareholders giving such proxies. Such proposal must be delivered to our Secretary at our principal executive offices.

CONTACTING THE BOARD OF DIRECTORS

      Any shareholder who desires to contact the Chairman of the Board or the independent Directors as a group may do so by writing to: Board of Directors, Hanover Capital Mortgage Holdings, Inc., 200 Metroplex Drive, Suite 100, Edison, New Jersey 08817. Communications received in writing are distributed to the Chairman of the Board or the other members of the Board as appropriate depending on the facts and circumstances outlined in the communication received.

OTHER MATTERS

      We know of no other matters to be presented at the meeting. If other matters are considered at the meeting, the proxies will vote on these matters in accordance with their own discretion.

      Shareholders who would like an additional copy of our Annual Report on Form 10-K may obtain it, free of charge, upon request to our Secretary. Any such requests should be directed to Suzette N. Bemois, Vice President and General Counsel, Hanover Capital Mortgage Holdings, Inc., 200 Metroplex Drive, Suite 100, Edison, New Jersey, 08817, telephone number (732) 548-0101.

      IT IS IMPORTANT THAT PROXIES BE RETURNED PROMPTLY. WE URGE YOU TO COMPLETE, SIGN, DATE AND RETURN THE ACCOMPANYING PROXY IN THE ENCLOSED ENVELOPE AS SOON AS POSSIBLE.

April 21, 2006

Audit Committee Charter

of the Board of Directors of

Hanover Capital Mortgage Holdings, Inc.

Organization

      The Audit Committee of the Board of Directors of Hanover Capital Mortgage Holdings, Inc. (the “Company”) shall be comprised of at least three directors, each of which shall meet the independence requirements established by the Securities and Exchange Commission (the “SEC”) and the national securities exchange or over the counter market upon which the Company’s common stock is then listed. Additionally, each Audit Committee member shall be able to read and understand fundamental financial statements, including a company’s balance sheet, income statement, and cash flow statement or become able to do so within a reasonable period of time after his or her appointment to the Audit Committee. At least one Audit Committee member shall, so long as the Company’s common stock is listed on the American Stock Exchange ( the “AMEX”), be “financially sophisticated” as defined by the AMEX.

      The board of directors shall endeavor to appoint at least one member of the Audit Committee who shall have such accounting and/or financial expertise so as to qualify such individual as an “audit committee financial expert” as defined under Item 401(h) of Regulation S-K adopted by the SEC.

Statement of Policy and Purpose

      The Audit Committee shall provide assistance to the members of the board of directors in fulfilling their responsibility to oversee the quality and integrity of financial reports of the Company, the Company’s compliance with legal and regulatory requirements, the qualifications and independence of the Company’s independent auditors, and the performance of the Company’s internal audit function and its independent auditors. In so doing, it is the responsibility of the Audit Committee to maintain free and open communications between the directors, the independent auditors, and the financial management of the Company. It is the expectation of the Audit Committee that the financial management will fulfill its responsibility of bringing any significant items to the attention of the Audit Committee.

Powers and Responsibilities

      In carrying out its responsibilities, the Audit Committee believes its policies and procedures should remain flexible, in order to best react to changing conditions and to ensure to the directors and stockholders that the corporate accounting and reporting practices of the Company are in accordance with pertinent requirements. The Audit Committee also maintains procedural guidelines, separate from this Charter, which governs how the Audit Committee operates internally. In carrying out these responsibilities, the Audit Committee will:

      1. Obtain annually the full Board of Directors’ approval of this Charter and review and reassess this Charter as conditions dictate.

      2. Appoint the independent auditors to audit the financial statements of the Company and its divisions and subsidiaries, and to perform other audit, review or attest services as the Audit Committee deems advisable.

      3. Engage the independent auditors to perform tax services and other de-minimus services subject to any prohibitions against non-audit services as set forth in the Sarbanes-Oxley Act, the national securities exchange or over the counter market upon which the Company’s common stock is then listed, or any other

applicable federal and state laws and regulations. The performance of such services by the independent auditors shall be approved in advance by the Audit Committee.

      4. Have a clear understanding with the independent auditors that the independent auditors are ultimately accountable to the Board of Directors and the Audit Committee, as the stockholders’ representatives, and that it is the Audit Committee, as the stockholders’ representative, to whom the independent auditors shall report directly, that has the ultimate authority in deciding to engage, evaluate and, if appropriate, terminate their services.

      5. Meet with the independent auditors and financial management of the Company to establish the scope of the proposed audit and quarterly reviews for the current year and the procedures to be utilized, the adequacy of the independent auditors’ compensation and at the conclusion thereof review such audit or reviews, including any comments or recommendations of the independent auditors.

      6. Review with the independent auditors and financial and accounting personnel, the adequacy and effectiveness of the accounting, and financial controls of the Company, and elicit any recommendations for the improvement of such internal controls or particular areas where new or more detailed controls or procedures are desirable.

      7. Review the financial statements contained in the Company’s quarterly and annual reports with management and the independent auditors to determine that the independent auditors are satisfied with their disclosure and content. Review with financial management and the independent auditors significant financial reporting issues and practices, including changes in, or adoptions of, accounting principles, disclosure practices and quarterly reports. Be responsible for resolution of any disagreements between management and the independent auditors.

      8. Meet separately, on a periodic basis, with management, with those responsible for the internal audit function, and with the independent auditors.

      9. Report the results of the annual audit to the Board of Directors.

      10. On an annual basis, obtain from the independent auditors a written communication delineating all their relationships and professional services as required by Independence Standards Board Standard No. 1, Independence Discussions with Audit Committees, and including therein all other matters required by AMEX. Upon receipt of the communication, the Audit Committee shall review with the independent auditor any disclosed relationships or services that could impact the objectivity and independence of the independent auditors, and, if deemed appropriate by the Audit Committee, take or recommend that the Board take the appropriate action to ensure such objectivity and independence.

      11. Prepare a report of the Audit Committee for inclusion in the annual proxy statement.

      12. Submit the minutes of all meetings of the Audit Committee to the Board of Directors.

      13. Establish a whistleblower policy to include procedures for: (i) the receipt, retention and treatment of complaints received by the Company regarding accounting, internal accounting controls, or auditing matters and (ii) the confidential, anonymous submission by employees of the Company of concerns regarding questionable accounting or auditing matters, and to investigate any such matter brought to its attention within the scope of its duties.

      14. Establish a hiring policy for employees or former employees of the Company’s independent auditors.

      15. At least annually, obtain and review a report by the independent auditor describing (i) the firm’s internal quality-control procedures; (ii) any material issues raised by the most recent internal quality-

control review, or peer review, of the firm, or by any inquiry or investigation by governmental or professional authorities within the preceding five years, respecting one or more independent audits carried out by the firm, and any steps taken to deal with any such issues; and (iii) all relationships between the independent auditor and the company.

      16. Conduct an evaluation of the independent auditor’s work throughout the year, addressing the independent auditor’s qualifications, performance and independence.

      17. Discuss with management the Company’s earnings press releases, including any non-GAAP information, as well as financial information and earnings guidance provided to analysts and rating agencies

      18. Discuss the Company’s major financial risk exposures and its guidelines and policies governing the process by which the Company assesses and manages its exposure to such risks.

      19. Review the performance of the Audit Committee on an annual basis and report the results of such assessment to the Board of Directors.

      20. Meet on at least a quarterly basis. A majority of the Audit Committee shall constitute a quorum for the transaction of business and the act of a majority of those present at any meeting at which there is a quorum shall be the act of the Audit Committee.

      21. With respect to advisors and legal counsel, the Committee shall, in its sole discretion, as it deems advisable, retain at the Company’s expense, accountants, legal advisors and other advisors and consultants to assist in the performance of its duties.

3362-PS-06