SECURITIES AND EXCHANGE COMMISSION

Washington, D.C. 20549

SCHEDULE 13D

Under the Securities Exchange Act of 1934

The Chalone Wine Group, Ltd.

(Name of Issuer)

Common Stock, no par value

(Title of Class of Securities)

157639105

(CUSIP Number)

Thomas J. Mullin, Esq.

Executive Vice President and

General Counsel

Constellation Brands, Inc.

370 Woodcliff Drive

Suite 300

Fairport, New York 11450

(585) 218-3650

With a copy to:

Richard F. Langan, Jr., Esq.

Nixon Peabody LLP

437 Madison Avenue

New York, New York 10022

(Name, Address and Telephone Number of Person Authorized to Receive Notices and Communications)

May 16, 2004

(Date of Event which Requires Filing of this Statement)

If the filing person has previously filed a statement on Schedule 13G to report the acquisition that is the subject of this Schedule 13D, and is filing this schedule because of § 240.13d-1(e), 240.13d-1(f) or 240.13d-1(g), check the following box ¨.

Note: Schedules filed in paper format shall include a signed original and five copies of the schedule, including all exhibits. See § 240.13d-7(b) for other parties to whom copies are to be sent.

The information required in the remainder of this cover page shall not be deemed to be “filed” for the purpose of Section 18 of the Securities Exchange Act of 1934 (“Act”) or otherwise subject to the liabilities of that section of the Act but shall be subject to all other provisions of the Act (however, see the Notes).

SCHEDULE 13D

This Schedule 13D relates to shares of the common stock, no par value (the “Common Stock”), of The Chalone Wine Group, Ltd., a California corporation (the “Issuer”). The Issuer’s principal executive office is located at 621 Airpark Road, Napa, California 94585-6272.

This Schedule 13D is being filed by Constellation Brands, Inc, a Delaware corporation (“Constellation”). Constellation is a leading international producer and marketer of beverage alcohol brands with a portfolio across the wine, imported beer and spirits categories. The address of its principal business and its principal office is 370 Woodcliff Drive, Suite 300, Fairport, NY 14450. During the last five years, Constellation has not been convicted in a criminal proceeding (excluding traffic violations or similar misdemeanors) and has not been a party to a civil proceeding of a judicial or administrative body of competent jurisdiction as a result of which it was or is subject to a judgment, decree or final order enjoining future violations of, or prohibiting or mandating activities subject to, Federal or state securities laws or finding any violation with respect to such laws.

The following sets forth with respect to each executive officer and director of Constellation the person’s (a) name, (b) residence or business address, (c) present principal occupation or employment and the name, principal business and address of any corporation or other organization in which such employment is conducted, and (d) citizenship.

During the past five years, none of the individuals named above has been convicted in a criminal proceeding (excluding traffic violations or similar misdemeanors) or has been a party to a civil proceeding of a judicial or administrative body of competent jurisdiction, as a result of which he or she was or is subject to a judgment, decree, or final order enjoining future violations of, or prohibiting or mandating activities subject to, Federal or state securities laws or finding violations with respect to such laws.

Constellation may be deemed to constitute a member of a group within the meaning of Rule 13d-5 promulgated under the Securities Exchange Act of 1934, as amended (the “Exchange Act”), together with Domaines Baron de Rothschild (Lafite), a “societe en comandite par actions” organized and existing under the laws of France (“DBR”), and Huneeus Vintners, LLC, a Delaware limited liability company (“Huneeus”). Constellation expressly declares that neither the filing of this Schedule 13D nor any of the information contained herein shall be construed as an admission that it has formed or is a member of any such group. Information with respect to

the beneficial ownership of Common Stock of the Issuer by DBR and Huneeus and related information should be contained in Schedules 13D, as amended, filed or to be filed by each of them. Reference is made to Item 2 of those Schedules 13D, as amended, for information as to the identity and background of DBR and Huneeus and persons related to them.

Constellation has no responsibility for the accuracy or completeness of any information given with respect to any other person or entity, including, without limitation, DBR and Huneeus. Any disclosures made herein with respect to any person or entity other than Constellation are made on information and belief.

Constellation has not purchased any shares of Common Stock of the Issuer.

As more fully described herein, Constellation has entered into a Commitment Letter Summary, dated May 16, 2004 (the “Letter Agreement”), with DBR and Huneeus pursuant to which, among other things, they indicated their intent to enter into certain transactions among themselves and, in connection therewith and prior thereto, to propose to the special committee of the board of directors of the Issuer that the Issuer enter into certain transactions. A copy of the Letter Agreement is annexed hereto as Exhibit 1 and incorporated herein by reference. In connection with the transactions contemplated by the Letter Agreement, Constellation, DBR and Huneeus (collectively, the “Principals”) have agreed to establish a joint venture company (the “JV”) to which Constellation will contribute $54 million in cash and certain assets, as described more fully herein, and Huneeus will contribute certain assets and liabilities, and to which DBR will, upon the completion of the merger contemplated by the Letter Agreement, cause the Issuer

to contribute all of its assets and liabilities. Except for specified provisions of the Letter Agreement which are binding on the parties, the transactions contemplated by the Letter Agreement are subject to the preparation, execution and delivery of definitive agreements satisfactory to each of the Principals and any liability among the Principals shall attach only upon such execution and delivery of the merger agreement, contemplated by the Letter Agreement in a form acceptable to the Principals with the Issuer.

Constellation anticipates that the source of the $54 million to be contributed by it to the JV will be working capital of Constellation and borrowings obtained under Constellation’s credit facility. Constellation anticipates that the assets to be contributed by Constellation to the JV will be conveyed from among the operating assets of Constellation and/or one or more of its subsidiaries. Those assets are expected to consist of additional vineyard and winery operations to be contributed by Constellation.

The Letter Agreement provides that the JV will borrow a portion of the funds necessary for the merger contemplated by the Letter Agreement, and, in turn, lend those funds to the holding company that is expected to effect the merger. In addition, the Letter Agreement provides that DBR will contribute $10 million to the holding company.

The Principals entered into the Letter Agreement for the purpose of entering into certain transactions among themselves and, in connection therewith and prior thereto, to propose to the special committee of the board of directors of the Issuer that the Issuer enter into certain transactions. In connection with the transactions contemplated by the Letter Agreement,

the Principals agreed to establish the JV to which Constellation will contribute $54 million in cash and certain assets, including the Oakville Estates vineyard and certain related equipment, and Huneeus will contribute certain assets and liabilities, and to which DBR will, upon the completion of the merger contemplated by the Letter Agreement, cause the Issuer to contribute substantially all of its assets and liabilities. A copy of the Letter Agreement is annexed hereto as Exhibit 1 and incorporated herein by reference.

The Principals propose to effect the proposed transactions as follows: (i) DBR will incorporate a holding company (“Holdco”) and contribute $10 million in cash to Holdco, as described above, and contribute its shares of Common Stock (including any shares that may be obtained upon conversion of its 9% convertible subordinated promissory note due August 21, 2004) to Holdco in exchange for Holdco shares, (ii) the JV will loan to Holdco an amount equal to the difference between the aggregate cash amount to be paid in exchange for the outstanding publicly held Common Stock of the Issuer (other than the shares of Common Stock of the Issuer held by DBR, Holdco and their affiliates) in the merger described in (iv) below and the $10 million in cash to be contributed by DBR to Holdco; (iii) the source of funds for the loan by the JV will be $54 million in cash contributed by Constellation; (iv) the Principals will propose to the special committee of the board of directors of the Issuer that Holdco enter into a merger agreement (the “Merger Agreement”) with and merge into the Issuer (the “Merger”), in which all of the outstanding publicly held shares of the Issuer (other than the shares held by DBR, Holdco and their affiliates, which shares shall be cancelled) shall be converted into cash (such transactions, including the Merger, collectively, the “Transactions”). The Principals will jointly and severally guarantee the obligations of Holdco under the Merger

Agreement. The closing of the Transactions will occur simultaneously. The material terms of the Merger Agreement setting forth the terms and conditions of the Merger, including the cash price per share to be paid to holders of shares of Common Stock of the Issuer (other than DBR, Holdco or its affiliates) will require the approval of each of the Principals. The commitment of the Principals pursuant to the Letter Agreement is subject to the following conditions: the successful consummation of the Merger; and all requisite consents and antitrust law approvals being obtained.

The Principals have agreed not to take any action or enter into any agreement inconsistent with the Letter Agreement for a period of four months from the date of the Letter Agreement (the “Commitment Period”). During the Commitment Period, each of the Principals will act exclusively with the others with respect to the Merger and the other Transactions and will use reasonable efforts to cause the Merger and the other Transactions to be consummated in the manner contemplated.

Pursuant to the Letter Agreement, DBR has further agreed that, during the Commitment Period, it will not (i) at any shareholders meeting of the Issuer vote for any transactions proposed by a third party that would result in such third party owning more than 50% of the Common Stock of the Issuer or its successor or a transaction involving the sale of all or substantially all of the assets of the Issuer (each, a “Competing Transaction”), (ii) solicit any third party to propose a Competing Transaction or discuss any such proposal with any such third party, or (iii) sell its shares of Common Stock of the Issuer to any third party other than an affiliate of DBR. If DBR breaches the commitment described above, DBR will immediately reimburse Huneeus its

documented reasonable expenses incurred in connection with the Transactions. DBR has also agreed, pursuant to the Letter Agreement, to enter into a voting agreement upon the execution of the Merger Agreement whereby DBR agrees for a period commencing upon the date of the Merger Agreement and ending on the termination of the Merger Agreement (i) not to sell or otherwise transfer its shares of Common Stock of the Issuer, (ii) not to solicit proposals for Competing Transactions or enter into discussions with any third party with respect to a Competing Transaction, and (iii) to vote its shares of Common Stock of the Issuer at a shareholders meeting of the Issuer in favor of the transaction proposed by the Merger Agreement and against any Competing Transaction.

The Letter Agreement also provides that, if within six months following the date of the Merger Agreement, the Issuer enters into a definitive agreement for a Competing Transaction at a price per share higher than $7.80 (the “Reference Share Price”), DBR will within seven days of receipt of any amounts paid to it pursuant to the Competing Transaction, pay each of Constellation and Huneeus an amount equal to one-third of the difference between the aggregate consideration received by DBR pursuant to such Competing Transaction and an amount equal to the Reference Share Price multiplied by the number of shares of Common Stock of the Issuer held by DBR that are sold. In addition, in any circumstance in which the Issuer is obligated to pay a break-up fee upon termination of the Merger Agreement as a result of the Issuer entering into a Competing Transaction, the fee will be paid to and shared equally by Constellation and Huneeus. The Letter Agreement provides that none of the commitments and understandings agreed to by DBR will restrict any actions taken by any director of the Issuer who is an employee of or affiliated with DBR in furtherance of the director’s fiduciary obligations.

Each Principal has agreed to severally, but not jointly, indemnify and hold harmless each other Principal (each, an “Indemnified Principal”) against (x) JV Losses (as defined below) incurred by the Indemnified Principal (whether or not such JV Losses are occasioned by the acts or failures to act of the Indemnified Principal), in an amount that is proportionate to the Indemnifying Principal’s prospective equity percentage in the JV, and (y) Other Losses (as defined below) arising out of the conduct, omission or statement of the Indemnifying Principal.

The term “Losses” means any and all actual losses, liabilities, damages, judgments, settlements and expenses (including interest and penalties with respect thereto and reasonable attorneys’ fees and expenses and reasonable accountants’ fees and expenses incurred in the investigation or defense of any Proceeding (as defined below)) incurred by any Principal to which such Principal may become subject as a result of a claim of or Proceeding initiated by a third party, irrespective of whether such Losses have been incurred or suffered as a result of Proceedings initiated by a governmental entity or otherwise. The term “JV Losses” means any Losses, but only insofar as such Losses arise out of or are based upon the Letter Agreement or the making of the proposal with respect to the Transactions by the Principals to the board of directors of the Issuer, or the consummation of the Transactions (including the Merger) pursuant to the Letter Agreement; provided, however, that JV Losses shall not include any Losses arising out of or based upon conduct, omissions or statements described in clauses (A) through (E) of the definition of Other Losses, as defined below. The term “Other Losses” means any Losses incurred by a Principal which at any time arise out of or are based upon: (A) actions or discussions by another Principal inconsistent with the course of conduct or direction approved or consented to by the Principals; (B) the failure of another Principal to file, or timely file, any forms, reports, statements or other

documents required to be filed with the Securities and Exchange Commission (as applicable, an “SEC Report”); (C) any untrue statement or alleged untrue statement or omission or alleged omission contained in the SEC Report of another Principal, which statement or omission originated from such Principal; (D) any act or omission by such Principal which would constitute fraud, gross negligence or willful misconduct by another Principal; or (E) the violation of any obligation of such Principal under any agreement relating to the Transactions. The term “Proceeding” means any action, suit, litigation, arbitration, proceeding (including any civil, criminal, administrative, investigative or appellate proceeding and any informal proceeding), prosecution, contest, hearing, inquiry, inquest, audit, examination or investigation commenced, brought, conducted or heard by or before, or otherwise involving, any governmental entity.

Except for specified provisions of the Letter Agreement which are binding on the parties, the transactions contemplated by the Letter Agreement are subject to the performance of due diligence and the preparation, execution and delivery of definitive agreements, in each case satisfactory to each of the Principals, and any liability among the Principals shall attach only upon such execution and delivery of the Merger Agreement, in a form acceptable to the Principals, with the Issuer.

Any proposed change to the material terms of the proposals or transactions set forth in the Letter Agreement and the definitive agreements contemplated thereby will be subject to the unanimous approval of the Principals.

All legal expenses of the JV and each of the Principals incurred in connection with the Transactions and the other transactions contemplated by the Letter Agreement will be apportioned as provided in the Letter Agreement.

If the transactions contemplated by the Letter Agreement are completed, the Principals intend to operate the business of the Issuer as an integral part of the JV. To this end, immediately upon the consummation of the Merger, the Principals shall cause the Issuer to contribute all of its assets and liabilities to the JV. In addition, it is intended that DBR through its affiliates will enter into a business arrangement with the JV to develop the first estate in Napa Valley to produce wine under its guidance. In addition, the JV intends to offer a wine benefits program to all persons eligible for participation in the Issuer’s wine benefits program.

The Principals’ commitment in the Letter Agreement will expire on September 15, 2004 if the Merger Agreement has not been entered into prior to that date.

On May 17, 2004, DBR is delivering to the board of directors of the Issuer a letter (the “Proposal Letter”) to propose to the Issuer the business combination transaction contemplated by the Letter Agreement. The proposal contemplates that the Merger would be effected for cash consideration of $9.25 per share of Issuer Common Stock (other than Common Stock held by DBR). A copy of the Proposal Letter is annexed hereto as Exhibit 2 and incorporated herein by reference.

On May 17, 2004, Constellation is issuing a press release, a copy of which is annexed hereto as Exhibit 3 and incorporated herein by reference.

Although the foregoing represents the general range of activities presently contemplated by Constellation with respect to the Issuer, Constellation intends to continually review the Issuer’s business, affairs and financial position, as well as conditions in the securities market and general economic and industrial conditions. The possible activities of Constellation are subject to change at any time. Except as set forth above, Constellation has no present plans or intentions which would result in or relate to any of the transactions required to be described in subparagraphs (a) through (j) of Item 4 of Schedule 13D.

The foregoing actions may be taken by some or all of Constellation and the other Principals.

a. Constellation may be deemed to have formed a “group” within the meaning of Rule 13d-5(b)(l) under the Exchange Act as a result of having entered into the Letter Agreement. As a result, Constellation may be deemed to jointly be the beneficial owner of the shares of Common Stock owned by the other Principals. Based on information provided to Constellation by DBR, DBR owns 5,556,103 shares of Common Stock, representing approximately 46% of the outstanding Common Stock. Based on information provided to Constellation by Huneeus, Huneeus owns no shares of Common Stock.

As a result, to the knowledge of Constellation, members of the group (consisting of Constellation, DBR and Huneeus) beneficially own an aggregate of 5,556,103 shares of Common Stock, representing approximately 46% of the total number of shares of such class.

Constellation is not entitled to any rights as a shareholder of the Issuer with respect to Common Stock of the Issuer owned by the other Principals, and expressly disclaims that it has any beneficial ownership interest in any shares of Common Stock of the Issuer beneficially owned by either of the other Principals.

Constellation is filing this statement individually pursuant to Rule 13d-1(k)(2) under the Exchange Act and does not know or have reason to know any information concerning either of the other Principals other than the information set forth in the Letter Agreement or directly provided to Constellation by the respective party.

b. Constellation does not have the sole power to vote or to direct the vote nor the sole power to dispose or to direct the disposition of any shares of the Issuer. Constellation may be considered to share the power to vote or to direct the vote, or share the power to dispose or to direct the disposition of shares of the Issuer’s stock directly or beneficially owned by DBR or Huneeus.

c. Constellation has not effected any transactions in the Common Stock of the Issuer during the past 60 days. All transactions effected by DBR and Huneeus during the past 60 days should be contained in Schedules 13D, as amended, filed or to be filed by each of them.

d. Not applicable.

e. Not applicable.

Reference is made to Items 3 and 4 of this Schedule 13D for information with respect to any contracts, arrangements, understandings or relationships between Constellation and the other Principals and between Constellation or any person or entity with respect to securities of the Issuer. Except as set forth in Items 3 and 4 of this Schedule 13D, there are no contracts, arrangements, understandings or relationships (legal or otherwise), including, but not limited to, transfer or voting of any of the securities, finder’s fees, joint ventures, loan or option arrangements, puts or calls, guarantees of profits, division of profits or loss, or the giving or withholding of proxies, between Constellation and the other Principals and between Constellation or any person or entity with respect to securities of the Issuer.

The following are filed herewith as Exhibits to this Statement:

Signature

After reasonable inquiry and to the best of the undersigned’s knowledge and belief, the undersigned certifies that the information set forth in this statement is true, complete and correct.

Date: May 17, 2004

EXHIBIT INDEX