SAN FRANCISCO — A federal judge on Tuesday rejected a proposed settlement between Google Inc. and book authors and publishers as unfair and inadequate, dealing a new setback to the Internet giant’s ongoing effort to push its digital-books business forward.

Judge Denny Chin wrote in an opinion filed in U.S. District Court for the Southern District of New York that the agreement “would simply go too far” in terms of allowing Google to establish a business arrangement that grants it the right to “exploit entire books, without permission of the copyright owners.”

Google began a controversial effort to scan vast quantities of books in 2004, the year that the company went public. Google has argued that its efforts can expand access to otherwise difficult-to-access volumes.

Plaintiffs including the Authors Guild and the Association of American Publishers sued Google in 2005, complaining that the company’s digital scanning violated copyright protections.

According to its proposed settlement agreement, Google would gain the right to provide copyrighted works on its books service, apart from those actively withheld by copyright owners.

In his ruling on Tuesday, Judge Chin wrote that his concerns about the agreement could be eased by converting it to an “opt-in” arrangement for copyright owners, rather than “opt-out.” He urged the parties to consider revising the agreement.

Hilary Ware, Google’s managing counsel, said in a written statement, “This is clearly disappointing, but we’ll review the court’s decision and consider our options.”

“Like many others, we believe this agreement has the potential to open up access to millions of books that are currently hard to find in the U.S. today,” Ware said. “Regardless of the outcome, we’ll continue to work to make more of the world’s books discoverable online through Google Books and Google eBooks.”

Google’s eBooks service is designed to enable users to shop for and buy digital books from Google and other retailers, and to get access to them on multiple devices.

The Open Book Alliance, which includes Google rivals Microsoft Corp. and Amazon.com Inc., has argued that the proposed settlement raises too many concerns, and had urged Judge Chin to reject it.

Gary Reback, who is counsel to the Open Book Alliance, said Tuesday that Judge Chin’s opinion includes “everything we were asking for.”

In particular, Reback praised Chin’s tying of Google’s books services to its market-dominating search engine. The proposed settlement agreement “would arguably give Google control over the search market,” Chin wrote.

Reback said that the antitrust, privacy and approach to copyright ownership cited by Chin are the “three big issues Google is facing everywhere now.”

“This mirrors the bigger Google question,” he added.

Google has faced increasing scrutiny from regulators and rivals related to its acquisitions, collection of private data and use of copyrighted material.

Google had revised its settlement agreement with authors and publishers, reached in 2008, the following year. However, the Justice Department criticized the planned arrangement, saying in a filing that it could lend Google “significant and possibly anticompetitive advantages.”