Most countries in the world pursue a private land system, while China alone insists on the public ownership of land with collective ownership and household contract as the core. This paper borrows the concepts and tools of the New Institutional Economics, discards the traditional public-private antagonism of the land system and regards the collective and the family as two separate systems when discussing the rationality of the Land Tenure System in China on the basis of regarding. In terms of the stability of the system, both the collective and the family are the natural selection of history. The provisions of the law only make the original informal system manifest as a formal institutional arrangement. In addition, with the support of the informal system, the land system is more likely to form an efficient institutional structure. As far as the collective is concerned, informal institutional factors like customs, culture, trust, and public opinions affect the decision-making behavior and behavioral performance of farmers in various fields, which is related to the realization of collective action or the management of public affairs. As far as the family is concerned, they share the same goal and communicate more conveniently. Aside from these advantages, they play greater roles in different labor positions and work more effectively, reducing the management cost of the system, including negotiation costs, information costs and supervision costs, etc. In terms of the flexibility of the system, both the collective and the family have good internal adjustment functions, and they are able to adapt to the changes in the land system. As far as the collective is concerned, the collective organization, as the agent of the peasant collective, has always embodied the dual principal-agent relationship, exercising different functions in different periods. This agent has successfully handled the relationships between small farmers and the government, and between small farmers and the market, saving transaction costs at different levels. As far as the family is concerned, the transformation of agricultural operation subjects from contracted farmers to new agricultural operation subjects indicates that the design of ″separation of three rights″ is conducive to improving the efficiency of farmland allocation without breaking the original interest pattern. In addition to helping deepen the understanding of Chinese and foreign scholars on China’s Land Tenure System, this study also has important theoretical values and policy implications. The theoretical values are as follows: (1) Organization is also a system. Different organizational forms embody different institutional arrangements. The analysis of the organization should account for the system’s classic definition and the basic functions of the system. (2) China’s practices indicate that the issue of land system is not only one of ownership, but also one of the principal-agent. In the future, the principal-agent theory may find better application in the field of land system. (3) It is a reasonable option to establish a formal system alongside the informal system. The unreasonable elements in the informal system should also be eliminated through the establishment of a formal system. The policy implications are as follows. First, it is necessary to clarify that the villagers’ group should be the main body of the collective land property ownership and to explore the effective realization of family management under collective ownership. Second, attention must be paid to the functional coordination between the formal system and the informal system, especially the function of the informal system. Third, within the framework of collective ownership, the relative functions of collective organizations may be selectively dormant or alive in order to deepen the reform of the ″separation of three rights″ of agricultural land on the basis of family management.