Smith explains why benevolence is desirable but justice is essential to how we measure our behavior in the eyes of others.

Justice and Beneficence

Adam Smith was a Scottish political philosopher and economist, considered one of the forefathers of classical economics and a pioneer of the study of political economy. Smith graduated from Balliol College at Oxford, and later served as the chair of moral philosophy at the University of Glasgow. He departed from his academic position after 12 years to tutor the Duke of Bucchleuch in Switzerland. His two major works, An Inquiry into the Nature and Causes of the Wealth of Nations, and The Theory of Moral Sentiments, were composed after he left the service of the Duke on a lifetime pension.
The former work, considered his magnum opus, is often referred to by its abridged title The Wealth of Nations. It was first published in 1776, the same year as the American Declaration of Independence, and is considered a foundational text in modern economic theory. It is noted for its influence on the American founding fathers James Madison, Alexander Hamilton, and Thomas Jefferson, and upon the economic theories of Karl Marx, John Maynard Keynes, and Milton Friedman.
The latter work expressed Smith's deep interest in moral philosophy, and expanded upon the philosophical, juridical and ethical framework of his earlier work. While known primarily as an economist today, Smith's work and interests lay primarily in the fields of theology, jurisprudence, and moral philosophy.

It is thus that man, who can subsist only in society, was fitted by nature to that situation for which he was made. All the members of human society stand in need of each others assistance, and are likewise exposed to mutual injuries. Where the necessary assistance is reciprocally afforded from love, from gratitude, from friendship, and esteem, the society flourishes and is happy. All the different members of it are bound together by the agreeable bands of love and affection, and are, as it were, drawn to one common centre of mutual good offices. But though the necessary assistance should not be afforded from such generous and disinterested motives, though among the different members of the society there should be no mutual love and affection, the society, though less happy and agreeable, will not necessarily be dissolved. Society may subsist among different men, as among different merchants, from a sense of its utility, without any mutual love or affection; and though no man in it should owe any obligation, or be bound in gratitude to any other, it may still be upheld by a mercenary exchange of good offices according to an agreed valuation. Society, however, cannot subsist among those who are at all times ready to hurt and injure one another. The moment that injury begins, the moment that mutual resentment and animosity take place, all the bands of it are broke asunder, and the different members of which it consisted are, as it were, dissipated and scattered abroad by the violence and opposition of their discordant affections. If there is any society among robbers and murderers, they must at least, according to the trite observation, abstain from robbing and murdering one another. Beneficence, therefore, is less essential to the existence of society than justice. Society may subsist, though not in the most comfortable state, without beneficence; but the prevalence of injustice must utterly destroy it. Though Nature, therefore, exhorts mankind to acts of beneficence, by the pleasing consciousness of deserved reward, she has not thought it necessary to guard and enforce the practice of it by the terrors of merited punishment in case it should be neglected. It is the ornament which embellishes, not the foundation which supports the building, and which it was, therefore, sufficient to recommend, but by no means necessary to impose. Justice, on the contrary, is the main pillar that upholds the whole edifice. . . . The principle by which we naturally either approve or disapprove of our own conduct, seems to be altogether the same with that by which we exercise the like judgments concerning the conduct of other people. We either approve or disapprove of the conduct of another man according as we feel that, when we bring his case home to ourselves, we either can or cannot entirely sympathize with the sentiments and motives which directed it. And, in the same manner, we either approve or disapprove of our own conduct, according as we feel that, when we place ourselves in the situation of another man, and view it, as it were, with his eyes and from his station, we either can or cannot entirely enter into and sympathize with the sentiments and motives which influenced it. We can never survey our own sentiments and motives, we can never form any judgment concerning them; unless we remove ourselves, as it were, from our own natural station, and endeavour to view them as at a certain distance from us. But we can do this in no other way than by endeavouring to view them with the eyes of other people, or as other people are likely to view them. Whatever judgment we can form concerning them, accordingly, must always bear some secret reference, either to what are, or to what, upon a certain condition, would be, or to what, we imagine, ought to be the judgment of others. We endeavour to examine our own conduct as we imagine any other fair and impartial spectator would examine it. If, upon placing ourselves in his situation, we thoroughly enter into all the passions and motives which influenced it, we approve of it, by sympathy with the approbation of this supposed equitable judge. If otherwise, we enter into his disapprobation, and condemn it. Were it possible that a human creature could grow up to manhood in some solitary place, without any communication with his own species, he could no more think of his own character, of the propriety or demerit of his own sentiments and conduct, of the beauty or deformity of his own mind, than of the beauty or deformity of his own face. All these are objects which he cannot easily see, which naturally he does not look at, and with regard to which he is provided with no mirror which can present them to his view. Bring him into society, and he is immediately provided with the mirror which he wanted before. It is placed in the countenance and behaviour of those he lives with, which always mark when they enter into, and when they disapprove of his sentiments; and it is here that he first views the propriety and impropriety of his own passions, the beauty and deformity of his own mind. . . . When I endeavour to examine my own conduct, when I endeavour to pass sentence upon it, and either to approve or condemn it, it is evident that, in all such cases, I divide myself, as it were, into two persons; and that I, the examiner and judge, represent a different character from that other I, the person whose conduct is examined into and judged of. The first is the spectator, whose sentiments with regard to my own conduct I endeavour to enter into, by placing myself in his situation, and by considering how it would appear to me, when seen from that particular point of view. The second is the agent, the person whom I properly call myself, and of whose conduct, under the character of a spectator, I was endeavouring to form some opinion. The first is the judge; the second the person judged of. But that the judge should, in every respect, be the same with the person judged of, is as impossible, as that the cause should, in every respect, be the same with the effect. To be amiable and to be meritorious; that is, to deserve love and to deserve reward, are the great characters of virtue; and to be odious and punishable, of vice. But all these characters have an immediate reference to the sentiments of others. Virtue is not said to be amiable, or to be meritorious, because it is the object of its own love, or of its own gratitude; but because it excites those sentiments in other men. The consciousness that it is the object of such favourable regards, is the source of that inward tranquillity and self-satisfaction with which it is naturally attended, as the suspicion of the contrary gives occasion to the torments of vice. What so great happiness as to be beloved, and to know that we deserve to be beloved? What so great misery as to be hated, and to know that we deserve to be hated? . . . Our sensibility to the feelings of others, so far from being inconsistent with the manhood of self-command, is the very principle upon which that manhood is founded. The very same principle or instinct which, in the misfortune of our neighbour, prompts us to compassionate his sorrow; in our own misfortune, prompts us to restrain the abject and miserable lamentations of our own sorrow. The same principle or instinct which, in his prosperity and success, prompts us to congratulate his joy; in our own prosperity and success, prompts us to restrain the levity and intemperance of our own joy. In both cases, the propriety of our own sentiments and feelings seems to be exactly in proportion to the vivacity and force with which we enter into and conceive his sentiments and feelings. The man of the most perfect virtue, the man whom we naturally love and revere the most, is he who joins, to the most perfect command of his own original and selfish feelings, the most exquisite sensibility both to the original and sympathetic feelings of others. The man who, to all the soft, the amiable, and the gentle virtues, joins all the great, the awful, and the respectable, must surely be the natural and proper object of our highest love and admiration. The person best fitted by nature for acquiring the former of those two sets of virtues, is likewise best fitted for acquiring the latter. The man who feels the most for the joys and sorrows of others, is best fitted for acquiring the most complete control of his own joys and sorrows. The man of the most exquisite humanity, is naturally the most capable of acquiring the highest degree of self-command.