He was censured for being too cautious early in the war
Shortly after the beginning of hostilities in the Civil War (he had suffered a minor defeat in battle) Lee was sent to defend the city of Savannah and the coastlines of Georgia and South Carolina. The Confederates at the time had no Navy to oppose the United States fleet, and the existing defenses were those which had been built by the federal government to defend the US coastline from foreign enemies.
These centered around Fort Pulaski, on an island in the mouth of the Savannah River. Lee recognized that the use of Savannah as a port was dependent on Fort Pulaski and that the fort could not be held against a determined siege, but that the city could be kept out of Union hands with additional defenses.
Lee strengthened the existing Fort Jackson, about two miles from Savannah, and built supporting batteries and trenches to defend it from attack. Lee’s defenses were so strong that the Union could not advance upon the city and troops defending Savannah were detached and sent north to aid in the defense of Richmond, Virginia. When Fort Pulaski fell to the Union they were still unable to press forward to take Savannah.
Lee was detached and sent to Richmond as a military advisor to President Jefferson Davis, whom he had known during the Mexican War. In Richmond, Lee built a series of trenches and redoubts for the defense of the city which weren’t needed immediately, but which proved useful when Grant’s troops arrived before the city in 1864.
Because of his cautious approach to his duties in the early days of the war, and in part because of the fall of Fort Pulaski, the southern press took to calling him “Granny Lee”. The Richmond newspapers, eyewitnesses to the digging of fortifications all around the city which then went unused, added the sobriquet, “King of Spades.”
As the Union Army approached Richmond in the spring of 1862 the Army of Virginia was pushed back and its commander, Joseph Johnston was wounded. Davis selected Lee to replace Johnston in command, to the howls of indignation from the southern press. Even as Lee renamed his army to the Army of Northern Virginia, a clear indication that he intended to move it closer to Washington, the press believed that he was the wrong man to command it.