Basically: if your target number for a skill test is too low, the odds of success, even with extra dice bought, becomes too small. At 4, you’ve only got a little over 1 in 3 chance of passing a difficulty 1 test with the base two dice.
So, we set a hard minimum of 4 for PCs, partly to avoid those low numbers and partly to curb extreme min-maxing.