Why the file goes further than the gap
Hold a string down at the third fret and it becomes a straight line from the nut slot to the top of the third fret. The first fret is under that line, and the gap you measure over it is the distance between the string and the fret top.
Now look at what that gap is a fraction of. The slot floor sits some height above the plane of the fret tops. The string starts there and comes down to zero at the third fret, so at the first fret it has already used up part of its descent. The remaining height is the slot height multiplied by one minus the ratio of the two fret distances.
On a 25.5 inch scale the first fret is 1.4312 inches from the nut and the third is 4.0571. The ratio is 0.3528, so the string at the first fret sits at 0.6472 of the slot height. Turn that around: every thousandth you take out of the slot closes the gap by 0.647 thousandths, and closing the gap by one thousandth needs 1.545 out of the slot.
That factor is why nut work sneaks up on people from the other side compared with saddle work. At the saddle you move twice as much as you meant to. At the nut you move about one and a half times as much, in the same direction. Both are levers, both are exact, and both are the difference between two goes at it and six.
Holding at the second fret instead of the third
The second fret is at 2.7821 inches on a 25.5 inch scale, so the ratio becomes 0.5144 and the string sits at 0.486 of the slot height at the first fret. The lever goes up to 2.06. That makes the second-fret method roughly twice as sensitive as the third-fret method: the same slot height shows up as a smaller gap, so small gaps become hard to read and a feeler gauge that fits tells you less.
Neither method is more correct. What matters is that a gap figure means nothing until you say which fret was held. Somebody quoting a first-fret gap of two thousandths held at the third and somebody quoting two thousandths held at the second are describing nuts that differ by about a third in slot height. Write down both frets every time.
What a high nut actually does
It makes the first position hard to play and it makes it play sharp. Hard to play is obvious: the further the string has to travel to reach the first fret, the more force it takes. Sharp is the interesting half. Pressing a string stretches it, the stretch raises its pitch, and the amount of stretch grows with how far the string has to move. So a nut standing high pushes the first few fretted notes sharp while the open strings are perfectly in tune, and no amount of work at the saddle addresses it, because saddle compensation is set from the twelfth fret where the error has largely gone.
The diagnostic is the same measurement this page is built around. Fret at the third, look at the first, and if the gap is large the nut is a candidate. A tuner will show the same thing from the other direction: play the open string, then the note at the first fret, and see how far apart they land.
Depth, width and the part this page does not do
Slot depth is only one of the three things a nut slot has to get right. Width matters too: a slot cut for a light gauge pinches a heavier string, which then binds instead of sliding and comes back out of tune every time it is bent. A slot cut too wide lets the string rattle sideways in it. And the front-to-back shape matters, because the string should leave the nut at a single defined point at the fretboard side of the slot rather than sitting on a long flat.
None of those are arithmetic and none of them are on this page. What is here is the one part that is geometry: how far down, given how far you want the gap to change.
Where this connects
The nut controls the low frets and the saddle controls everything from about the fifth up, which is the string action and saddle height calculator. The bow between the two is the neck relief calculator, and it should be settled before either end is touched. Slot width follows the string gauges, which are also what the string tension calculator works from, and where the slots sit across the nut is the string spacing layout calculator. The fret distances every figure here is measured to come from the fret position calculator.
Questions people ask
How do I measure the gap at the first fret?
Hold the string down at the third fret with one finger, close to the fret, and slide feeler gauges between the top of the first fret and the underside of the string until one just touches without lifting the string. Read that gauge. Do it with the instrument tuned to pitch and in playing position, and do each string separately, because the low strings and the high strings almost never come out the same. Note which fret you held at, because holding at the second gives a different number off the same nut.
Why does the slot have to come down more than the gap changes?
Because the first fret is not at the far end of the lever. The string runs from the slot floor down to the top of the fret you are holding at, so by the time it passes over the first fret it has already lost part of its height. On a 25.5 inch scale holding at the third fret, the first fret sees about 0.647 of whatever the slot is standing proud by. Reverse it and every thousandth of gap you want to close needs about 1.545 thousandths out of the slot. Cut only what the gap says and you will be back at it.
What happens if I cut a slot too deep?
The open string buzzes against the first fret, and there is no way to put material back into the slot. The repairs are all worse than the mistake: a shim in the bottom of the slot, which is fiddly and can move; filling the slot and recutting it; or a new nut. This is why the sensible method is to take a fraction of the calculated amount, put the string back to pitch, measure again, and repeat. The calculation tells you where you are going, not how fast to get there.
Can a high nut make an instrument play sharp?
Yes, in the first few frets specifically, and it is one of the most common causes of an instrument that tunes perfectly open and sounds wrong in first position. Pressing a string down stretches it and the stretch raises the pitch, and how much it stretches depends on how far it has to travel. A high nut means a long travel at the first few frets and almost none higher up, so the error is concentrated exactly where chords live. Saddle compensation cannot correct it, because compensation is set from the twelfth fret where that error has already faded.
Should the slot depth be the same on every string?
The heights above the frets generally are not the same, and the slot depths measured from the top of the nut certainly are not, because the strings differ in diameter. A wound low string vibrates in a wider arc than a plain treble string, which is why the gaps people work to usually taper across the set. What those gaps should be is a decision about the instrument and the player, and this page takes both the measured and the wanted figure as inputs rather than proposing either. The columns marked slot floor are heights above the fret plane, not depths from the top of the nut, which depend on how tall the blank is.