Short answer: You probably got confused which value stand on which side during the comparison: https://www.khronos.org/registry/OpenGL-Refpages/gl4/html/glStencilFunc.xhtml
Have a look at this pseudo code for the stencil test:
maskedBufferValue = bufferValue & stencilFunctionMask;
maskedStencilRefValue = stencilRefValue & stencilFunctionMask;
if (NOT StencilCompFunc(maskedBufferValue, maskedStencilRefValue))
UpdateStencilBuffer(hasStencilTestPassed, hasDepthTestPassed, stencilMask);
So first we use the mask (3. value of glStencilFunc) with a bitwise AND operation on the value currently stored in the stencil buffer and on the specified reference value (2. value of glStencilFunc). If you set the mask to 0x00, both result values are always 0. If you set it to 0xFF it results in a plain assignment of the original values. So in your case it is not that important since you are only using 0xFF.
So lets walk through the original example:
First you draw the floor. StencilFunc at this point is: glStencilFunc(GL_NOTEQUAL, 1, 0xFF), since it is the last call to this function in the previous loop iteration. Stencil mask is 0x00 which disables writing to the stencil buffer. So assuming that you cleared the stencil buffer correctly (see pitfall at the end), the following happens here:
Since the stencil function mask (don’t get confused with the stencil write mask) is 0xFF we take the unmodified values for our comparison. It is the first draw after clearing. The buffer values are 0 everywhere. So you are always comparing 0!=1 which results in true. Only if this function would have yielded a false, the fragment would have been discarded. So as a result, everything passes and is drawn. The stencil buffer is not updated, since the stencil mask is 0x00.
Now we get to the containers. We use glStencilFunc(GL_ALWAYS, 1, 0xFF) and glStencilMask(0xFF). The stencil comparison will always yield true and the values are not modified. So we get straight to the buffer update. Since the stencil test passed there are now two possible results: The depth test fails and you will do nothing since you specified GL_KEEP as second param of glStencilOp or the depth passes and the 3. action specified by glStencilOp is used for the update which is GL_REPLACE. This states that the reference Value of glStencilFunc replaces all the bits specified by stencil mask in the current buffer. In this case, all bits are replaced (0xFF). The result is, that every fragment which belongs to a container and passes the depth test has now a stencil value of 1, while the rest is zero.
Last we get to the scaled containers glStencilFunc(GL_NOTEQUAL, 1, 0xFF), glStencilMask(0x00). So a stencil passes if bufferValue != 1. This is everywhere except for the location where the original container was rendered and the fragment passed the depth test. This results in an outline since the containers are scaled up and you can only draw at positions that are not part of the original container. The stencil buffer is not updated.
Now what happens, if we set GL_LESS instead of GL_NOTEQUAL?
Floor: glStencilFunc(GL_LESS, 1, 0xFF) — All values in the buffer are 0 and the refValue is 1. Values after the mask are the same. 1 < 0 always fails. This is why your floor is never drawn.
Containers: Same as before.
Scaled Containers: glStencilFunc(GL_LESS, 1, 0xFF). Two possible cases for the stencil test: 0 < 0 (fail) or 1<0(fail). Scaled container fragments can never pass the stencil test!
Hope that helps understanding why you get the described result.
PITFALL: When I was implementing the stencil stuff I searched half of the day for an error in my code which was caused by a missing glStencilMask(0xFF) before clearing the stencil buffer. The stencil mask seem to affect the clear command too. So if it is set to 0x00 a call to glClear(GL_STENCIL_BUFFER_BIT) has no effect. Keep that in mind, it might save you the same trouble I had — I asked myself, if this is a bug, but if you think about it makes sense. This can be used to clear only specific bits from the stencil buffer allowing you to do really crazy things.