Welcome to ModelEngineMaker !If you have problems registering or logging in, please use the contact menu option to request assistance.
If I've understood correctly an 8mm cutter is being used to create an internal 4mm radius. If one thinks about the finish cut (small DOC) the cutter engagement on the flat sides is small, equals small forces and small deflection. When the cutter reaches the corner the cutter engagement suddenly increases to around a quarter of the cutter circumference which inevitably leads to chatter and cutter deflection. In the CNC world it is a no-no to use a cutter of radius X to form an internal corner of radius X, a smaller diameter cutter is used.In the manual world one solution is to use multiple stepdowns, although there might still be some chatter. It would be better to use a short 8mm cutter for the first step down or two and then a long series cutter, but with short flutes. That way the plain upper part of the cutter can be guided by the existing radius. A carbide cutter may help as they are stiffer than HSS.It's not difficult to create square internal corners with a set of decent files. Just don't use a square file, flat and three-square files are better.Andrew