A vortex engine is a new concept for building regeneratively cooled rocket engines; specifically the combustion chamber.
Normal regeneratively cooled engines have a jacket through which the fuel passes on the way to the rocket injectors. This sucks heat from the interior lining and enables the interior surface to be kept below its melting point, when the hot gases only a millimeter away are at perhaps 3000 C (5000F).
This works great, and the upside is that it in fact it improves performance by one or two percent, because the heat is carried back into the combustion chamber, rather than radiating away uselessly (as in radiatively cooled engines). The downside is that the double jacket adds weight, and is fiddly/expensive to construct.
Instead the new idea is to add one of the fuels (typically LOX) into the combustion chamber around the throat of the engine, around where the exhaust leaves the combustion chamber. Normally that would be silly, because it would get sucked out and fail to combust. The trick is to add a huge swirling motion to it by injecting the fuel in tangentally at extremely high speed.
This means the centrifugal force prevents it from leaving, and in fact it spirals up the combustion chamber to the other end, where it finally mixes with the other fuel. The fuel mixes well because of the high relative velocities and the high 'shear'. The fuel then combusts and is kept in the center of the engine by the fact that it is hotter and lighter than the cold gas around it, and passes back along the axis of the engine and leaves through the throat and expands out the nozzle.
In fact, you don't even need to inject it particularly fast- there's a powerful tornado effect caused by the hot gas spiralling out of the nozzle that speeds up the cold gas as well. And the cold gas spiralling up the wall actually keeps the wall cool from the inside. The first demo of this actually had a plastic combustion chamber that did not melt(!), with hot gas leaving at about 3000C! Outstanding!
However due to the extremely high speeds that are reached at the throat with any rocket engine, this technique does not protect the throat or the nozzle region of the rocket engine, and you still have to deal with temperature issues there.