Is it only when parties sign up to a written agreement?

In its simplest form this would make sense and is easily measurable. However for me there is a problem with this as a sign of success

Having a background in Quality, it was drummed into me that before you can compare data, you need to make sure, that what you are using to produce the result, i.e. the equipment is calibrated the same, so that you are not comparing “apples with oranges.”
Great when looking at machinery, but here we are talking about People.

Not one person is calibrated the same. We all act differently, with different needs and wants. We have different background, and cultural influences. The list of differences between each of us as human beings is immense.

Obtaining the same outcome, with so many different variables is going to be difficult, and we have to accept that results may vary.
So if we are only looking at a measurable success for mediation as being a written an agreement, then we are in danger of identifying other successes.
Bringing people or parties together to discuss and communicate with each other is a primary goal for me, if during the mediation process I can encourage positive communication and listening, then I count this as a success.

Recently I had a case that involved a tenant and a landlord. An ongoing dispute had been taking place for approx. one year with no end in sight, and a total breakdown in communication.

Eventually mediation was advised, and the parties agreed to meet. During the session, both parties were able to identify that, perhaps they did not fully understand the complexity of the situation, regarding both their contractual obligations, and the emotional impact the situation was having.
Whilst both parties came to the table wanting a resolution, the parties were unable to agree on a way forward, and no written agreement was reached, however both parties had a better understanding of each other, and respectful dialog has been re-instated. They still have work to do, but both parties were thankful of the mediation process for allowing them to talk in a safe environment and to obtain clarity of their respective situations.

Ultimately the sharing of their frustrations at the situation, allowed them to move forward and away from the negative situation they had been in.

People and parties moving forward in a positive manner, Is a success in my eyes every time.