Anchor
General General

General concept of synchronisation

Time synchronisation is not absolutely necessary for Protection function related GOOSE/MMS - the system will work with all IEDs running independent clocks. However for us humans to know when things happened, we need synchronised time stamps.  That is logical as conventional protection relays have worked for decades without knowing what time it was.

However if you are using Sampled Values (IEC 61850-9-2), THEN you MUST have better than 1microsecond COHERENCY of time synch across all Merging Units so you really MUST use IEEE 1588 v2 PTP. 
Note IEC 61850-9-2LE actually stipulated 1-PPS via optical fibre but in reality that is nowhere near good enough - basically it was just that was the best they could find when it was written! It is interesting that some "LE" MUs use RJ45 connections or coax despite the fibre stipulation!
IEEE 1588 works over the same LAN as the GOOSE/SV/MMS but you need special switches (and of course MU's) that support PTP.

As far as SV is concerned, it doesn't matter how the relay is synchronised as they use the SV Sequence Number to time align the SV messages based on the counters all starting from 1 within  microsecond of each other and reset and start all over again at the start of the next 1-second window.

I emphasise COHERENCY of time synch as SV doesn't actually need to know the actual time - just that every MU starts its 1-second window of sequence counter at exactly the same instant. Consider MU1 starting its Sample counter at 1 but MU2 doesn't start its counter at 1 until 0.5 ms later.  You will note that SV don't include a time stamp in the message - just the SV Counter with respect to the current 1-second window so two messages with "SeqNbr = 2365" are known to have been taken at precisely the same instant - within 1 microsecond coherency.

Time synchronisation for Sampled Values is further discussed on this page: What does time synchronisation mean for Sampled Values?

PTP involves selection of different types of clocks and different modes within those clocks - clocks being required to be part of ALL the LAN switches to make it work.

• Grand Master Clock
• Boundary Clock
• Transparent Clock
• Ordinary Clock

Anchor
GOOSE GOOSE

The meaning of time and establishing Sequence of Events in GOOSE

So that brings us to the meaning of time itself in say a GOOSE message.

Interestingly, or annoyingly (!!), time is not necessarily real time …. “it depends …”

Consider the network below where two GOOSE are issued by two publishers for the exact same event - let’s say an opto input being energised on each IED from the same external contact (the one scenario I can concede to the use of GGIO … although a "good vendor's tool" would allow me to rename GGIO to a more suitable specific semantic for my specific application - but that is a whole other discussion Say No to GGIO ).  For the moment lets just assume that the internal processing time of detecting the opto input state change to publishing the GOOSE message on first fast retransmit are exactly the same for both IEDs.

Let’s assume each GOOSE has a dataset of precisely one, and only one, bit being the on/off status of the GGIO i.e. that would be <<GGIO.Ind1.stVal>>.

When the GOOSE is created to be published on the LAN, each IED will add a time stamp <<T>> in the GOOSE PDU “header” (refer IEC 61850-8-1 Ed2 Table A.1) . 

Which leads us to IEC 61850-7-2 (Ed2), clause 18.2.3.5 T – time stamp
"The parameter T shall contain the time at which the attribute StNum was incremented."
Which also means when SqNum was reset to "1".

We might have some confidence of time synch amongst the publishers and the subscriber, so we could assume that the subscriber will therefore know when each message was created by each Publisher and then use that <<T>> to reconstitute the real sequence of messages being published.

However in reality, at best, all that <<T>> gives us is some general idea of the latency of the LAN from creation of the GOOSE PDU to the reception by the subscriber, … not completely the full definition of GOOSE "3 ms" latency", but very close to it. 
Certainly is not the basis of the Sequence of Events as when they happened according to the Publisher IED ... you could go looking for the time stamp of the message when the status of that element first changed.

Furthermore, that time stamp is updated each time a dataset element changes - we do not easily know which element that relates to.  If we wanted to know when a particular element in the dataset changed status we would have to go back through all the previous messages received (if we have captured them in some way) to pick the one when our particular "element of interest" changed so we can see the time stamp!!

So what does the subscribing IED/sniffer really know about time?

It will know when it received each message .. but that is all it can be guaranteed to know.  In fact any subscribing IED can only work on when it receives the first change of state of that element so the Subscriber's Sequence Of Events is really based on reception time of the message.

The difference in the receiving time will be clear, but the actual time of reception is dependent on the time synchronisation of the Subscriber with the rest of the network IEDs and/or a clock master.

And in fact, the two messages from the two different Publishers will arrive sequentially because that is the nature of Ethernet as a serial process of one message after the other, i.e. different times despite the trigger of the opto energisation being identical.

… and further, depending on where in the network the subscriber is connected, one message will arrive before the other, whilst connecting somewhere else in the LAN they may arrive in the other reverse sequence, or at least with different time differentials between them.

.. or worse, consider an RSTP ring network.  So measuring the latency of a publisher to subscriber is dependant on the number of intervening switches and links.  The virtual break point of the ring can change position in the ring when thing s"happen" that forces a ring re-convergence.  Once that happens, the same publish/subscribing pair of IEDs will see a different number of switches and links between them so the latency will almost inevitably be different.

So we are left with a very useful but subjective view of when things actually happened “at source” as a Sequence of Events.  We can derive a “Sequence of Arrival” such as provided by the FMTP GridEX "GOOSE Multimeter" "Multimeter" for GOOSE and Sampled Values:

The only way to know when things happened “at source” when the publisher IED had a function status chage is that each GOOSE dataset is formed including the <<.t>> Data Attribute of the respective Logical Node’s Data Object and Data Attributes  … in this example case this means <<GGIO.Ind1.t>> or in other words, the dataset should include three elements of <<GGIO.Ind1.stVal>>, <<GGIO.Ind1.q>> and <<GGIO.Ind1.t>> as this is the time of the last change of either the <<.stVal>> or <<.q>> Data Attributes.

Perhaps part of the reason why the <<.t>> Data Attribute is sometimes (often) not included in many datasets is that, as I first mentioned above, it is technically not required for the subscribing IED function in any way - GOOSE subscription works quite happily without knowing the actual date/hour/minute/second/millisecond.  Wire based contact output - opto input systems don't know when the function that caused the contact output to change actually decided the function had changed state ... the receiving relay only knows what time the opto input was turned on/off.
The second reason is perhaps a concern about bandwidth - the <<.t>> Data attribute will add a significant number of extra bits to the length of the dataset and hence of the GOOSE message - particularly of you have say 10 different elements each with their own last change <<.t>> Data Attribute. My thoughts on bandwidth are that it is becoming less relevant - when the Standard first started to be developed, LAN bandwidths were commonly 10 or 100 Mbit/s, but these days, IEDs generally have at least 100 Mbit/s on their own ports and the LAN backbones generally support 1 Gbit/sec or even 10 Gbit/sec
.. and then we can challenge the whole misconception of "GOOSE flooding" which, as I discuss here: GOOSE "Flooding", doesn't really cause a problem. 
The maximum size Ethernet frame is 1542 Bytes i.e. 12336 bits, hence even a 100 Mbit/s link can handle AT LEAST 8106 messages per second even if they were all that maximum size frame with a LOT of elements in the dataset.  If each message is say only a few hundred bytes as even larger size GOOSE are likely to be, the number of GOOSE messages per second allowable on that link will be several/tens/hundreds of thousands of messages per second!

Finally we need to consider what each IED considers as the "current time".  We have all seen those war movies where an attack is to happen "precisely at 11 o'clock" so the last thing the team leader says before the team breaks up is "synchronise our watches - the time now is ...".  The reason that is the last thing to do is that the longer two free running clocks are able to drift from each other, the bigger the amount of drift, especially of one clock is not all that stable with high drift rate.  So it is with multiple IEDs on the network. If they each have their own free running clock:

So Sub1 would see the two messages arrive based on its own time stamp, but the <<T>> of the PDU header from Pub1 and its <<.t>> in its dataset will be both VERY different to the <<T>> and <<.t>> in the message from Pub2.
Hence in considering what time things happen, we first have to ask "do all IEDs have the same understanding (synchronisation) of what the current time is"!
The only "absolute" is that Sub1 knows the time it received each of the messages based on its own time clock.

So what does this mean when we want to do some debugging of GOOSE in FAT/SAT/Commissioning/Maintenance where technicians and engineers are trying to sort out why something happened at the GOOSE level – when did a publisher IED see an event vs another IED vs when did a subscriber know about it from receiving GOOSE??? This then comes down to how well the dataset has been constructed by the IED Vendor (if it is fixed dataset) or by the Systems Integrator or by the Tester.  If well-constructed, you can analyse the datasets to extract the <<.t>> Data Attribute time stamps as the basis for Sequence of Events .. of course assuming all IEDs are appropriately time synchronised.

Anchor
SV SV

Time COHERENCY for Sampled Values

all MUs must be precisely synchronised to start each 1-second window within 1 microsecond of each other, i.e. 1 microsecond COHERENCY. 

We don’t care even what year it is saying the date/time is but we MUST start each 1-second window with 1 microsecond COHERENCY.
…. otherwise the samples appear all skewed

Consider a current differential application using two Merging Units:

Other than for transformer differential applications, clearly the current flowing through the red sensor (be it conventional or non conventional instrument transformer) is the same as the current flowing through the blue sensor.  
Therefore the relay should see the exact same waveform presented by each Merging Unit.

The samples captured by each Merging Unit at each instant are contained in the Sampled Values dataset being published by each Merging Unit. 
At each instant of sampling, the merging Unit tags the samples with a counter <<seqNum>> which represents the "nth" sample within the 1-second sampling window, i..e at a sampling rate of 4000 Hz, the counter will increment from 1 to a maximum of 4000 every 0.025 milliseconds.  At the start of the next 1-second window, the value of <<seqNum>> resets to a value of 1, and starts incrementing again.

No error

In the example above each Merging Unit has its own independent clock and therefore there is no inherent time synchronisation between the two clocks, hence their 1-second windows will start at a different instant in real time.

A common time synchronisation mechanism in substations for SCADA Sequence Of Events log purposes has been the IRIG-B or 1 Pulse Per Second signal, generally distributed over coaxial cable. 

The overall length of the coaxial cable path means there is a real time difference between the synchronisation of the first IED and that of the last IED in the loop due to the path latency. In general this means their is a spread of time across the IEDs but will generally within about one millisecond accuracy time stamp.

We can now consider what effect a one, five and 10 millisecond difference in the start of the 1-second window means to differential relays in particular, or indeed any relay which is using two or more sets of Sampled Value streams: 

Use case 50 Hz, 4000 Hz sampling,  DC offset at fault inception.

Four results are provided:

1. one microsecond skew of 1-second window start
2. one millisecond skew of 1-second window start
3. 5.54 millisecond skew of 1-second window start
4. 10 millisecond skew of 1-second window start

This shows that COHERENCY of time synchronisation of the Merging Units is critical, even if they don't have the same actual time, they just need to start their respective 1-second windows at the same instant, or at worst, within one microsecond of each other.

In theory you could use IRIG-B (U.S. military's Inter-Range Instrumentation Group, time code B) or 1-Pulse-Per-Second  (1PPS) usually over coax. 

IEC 61850‑9‑2LE recommended 1PPS over fibre rather than NTP or SNTP, 

Even so most MU suppliers only provided IRIG-B over coaxial cable at least initially as that is how most relays were being time synchronised for Sequence OF Events.
IRIG-B/1PPS will only give you at best 0.1 millisecond coherency … but in a small network with only a few switches and low bandwidth utilisation, you might be lucky.

The “proper” synch method for Sampled Values is IEEE 1588 (2008) PTP v2 distributed over the same LAN as the SV, GOOSE and MMS.  
However you must have a Master or Boundary Clock, and all the switches must support Transparent clock configuration and the IEDs of course must support operating as IEEE 1588 slave clocks.

Note within the one substation, it doesn't matte of the Master clock is showing precise actual time, ... just that all devices within the sub are synchronised to start their one second window at precisely the same instant within one microsecond of each other.
If you have substations at different locations relying on time synchronisation, you would then need GPS at both to synchronise the master Clock in each (technically the two clocks in each sub would now be called Boundary clocks as the GPS signal is the Master clock.

IEEE 1588 has a number of implementation profiles. 
There is a specific implementation for IEC 61850 based systems described in IEC 61850-9-3, first published in 2016.

Protection Operation Forensics Based On Time

Anyone who has tried to provide forensic analysis of this knows that the more information recorded about what happened and when (with respect to each other), the better. 

A common problem for many asset owners seeking to know why their circuit breaker tripped is the lack or somewhat limited Sequence Of Events records and/or waveform capture.   

My saying is:

You have no right to expect to get something if you do not remember to specify that you MUST have it.

The typical contractor excuse of not recording all signals is that individual wire‑based systems would require a 10‑fold or more increase in the number of wires and associated terminations to draw, install, and commission as well as an I/O count limitation on the relays and recording units in the first place.  That equals $$$$ which may make them uncompetitive and/or the asset owners budget just doesn't stretch that far in consideration that such records will hopefully only be needed "once in a blue moon"!!

Enter the virtual world of IEC 61850. 
No physical issues limiting recording all signals as traffic flow .. well, apart from having a suitable traffic recording unit with appropriate triggers and storage size.. 👍👌

But first – what is the meaning of time (stamps) of GOOSE. 

Actually, the normal Ethernet message time stamp included in all frames is meaningless.  The Ethernet GOOSE frame has a time stamp <<T>> of when it was sent.  But GOOSE has the heartbeat and fast repetition so for any one change of status, there will be a continuous stream of repeat messages with that same status all with different <<T>>.

The only REAL time as far as the various functions in the system is concerned is the time the first change of state message is RECEIVED by the subscribing IED.  That message may not be the first of the repeat messages series if there has been for example an RSTP reconvergence.

GOOSE is intended to be FAST so message frames were stripped down to bare minimum Ethernet length.  Usually they contain several status elements as 1 or 2 bits each as Pxxx.op.stVal plus their associated 8 bits quality flag Pxxx.Op.q.  So in principle there is no problem to have say “100” elements representing all stages of protection function operations.
e.g. a 4‑stage overcurrent and 4‑stage earth fault has at least eight PTOC.Op.stVal and eight PTOC.Op.q values in its dataset representing when the relay operated.  Add however many more Pxxx Logical Nodes are in the IED.

However equally the ONLY element that may be needed in the dataset is the "grouped" signal PTRC.Op.stVal and PTRC.Op.q in order to keep the "trip" GOOSE as FAST AS POSSIBLE with the smallest possible dataset and hence bandwidth requirement on the LAN.

But of course protection functions and their forensics often need to know what something simply STARTED to operate – it was above threshold and started timing out, but not yet resulting in a Pxxx.Op.stVal change of state.  We need to know the value of Pxxx.Str.stVal changed state.  So our 4-stage OC and 4-stage EF functions now have 16 status elements alone.

And not to forget that the protection forensics may also need to know when the quality flag Pxxx.Op.q (which has 12 bits itself) of the protection function changed state.  

• Pxxx1.Op.stVal 1 bit
• Pxxx1.Op.q 12 bits
• Pxxx1.Str.stVal 1 bits
• Pxxx1.Str.q 12 bit

26 bits per Pxxx Logical Node. Hence just for 4-stage OC and 4-stage EF requires 204 bits in the dataset t be sent on the wire.  If there was 100 Pxxx elements to be transmitted, it would be 2600 bits in the dataset plus of course the ethernet frame header and footer.

You might say "So what? - the LAN is at least 100 Mb/s, perhaps 1 Gb/s on the backbone"  
True.
GOOSE bandwidth even after a change of state is minimal - it does not "flood" the network as analysed here: GOOSE "Flooding".  
A fast retransmission with a one millisecond initial retransmit results in just 10 messages in that first second after the event so the 100 Pxxx elements from one IED only requires 26 kb/s bandwidth for the dataset payload of the frame.
OK, there may be even a dozen or so IEDs all "rattling through" a series of fast retransmit sequences at the same time, but still, at 1 Gb/s on the backbone or even a subscriber capable of only 100 Mb/s is hardly going to be bothered by the message rate or size.
Of course if you still had some concern about that, VLANs and/or Multicast Filters can assist as discussed here: /wiki/spaces/AP/pages/1487896577

But still that information does not tell us WHEN any <<stVal>> changed state in the publisher IED.

The subscribing IEDs (including any traffic recording device as Sequence of Events /waveform capture) only know when they RECEIVE a message with the first changed value.

It is almost possible to deduce when the first change of state message was sent by the publishing IED based on the subscribing IEDs analysis of the SeqNum of each message and the Ethernet frame time stamp <<T>> if you know what the initial setting of the fast repetition <<timeAllowedToLive>> value is for each publisher IED’s messages. A “simple” algorithm could then tell you what time the Ethernet message <<T>> was for the first change of state message.

But STILL that is not the EXACT time the Pxxx.Op.stVal changed value!  It is only the time the message was created to be published on the LAN which takes some processing time inside the IED after the change of state actually occurred.  To understand the difference, the maximum GOOSE latency of three milliseconds is from the time of change of state of the function, not from when it appeared on the LAN.

If we want the exact time that a function Pxxx.Op.stVal changed state we need to know its change of state time

Consider the time stamp to the millisecond of the time a LN status changed is 64 bits  ...

• times two for two Data Attributes <<.Op>> and <<.Str>>
• plus the 26 bits of <<.Op>> and <<.Str>>
• gives 154 bits per Logical Node
• times "100" Pxxx LNs in the one IED is 15400 bits, plus header and footer, to be sent PER MESSAGE ... that is a LOT of bits on the LAN!!
• times 10 messages in the first second after an event status change is 154000 bits per second = 154kb/s

One could argue that is still relatively insignificant in terms of overall bandwidth of even a 100 Mb/s port speed.

However 1 second is "several life times" as far as protection time sequences are concerned where the circuit breaker probably needs to trip in well under 100 milliseconds.
So whilst port speed si important, message latency and delays is super-important!

The 15400 bits of data set alone through a 100 Mb's port takes 0.154 seconds to transmit out the port .. and that is repeated on the ingress port to the first switch and the last egress port to the destination IED as well as every switch-to-switch port pairs in between albeit at faster speed of say 1 Gb/s switch-to-switch, and then repeated on the ingress.
Given the fastest GOOSE needs a function-to-function latency of 3 milliseconds, the size of each message as it passes through a port becomes of concern - after all, that is why GOOSE and SV use "stripped down" Ethernet frames without the Layer 3 IP Address fields - the Ethernet Standard itself was specifically modified in the mid 1990's to provide for GOOSE and SV Ethernet frame types!.

We must also remember the queuing nature of switches themselves .. one they start sending a message, it has to wait till that has cleared the buffer before another message can be selected to be sent.  So if a normal heartbeat message containing 100's of time stamps is in "mid-transmit" and another message arrives representing a change of state of a function in another IED, the import Pxxx.Op.stVal is going to be held up till teh longer message has all been sent. 

So if we are concerned about the shortest possible length GOOSE message, why would we want to add in a LOT of time stamp overhead that the receiving IEDs will not use anyway?

Clearly sending every Pxxx.Op.t in every "trip Goose" is counter productive to the intent of GOOSE as a FAST transmittal of signals. 

In any case the subscribing IED doesn't care what that value of time was.  It only cares when it received the first message with a changed value of Pxxx.Op.stVal 

So we have a conundrum - we want to send GOOSE as fast as possible, but we also want to record the time stamp of every change of Pxxx.Op.stVal, Pxxx.Op.q, Pxxx.Str.stVal, Pxxx.Str.q

We could rely on IEC 61850-8-1 MMS Reporting to send a complete message with all the necessary information from the source SERVER IED and somehow an MMS CLIENT IED device could process that to derive the Sequence of Events.  

However given that GOOSE is such a small bandwidth, we could structure two GOOSE messages:

1. one with a high priority tag and a 1 ms initial fast retransmit with just the "essential trip information"
2. another with a lower priority tag with say a 10 ms initial fast retransmit including ALL the <<.t>> time stamps