Time Synchronization and ShoreTel Reporting
Recently we have seen several companies with discrepancies in reporting and analytics for the ShoreTel system that have been due to time synchronization issues between different components in the ShoreTel system. So we asked our CTO, Andrew Gaskill, to write up a quick blog post to discuss this and how to make sure your ShoreTel system stays in sync.
In the ShoreTel CDR database, calls are broken out into so-called “connect” records for each party involved in the call. For example, if a call comes in via PRI Trunk 1a and is answered by the Customer Service workgroup where it is queued for 20 seconds and is then answered by workgroup agent Joe Smith, there are connect records for the PRI trunk, the customer service workgroup, and for Joe Smith. These records are not linked in a “cause-and-effect” way such that looking at the workgroup’s connect record and the agent’s connect record it is readily apparent that one is the direct result of the other. Instead, the only thing we know is that the agent became involved in the call later in time than the workgroup queue, and therefore we know that to determine the talk time of the agent who answered that call, that’s the connect record to look at.
Consider now what happens if time gets out of sync within the ShoreTel system. The start and end times for connect records come from the switches themselves. If the switch managing Joe Smith’s phone in our example above is running 25 seconds behind the ShoreTel server, then the time that he became involved in the call will appear to be 5 seconds before the call entered the Customer Service workgroup queue. Therefore by a timeline reckoning he can’t have been the agent who answered the call and therefore the queue call will have no talk time.
Workgroup calls do have one additional piece of information attached to them which is the extension of the agent who answered the call, which lets us use more than just timeline information, but we can’t ignore it entirely since a call can be queued multiple times during its lifetime and even be answered by the same agent multiple times, therefore we still disregard activity prior to the call entering the queue. For non-workgroup calls, start and end times are used exclusively to determine the previous and next parties on the call.
So how can time get out of sync? If all your switches are managed by the ShoreTel HQ server it’s fairly unlikely — unless there is a management connectivity problem between the server and the switches, which would show in big red or yellow blocks on Director’s switch connectivity status page and be manifested in other ways as well, the time will always be synchronized. Once DVS servers or voice-model switches are introduced however, the potential arises for time synchronization issues. If a switch is managed by a DVS server, it will get its time from that server rather than the HQ server, so it is imperative that DVS servers are time-synchronized with the HQ server. Voice-model switches use an SNTP server directly to synchronize their time, and so must also use a common time source.
UPDATE: As pointed out in this Microsoft support article (http://support.microsoft.com/kb/939322), Windows time sync is only designed to meet the needs of Kerberos authentication, which only requires that computers’ time be loosely synchronized, so it does not strive for accuracy better than a few seconds. In many cases, better accuracy is needed for ShoreTel reporting. One third-party application Brightmetrics has tested and recommends is Meinberg’s Windows port of the reference NTP implementation, which targets millisecond accuracy between computers. In this case, either the third party time sync utility can be installed on all DVS servers and they can all point to the HQ server as their time source, or the third party time sync utility should be installed on all DVS servers as well as the HQ server and all should point to a common time source.
The original content of this post follows, discussing Windows built-in time synchronization, which, as stated, can only guarantee accuracy within a few seconds.
If your ShoreTel HQ server and all DVS servers are members of the same Active Directory forest, then by default their time will be synchronized using AD time services. If they are members of different AD forests, then all AD forests should be configured to use a common time synchronization source — see Configuring a time source for the forest. If they are not domain members, then their time source should be configured manually — see Documentation for the “net time /setsntp” command. Voice-model switches are either configured with an SNTP server at configuration time, or get it from DHCP (see your ShoreTel Admin Guide. In the 12.3 guide, it is section 6.5.2), just like IP Phones.
Which server to use as the SNTP server for voice-model switches and non-domain-member DVS servers depends on the HQ server. If the HQ server itself is a domain member, you can point everything to a domain controller in that domain — either a single domain controller for all devices or a domain controller at the device’s local site. Since the domain controllers should all have synchronized time, any domain controller in the domain should suffice for any given device.
If the HQ server is not itself a domain member, then a common solution is to make the ShoreTel HQ server itself an authoritative time source and point everything else to it. See the “Configuring the Windows Time Service to use an external time source” section in this Microsoft knowledgebase article, and specifically the note about using 0xA in the AnnounceFlags, in case the ShoreTel HQ server does not have internet connectivity.
It’s important to understand that this issue can be raised regardless of the reporting platform utilized. Because of our robust analytics capabilities it is sometimes easier to notice issues like this in the Brightmetrics system but the issues cause the same anomalies in ShoreTel’s built in reports as well.
If you have any questions about this or any other Brightmetrics issues, please feel free to contact me at firstname.lastname@example.org.