Deploying a Sonus Cloud Link CCE

I recently was working on a Sonus v2 Cloud Connector Edition (CCE) working with the new hardware, testing the install, etc. when I ran into a deployment snag. In general, I find the there are a lot of post and blogs about when to use CCE and when you cannot, but few on how to properly deploy CCE let alone the Sonus version.

I decided to show a walk through on the process I used to deploy, why I used the names/options that I did, and the errors and gotchas I ran into.

To start, I am working with a Sonus SBC 1k Cloud Link with the CCE package installed. From a Sonus part number, that would be SBC-1K-SIP-E-CL. The Cloud Link version includes an imbedded Windows 2012 R2 Server with 32 GB of RAM, dual Xeon quad-core 1.7GHz (that’s 8 physical cores. 16 logical), and a 512GB SSD drive. All-in-all, not a bad server and very capable of running the “small” CCE build. That, along with the nicely integrated SBC makes this solution simple…as long as you know the answers to the questions.

Starting off the build of the ASM is not any different from other ASM buildouts – the only difference being the Task option on the Cloud Link version shows Office 365™ Cloud Connector Edition vs. Skype ™for Business Survivable Branch.

Blog Image 1.png
Blog Image 2.png

Selecting the Office 365™ Cloud Connector Edition brings you to the beginning process. The very first thing that needs to be done is getting the ASM an IP. This IP is the IP of the Hyper-V host. It is not meant to be domain joined, and the option is not present. If you really wanted to change the name of the server you could, but even that is not required as again – it is simply the host computer for the VMs.

It is critical that the Remote Desktop enable option is changed to Yes as there are tasks that must be completed from the host server. The server should also be able to be reached from within your network – whatever that means to you. DHCP is an option – I am old school and believe all servers should have a static IP but again, whatever works for you. For me this was my internal server VLAN, internal DNS, internal everything as I wanted to be able to easily manage the server.

Blog Image 3.png

After you have configured the IP and RDP, the next step is to create the certificate for the Skype Edge server. This is where you must be careful to enter the correct names. There are references to the deployment requirements on TechNet here Plan for Skype for Business Cloud Connector Edition although to me, Sonus solution hid the configuration too well making things unclear. Specifically, step four is where you configure the settings on the CCE including the site name for the CCE. This site name is also your edge pool name – a name which must be a CN or SAN.

In my case, I thought I would be witty and use BCLCCE.bricomplabs.com as the common name. The CSR request simply stated the edge server public FQDN and left it up to me to complete. The wizard also complained if SIP was not in the name so that had to be there too which was a bit confusing since the DNS name would never point there. In the end, the fields looked like the following:

Blog Image 4.png

CN=BCLCCE.bricomplabs.com
SAN=DATA-CENTER.bricomplabs.com,SIP.bricomplabs.com, BCLCCE.bricomplabs.com

In the SAN list make sure you have your SITE name (exiting or the name you plan to use), SIP, and optionally another name to which you would be tied to the service (although 100% unnecessary). Remember to configure all of the DNS records publicly to make sure things route.

Once the CSR has been created, have it issued by your favorite PUBLIC CA, and make sure your favorite public CA is a mainstream one – one whose roots are part of the base Windows 2012 R2 OS. In my case, I used DigiCert (http://www.digicert.com) – an awesome go-to CA who works flawlessly.

Step 3 is to import (paste) the resultant certificate. The cert should be in DER format and in the case of DigiCert simply select the option on their page to download copy/paste under Download Certificate. That will expose three text blocks, the one you want will be the top block for your certificate where you can simply copy and paste the result.

Blog Image 5.png
Blog Image 6.png

You are now ready to begin the configuration of your deployment and a critical junction. Incorrect information going forward means clicking Reinitialize on the ASM and starting over. 😊 Below is a summary of the deployment and the options selected.

Blog Image 7.png

In this list, there are some key components that we need to complete. It is also important to note that the defaults are there just because but more than likely mean nothing to your deployment. The first thing you need to identify is the CCE Site Name. Again, this will be the pool name of your edge and will need to be in your certificate.

The external network gateway is in relation to the second NIC of the Edge server – no different from the second NIC of a traditional Edge server. This NIC cannot be on the same VLAN as your internal networking but like a traditional Edge, NAT is supported. In my example the external is a 172.x.x.x/24 address, I am using public Level3 DNS, and because it is a private IP I am listing the Edge server External IP.

The internal network is where the internal NICs on the servers will live. There are three switched networks on the servers – Internet, Corpnet, and Management. The Internet switch and NIC live on the Edge server while the Corpnet and Management live on all the other servers. The Management virtual switch is internal only – for server to server communication and the IP scheme is 192.168.213.0/24 with no default route. The Corpnet is the internal network where the Hyper-V host lives as well as all other internal servers (again, in my network the server VLAN).

Blog Image 8.png

The information you are entering for the Internal Network is used partially for the configuration – and partially still a mystery. The Gateway is obvious and is the gateway used for the Corpnet NICs however the Internal DNS is not used (the four servers all use the AD server as they should). The four IPs of the VMs themselves are also Corpnet IPs and in my case, were 10.x.x.10, 10.x.x.20., 10.x.x.30, and 10.x.x.40 – but as long as they are unique and valid, they can be whatever you want.

One you have configured and saved the CCE configuration move on to Step 5 – Prepare CCE. In this process the data that was previously entered is saved locally to the Hyper-V host (C:\UX\CCE\CcAppliance) and will be used by the next PowerShell commands.

Assuming no errors and all is ready, RDP to the Hyper-V host. If you have not already, set the Administrator password of the ASM via the web GUI of the Sonus at Settings | Application Solution Module | Change Admin Password. Drop the option down to User Configured, enter and confirm a unique strong password, and click OK. Using \Administrator and the Password you just created, RDP to the ASM address set in step 1 above.

One on the server, start a PowerShell command with elevated admin rights. From within the prompt, start the process by entering Register-CcAppliance. You will need to set admin passwords, recovery passwords, and enter your admin login for your O365 tenant. Assuming you have the correct rights the process will complete creating an appliance in the cloud which you can see using the Skype for Business Online PowerShell command Get-CsHybridPSTNAppliance.

The final stage of the process is the installation and configuration of the VMs. This entire process is completed with the simple command Install-CcAppliance. Using previous configuration entries saved off to the INI and the certificate (also saved off), the nest steps are hands free, it just takes time. This is where I ran into my errors due to the lack of pool name in the edge certificate. During the creation process the Edge sever is started and an error is thrown which appears to be a Cyphers issue:

Event ID 14397 – A configured certificate could not be loaded from the store.

Extended Error Code 0xC3FC7D95 (LC_E_VALIDATION_CERT_NO_KEYEXCHANGE)

Should you run Get-CsCertificate you will see your public certificate associated with AccessEdgeExternal, DataEdgeExternal, and AudioVideoAuthentication. An internal certificate will also be seen and associated with Internal. All of this appears to be valid and yet the service will not start. The key to finding that it was a pool name issue was manually assigning the certificate to the three external services again using

Set-CsCertificate -Type AccessEdgeExternal,DataEdgeExternal,AudioVideoAuthentication -Thumbprint xxxxxxxxxxxxxxxx -Force

Doing so revealed the error that data-center.bricomplabs.com was not a name on the certificate and that’s when the lightbulb appeared. The fix is to undo everything and start over (unfortunately) which includes Unregister-CsHybridPSTNAppliance, OPTIONALLY Remove-CsHybridPSTNSite, and Reinitialize the ASM in the Sonus GUI.

Once the CCE appliance is configured, make sure to run through the SBC configuration - otherwise there will not be anything to link the calls to. The CCE does not use TLS so an SBC certificate is not required, only basic integration configuration as described on the Sonus site (and identical to any other SBC config). https://support.sonus.net/display/UXDOC61/Configuring+the+SBC+Edge+for+a+Single+CCE

 

Update 2/24/2017

Internal DNS on the Internal NIC settings

As mentioned in the comments (thank you Jason) the DNS entry/mystery is solved as the internal DNS added during the CCE Setup page is added as a forwarder. Why not just use root hints - none from what I can see in v1 but future version may rely on knowing the DNS of internal servers.

Blog Image 9.png

SIP Domain Name on the Certificate

The addition of the SIP.DOMAIN.COM to the certificate - that mystery was resolved as well (thanks Carolyn). When a CCE user makes a call to the on-premises edge, a check is made against the edge for SIP.domain for the sip domain of the user making the call. This is how the CCE authenticates/permits this call from the CCE online user to the on-premises edge. If you don’t have SIP in the SAN name then the outbound call for the user will fail with the following error:

504  Server time-out

ms-diagnostics:  1017;reason="Cannot route From and To domains in this combination";cause="Possible server configuration issue";summary="The domain of the message that corresponds to remote peer (external) is not shared between local and remote deployments";external-domain="bricomplabs.com";external-type="domain-type-local";internal-domain="bricomplabs.com";internal-type="domain-type-local";source="sipfed2a.online.lync.com";OriginalPresenceState="0";CurrentPresenceState="0";MeInsideUser="No";ConversationInitiatedBy="0";SourceNetwork="0";RemotePartyCanDoIM="No"

Final Certificate Requirements

In the end I updated my certificate to only include the required DNS names and to lessen the confusion. The certificate in the end has the CN of the site as well as the site and sip as a SAN.

CN=DATA-CENTER.bricomplabs.com
SAN=DATA-CENTER.bricomplabs.com,SIP.bricomplabs.com

Unable to patch Skype on Sonus SBA

I recently was working on a Sonus SBA, patching away, when I found the patch process failing me. The copying, installation, and what appeared to be overall process was generally working but when the last patch (server.msi) was being applied, the process would fail.

Looking at the ASM/server directly I found the Skype server service was not starting – in fact it was unable to fully start as the DynDB database and log files were missing. Odd – must have been dropped during the last patch process, more than likely when the databases were being upgraded. Simple fix (or so I thought) – run Install-CsDatabase -LocalDatabases -Verbose and the missing database and log would be created…or should have been. However, the process failed with an error stating network name not found:

RtcDyn db state is: DbState_DoesNotExist

Dyn Data Path = C:\CsData\RtcDatabaseStore\rtclocal\DynDbPath, Dyn Log Path = C:\CsData\RtcDatabaseStore\rtclocal\DynLogPath.

Creating database rtcdyn from scratch. Data File Path = C:\CsData\RtcDatabaseStore\rtclocal\DynDbPath, Log File Path= C:\CsData\RtcDatabaseStore\rtclocal\DynLogPath.

Clean installing database rtcdyn.

System.IO.IOException: The network name cannot be found.

Now that is just silly – C:\ not found? I could navigate to the path, the permissions matched the FE servers I have in production, so something else was off. Turns out, when the database install process happens local (SBA/SE) or remotely (FE) the installation still uses the \\servername\c$ method to connect and create the databases. In the SBA case, it was hardened by Sonus security template and the C$ was removed.

Blog Image 10.png

It is also interesting to note this impacts tools like SCCM and its ability to push the client – no C$ = no ability to connect. So, in our case, the fix was simple. Add the C$ share to C:\ and re-start the upgrade. To add the share simply right-click in the Shares window, select New Share, and start the share wizard.

Blog Image 11.png

Enter C:\ as the folder path you wish to share.

Blog Image 12.png

Windows will warn you this is a bad idea – acknowledge that you know more than the system by clicking Yes.

Blog Image 13.png

Enter the hidden share name of C$ (and optionally enter a description (the standard being Default Share)).

Blog Image 14.png

Select the second radio button, granting administrators full access and others read. The permissions will be reset after a reboot and selecting the second option allows you to validate/test the process.

Blog Image 15.png

The final result should show the admin share in your list.

As mentioned, this is only a temporary fix – the share is removed when rebooted and the system policies are reapplied. So make sure you perform this workaround just before the patch is installed and all should work as expected.

Blog Image 16.png

RtcDyn db state is: DbState_DoesNotExist

Dyn Data Path = C:\CsData\RtcDatabaseStore\rtclocal\DynDbPath, Dyn Log Path = C:\CsData\RtcDatabaseStore\rtclocal\DynLogPath.

Creating database rtcdyn from scratch. Data File Path = C:\CsData\RtcDatabaseStore\rtclocal\DynDbPath, Log File Path= C:\CsData\RtcDatabaseStore\rtclocal\DynLogPath.

Clean installing database rtcdyn.

Creating database rtcdyn. Attempt: 1

Setting the database rtcdyn to restricted mode.

Database rtcdyn set to mode Restricted.

Setting database options.

Begin transaction.

Creating objects from dbcommon.sql.

Creating database objects.

Executing Types.sql...

Soder on November 28, 2016 asked "Was Sonus notifed, or they simply dontcare at all?" 

Brian then replied "They were not but the issue is with the cmdlet Install-CsDatabase which is part of Skype and not something Sonus created."

Soder replied "Ok, seems I can reply now.So. You wrote
"In the SBA case, it was hardened by Sonus security template and the C$ was removed."
Its Sonus fault. Doing these "hardening" things without doing their homework (plan careefully, do impact analysis, etc. etc.). And as you say the C$ gets removed if the hardening is activated, it means all Sonus equipment owners are affected all over the world. I wouldnt consider this a minor small tiny issue, right?"

Brian replied "Still not in the 'Sonus is at fault' court - the security templates are provided by Microsoft but I will ask for clarity from both Sonus and the Microsoft PG and post a reply. It should be noted that the application of the Security Template - while permanent - is optional."

Soder replied "Well, if the template comes REALLY from Microsoft, thats a different beast (is that like a Security Policy template called Hardened Workstation or something similar?). But still, Sonus allows the enablement / disablement of this template on their GUI, so assume they have tested this before rolling out to customers. I really want to believe they dont just accept everything unconditionally from MSFT and implement without checking its effects first. I know this is a completely optional feature, so it may not affect literally everybody. But as the option is offered to customers without any warning that it may break the whole Sfb update procedure, I am still confident the ball is on Sonus side."

Brian replied "So yes - the security templates are offered from the Microsoft security team and you can apply the XML files to any computer/server - the process stems back to the Windows 2000 days. The key point to remember is the SBA image/process is a managed MS process much like the LPE phones. Sonus and the SBA image are up to the mercy of the MS team and their direction. So, the only question is where did the direction come from for the template - MS or did Sonus do this on their own. That question I have asked and am awaiting the answer."

Mark H on December 1, 2016 stated "I am running Lync 2013 SBA from Sonus and I see the share is disabled as well  but CU are installing properly.
Maybe Microsoft changed something on their installer."

Kevin I replied "Kevin I from Sonus here. The CU Installer has changed with Skype for Business June CU. In previous CU's the updater did not use the SBA C$ share to update. The security template was changed based on customer feedback to remove access to the c$ share. It was changed a couple of years ago with no issues until this CU.

We have implemented a change to our CU installer to restore the share before the CU installs and then remove it afterwards. 

We are also working with Microsoft to understand if there is a way to change the way the CU's are applied to the SBA. Until then the change I mentioned above in the installer will stay to ensure the CU will install moving forward."

Soder replied "Kevin I:
maybe this is not the best platform to discuss this topic, but why there is not a single(!) word about this whole "Security Hardening Template" topic in the 1700+ pages Complete Sonus Documentation PDF ? Regardless of v4.x v5.x or v6.x."

Brian replied "@Soder - have you seen this link? support.sonus.net/.../Applying+an+SBA+Security+Template"

Soder replied "You are right! I didnt find the topic because I was looking for the "hardening" keyword as seen on the button, not searched for generic expression like "template" or "security", as those spit out 1000+ hits on the 1700+ pages document. Yes, I was blind, but organization of content in the Sonus documentation is yet another (saddening) topic.