Preforming a migration of Computers and Servers for an organization that is continuously under the attack of the best Hackers around the world can become extremely challenging. In a recent engagement with a private organization that provides multiple forms of services to various areas of the government we experienced a multitude of security related challenges throughout the project. But the most distressing challenge was facing the reality of being compromised, twice.
During the peak of the migrations when we had the most momentum, moving massive numbers of computers and workstations per day, the customer experienced the first compromise that ultimately stopped the migration cold and shut down any progress for three days. Multiple service accounts were deemed to be “at risk” by the security team which invoked a massive effort to not just change passwords but to change service accounts across the board. About thirty days later the customer experienced a second compromise, this time the migration team was better prepared, keeping the duration lower though the recovery effort was still time intensive. Through both incidents the hackers were able to gain access to accounts that would ultimately allow access across multiple machines in multiple domains, a common setup in most migrations.
After the second compromise the migration team was determined to modify the process to reduce the potential risk for the remainder of the migration as much as possible. The first step was to no longer use the AD Trust that was in place. Using accounts from both the source domain and target domain without using any group memberships that would utilize the Trust to prevent lateral movement across domains. The second step was to eliminate the use of any group memberships that would grant administrative access to multiple machines. In effect, each service account was limited to domain user with some explicit permissions applied. The third measure was to disallow any interactive logons with the service accounts. The fourth was to limit the lifespan of the service accounts to a very short duration. For the fifth measure we kept the systems clean – we only deployed the agents to the systems we were processing and we removed the agents promptly when processing was complete.
The process changes significantly increased everyone’s level of effort, ours and the customers. We had to manage the continuous changing configuration of domain credentials for each collection. The customer needed to manage the creation and deletion of service accounts on a regular basis with a quick turnaround. The customer also had the monotonous task of adding and removing these accounts from each of the local systems being migrated. More work for everyone but in the end we significantly lowered the risk factor – acknowledging the fact that the risk is never totally eliminated. More importantly, there was not a third potential compromise event.
Lessons Learned and Best Practices for Minimizing Risk (summary):
- Separate service accounts for all domains involved in the migration.
- Assign rights specifically for AD rights – avoid groups, and in particular well known groups.
- Avoid well known group membership for workstation access. Build a process to add and rights on workstation to limit the timespan of control of the account.
- Do not allow interactive login of service accounts. Use the Delegation features in QMM (they do work… now!).
- Deploy migration agents to machines only when needed, and clean up early and often.
In the interest of security, we don’t want to say all the ways we know how a migration can temporarily impact security, or exactly all the technical details behind how those fixes listed above reduced risk. If you are planning a migration need to know, feel free to contact us!