Last updated on March 17th, 2015 at 11:55 pm
Coinapult is well-known to many in the cryptocurrency community. The company was founded by Erik Voorhees and Ira Miller in 2012, and raised $750,000 USD in a seed round led by Roger Ver, FirstMark Capital, and the Bitcoin Opportunity Fund. Coinapult is based in Panama City.
Account manager Robinson Dorion at Coinapult has sent over a timeline regarding the Coinapult Hot Wallet compromise.
At 9:27 UTC an unauthorized withdrawal for 150 BTC was sent from Coinapult’s hot wallet to this address: 12LszeXACdj9bdETzv8BkXyWeabZ1151aA. The address as of 7:25PM EDT contains 150 BTC worth approximately $43,080.00 USD according to Winkdex and remains unspent and unmoved.
Coinapult team members that are currently working to resolve the issue are Ira the CEO, Zach the IT Admin, GP the CTO, Cindy a developer and forensics expert, Justin the COO and Robinson a customer service employee.
The hot wallet was kept in a Tier 3 data center with only two team members having physical access. They include Ira and Zach both whom are currently working to determine how the compromise happened in addition to working to secure Coinapult. SSH access to the server is limited to four individuals inside the company which include Ira, Zach, GP and Cindy.
Coinapult states that connecting to the server using SSH requires users be signed in to the company VPN and use individual SSH keys for appropriate logging. Each of the production key holders’ laptops were inspected by the others for network activity from the time window with nothing suspicious found, however Zach’s laptop was exhibiting strange behavior reminiscent of a MITM attack.
The company stated that while everyone was using the same local network that Zach’s laptop was showing a Gabon based IP address while other team members are showing Panama IP addresses. Upon discovery of the discrepancy Zach powered down his laptop and the hard drive was removed for forensic analysis.
The company reported that on March 13th the data center where the finance server was hosted experienced an all day outage. The outage coincided with all Panamanian goverment websites and other local business sites and servers also being offline. The phone system at the data center was also down. During this outage Zach was logged in to nearly every machine at the data center as part of the recovery process from the outage.
Due to the outage GP emailed Justin, Ira and Zach a plan to transition all IT services to various servers outside of the data center in an effort to mitigate the risks across various data centers and may have inadvertently tipped of the attacker that a penetration of the Coinapult systems would need to take place before the moving of Coinapult’s servers.
The company reported the past two weeks have been unusually problematic for system issues and stability. Coinapult has experienced hard drive issues, CPU issues and other issues with the machines hosted in the data center and while the causes of these issues are known it could have been the masking of malicious activities.
The company has begun an analysis of all systems and found several clues regarding the compromise.
The /var/log/auth.log file has been modified. The file contains an additional blank line and the auth.log.1 file has been emptied. Prior to the compromise the auth.log.1 file would have been full of data from the past several days.
The /root/.bash_history file was also modified and shows some troubling access to the machine.
The last four entries of that file are:
- nano auth.log
- nano syslog
- nano ufw.log
The company has found this to be out of the standard Coinapult usage and likely run by the attacker with the intent to doctor the files after leaving the system. The Coinapult team believes a root kit could have been used and is hopeful that a forensic analysis of the hard drive would help determine if that is the case.
Coinapult has provided the following timeline regarding events. All times listed are UTC -5.
1:49 – Ira requests hot wallet top off with 100 BTC from Bitfinex.
2:36 – Ira logs in to VPN (according to his syslog)
2:36 – Ira logs in to Finance server (according to server log)
2:37 – Ira runs sendmany to split outputs for optimal sending performance during the night. This was unnecessary as the 100 BTC had not shown up yet, but Ira did not notice that.
3:55 – Bitfinex sends 100 BTC withdrawal
4:15 – <Customer 1> notifies Robinson about improperly canceled transactions
4:27 – Withdrawal by hacker is made
4:54 – Robinson sends out email about <Customer 1> transactions being stalled and hot wallet being suspiciously low
4:58 – Robinson calls Zach and Zach starts trying to connect to VPN (according to his syslog)
5:17 – Zach successfully logs in to VPN (according to his syslog)
5:22 – Zach logs in to Finance server (according to server log)
5:31 – Zach sends email saying processes are running but can’t assess hot wallet on his own
8:42 – Ira has done enough investigation to identify that there has been 150 BTC withdrawn to an unknown address. Emails this info to the others in company.
9:12 Majority of funds are withdrawn from hot wallet. Customers (i.e. <Customer 1>) are notified and public notice is placed on our website. Team investigates and identifies the contents of this report.
The Coinapult team has powered down and isolated all hardware in the data center. They are working to disassemble and run forensics on the hard drive to see if they can recover data from the manipulated logs or elsewhere. Zach has also begun disassembling his laptop to run forensics on it and all hardware is being moved out of the data center.
The company is requesting the data center provide all access logs and surveillance footage relevant to the situation and seeking to gather more information about the March 13th outage experienced.