On November 27th around 05:00 UTC, we were notified of a critical vulnerability in the proxy smart contract that has handled deposits to the dYdX exchange since November 24th. Around 12:00 UTC, the dYdX team executed a white-hat hack to rescue vulnerable user funds, totalling about $2M. These funds were sent to a non-custodial escrow contract where only the original owner of those funds can retrieve them.
Users who set an ERC-20 allowance via the dYdX deposit flow between November 24–27 should revoke all approvals on this contract. It is important that affected users avoid sending ERC-20 tokens to their wallet until this step is completed.
If a user connects any wallet affected by the bug to trade.dydx.exchange, they will see a prominent warning with instructions guiding them through the process of revoking approvals and, if any funds have been escrowed, retrieving escrowed funds. If you do not see a warning when you connect your wallet on trade.dydx.exchange, then your wallet has not been affected.
There were 730 affected addresses with allowances, of which 180 held funds directly at risk when the exploit was discovered. As of December 8 at 18:00 UTC, all but 91 of the original 730 addresses (none of which currently have exploitable funds in their wallet) have revoked their allowances. We are continuing to monitor affected wallets to ensure the safety of user funds to the best of our ability but affected users should revoke approvals rather than relying on our ability to assist protecting those funds.
No other smart contracts or components of the dYdX exchange and protocol are affected, and new approvals and deposits can be made safely. Funds on the dYdX exchange were not affected and wallets that did not interact with the vulnerable smart contract were not affected. If you believe you were affected, please connect your wallet to trade.dydx.exchange and follow the instructions if prompted, or contact us by selecting “Live Chat” within the help section of trade.dydx.exchange.
Timeline (UTC)
Nov 27 05:21 - We receive a report from a member of a trading firm regarding a vulnerability in the dYdX deposit proxy contract.
Nov 27 06:40 - The dYdX team confirms the bug. We ask samczsun and Georgios from Paradigm to join us in the war room as we respond to the vulnerability.
Nov 27 06:50 - The dYdX and Paradigm teams together decide the best course of action is to execute a white-hat hack of vulnerable funds to move them to a safe location.
Nov 27 07:33 - As samczsun finishes a proof of concept for a white-hat hack, we begin implementing the recovery flow in the frontend and implementing a bot to save any additional funds that become vulnerable.
Nov 27 08:01 - We determine that ~700 addresses have allowances set on the vulnerable proxy contract. Of these, about 180 have funds at risk, totaling ~$2M.
Nov 27 09:27 - We ask Etherscan to remove the verified source code from the proxy contract and we make the source code on GitHub private.
Nov 27 10:33 - The white-hat escrow contract is deployed.
Nov 27 10:56 - The setAllowance function is disabled in the dYdX exchange frontend.
Nov 27 12:04 - The first white-hat hack is executed via flashbots.
Nov 27 12:22 - The frontend is updated to show the warning and recovery process to users affected by the bug. Allowances for deposits are re-enabled using the old deposit flow.
Nov 27 12:32 - Announcements are made on Twitter and Discord. Users begin recovering their funds from the escrow contract.
Nov 27 12:43 - The second white-hat hack is executed via flashbots for two more addresses.
Nov 27 13:16 - Our recovery bot is deployed to automatically retrieve and escrow any more funds that become vulnerable.
As of Dec 8 at 18:00 UTC, $533,000 of additional funds were rescued by the bot, and nearly 80% of affected addresses have revoked approval from the vulnerable contract. Another estimated $211,000 in funds were exploited by frontrunner bots. We are in contact with some of these bot operators and working to return funds to their rightful owners.
Technical Details
The smart contract containing the vulnerability was a “currency converter” proxy contract designed to allow deposits to dYdX from a broad range of currencies. The contract works by converting a user’s funds to $USDC and depositing them to the dYdX exchange in a single transaction. The implementation was based on a previous smart contract that was used with an earlier version of the dYdX exchange. The old contract uses a standardized “exchange wrapper” interface to support calls to various decentralized spot exchanges.
The new contract intended to perform swaps using 0x API. Quotes returned from the 0x /swap/v1/quote endpoint include calldata that can be submitted to the 0x contract to execute a trade. In Solidity, the call to the 0x smart contract can be implemented as follows:
(bool success, bytes memory returndata) = exchange.call(data); require(success, string(returndata));
Note that both the exchange and data parameters are untrusted: they are provided off-chain by 0x API, and are supplied to the smart contract by the client/user. For our use case, this is safe when the call to the 0x contract is made via the exchange wrapper interface, as shown below:
During the development and review of the new contract, the smart contract logic was simplified with the goal of reducing the number of transfers required to perform a conversion and deposit. The new proxy contract flow was as follows:
The problem with the new design is that a single smart contract has both an ERC-20 approval set by the user as well as the ability to make arbitrary smart contract calls to an untrusted address with untrusted calldata. This allows an attacker to steal an ERC-20 balance, up to the approved amount, by passing in parameters with exchange equal to the ERC-20 token address, and data encoding a call to transferFrom() transferring funds to the attacker. Then the following line results in a transfer of funds from the user to the attacker:
(bool success, bytes memory returndata) = exchange.call(data);
How did this bug make it to production?
Many risks must be considered when using low-level external calls in Solidity, such as the call() function. This is especially true when allowing a smart contract to make an external call to an untrusted target with untrusted data. These risks were well understood by members of the team, which is one of the reasons dYdX developed an exchange wrapper standard used by our previous protocols.
This bug should not have made it to production, and the fact that it did reflects, regretfully, a lapse in our security review and launch process for this particular feature release. The currency converter proxy was a relatively short and simple smart contract, based on a smart contract that had previously been battle tested. Because of this, we let our guard down and allowed the contract to go to production sooner than it should have, without the same level of rigor in the review process that is typically afforded to our smart contract launches.
The affected contract had 100% code branch test coverage and thorough integration tests, but these do not make up for insufficient attention given to the security of the higher level smart contract design. We believe this incident was preventable, and we are taking several steps to improve our processes to avoid any such oversights in the future.
What will the team do to ensure this doesn’t happen again?
At dYdX, we hold ourselves to the highest smart contract security standards. This includes, but is not limited to, 100% code branch coverage on all smart contracts, and thorough external audits for major new smart contracts.
Our biggest learning from this incident is the need to apply security best practices more consistently. The criteria and the bar for launch-readiness should be maintained for new smart contracts and modifications alike, no matter how small. Going forward, we will require that without exception all smart contract code undergoes a thorough internal audit with a dedicated security reviewer before it is released.
In addition, we are developing explicit security guidelines that can be applied earlier in the development process. This includes principles for smart contract development that could have been used to avoid this incident:
New contracts and changes to the design of battle-tested contracts should be reviewed critically, and specifically with an eye toward known risky patterns and previously reported smart contract exploits.
Any smart contract review should take the time to explicitly consider the worst-case scenario of a new deployment or contract modification. In the present case, such an analysis would have made it explicitly clear that a bug in the proxy contract puts user funds at risk—including funds held in user wallets—despite the fact that the proxy is itself completely stateless and never holds user funds.
When implementing any contract, careful consideration should be given to admin safeguards, even if we don’t expect that they will ever be used. The worst-case analysis helps to give insight into the potential benefits of safeguards such as the pausable pattern.
Bug Bounty
This was a critical vulnerability and we are grateful to the person who made the disclosure for disclosing the bug quickly and privately. This responsible disclosure allowed us to act quickly and minimize impact to our users.
In accordance with our Vulnerability Disclosure Policy, we have paid the person who made the disclosure a bounty of 500,000 $USDC .
Although the amount may seem significant for the amount of funds at risk, we believe generous bounties should be paid for the responsible disclosure of critical vulnerabilities. dYdX will continue to pay generous bounties for the responsible disclosure of vulnerabilities going forward.
Reimbursement of User Gas Fees
As a result of this vulnerability, users incurred additional gas fees to revoke approvals and, for certain users, retrieve escrowed funds. Although generally we do not reimburse gas fees that users incur, in this case we will transfer 0.0115 ETH to all users who incurred gas fees to revoke approvals and an additional 0.0115 ETH to all users who incurred gas fees to retrieve escrowed funds.
Reimbursement of Stolen Funds
Even though we were able to successfully recover all initially vulnerable funds, user wallets remained vulnerable until they unset allowance on our webpage. This meant that any funds these wallets received were vulnerable to being hacked. Despite our efforts to execute fund recovery on an ongoing basis, a total of $206,134.87 and 1.5 WETH was stolen. We have fully reimbursed these losses (where the address had revoked the contract). Due to the possibility of faking funds being stolen and the amount of time that has passed since the incident, we will no longer reimburse losses going forwards.
Conclusion
While this incident represented a critical vulnerability and failing of our internal security standards, we were able to successfully safeguard the bulk of user funds. We greatly appreciate the responsible disclosure of the vulnerability and have paid a bug bounty of 500,000 $USDC .
We appreciate the community’s patience and understanding throughout this incident and its resolution. We look forward to continuing to build and ship high quality smart contracts alongside the community.
About dYdX
dYdX is the developer of a leading decentralized exchange on a mission to build open, secure, and powerful financial products. dYdX runs on audited smart contracts on Ethereum, which eliminates the need to trust a central exchange while trading. We combine the security and transparency of a decentralized exchange, with the speed and usability of a centralized exchange.