What I missed about BIP143's scriptCode

I recently tried to create a Transaction to spend from a P2WPKH address. I couldn't get this to work. The sendrawtransaction command kept failing with the error non-mandatory-script-verify-flag (Signature must be zero for failed CHECK(MULTI)SIG operation) (code -26). I narrowed the problem down to incorrectly hashing the sighash. I did not create the sighash as specified by the BIP143 Specification. I'll share my experience navigating this problem here.

A P2WPKH address is Segwit's version of a P2PKH address (See SegWit on Wikipedia). A P2PKH address uses the scriptPubKey shown below:

OP_DUP OP_HASH160 <pubkey_hash> OP_EQUALVERIFY OP_CHECKSIG 

It requires the spender to provide the following scriptSig:

<signature> <pubkey>

Here's a good article explaining p2pkh addresses. My (apparently flawed) understanding of P2WPKH was that it does the same thing but with SegWit so I assumed that it had the same scriptPubKey.

Spending from a P2WPKH address requires an ecdsa signature. The message that will be signed is the SigHash. The BIP143 Specification outlined the following items to be serialized into the SigHash:

     1. nVersion of the transaction (4-byte little endian)
     2. hashPrevouts (32-byte hash)
     3. hashSequence (32-byte hash)
     4. outpoint (32-byte hash + 4-byte little endian) 
     5. scriptCode of the input (serialized as scripts inside CTxOuts)
     6. value of the output spent by this input (8-byte little endian)
     7. nSequence of the input (4-byte little endian)
     8. hashOutputs (32-byte hash)
     9. nLocktime of the transaction (4-byte little endian)
    10. sighash type of the signature (4-byte little endian)

I found item 5, the scriptCode of the input to be confusing. I thought this might be referring to the scriptPubKey of the input UTXO but this led to wrong SigHash messages and wrong signatures, hence, my transactions were rejected. After hours of debugging, I discovered that further down in BIP143, there is a section that specifies what the scriptCode is for spending from a P2WPKH address, shown below.

For P2WPKH witness program, the scriptCode is 0x1976a914{20-byte-pubkey-hash}88ac.

0x19 indicates the number of bytes in the script. 0x19 is 25 in decimal. This indicates that the script has 25 bytes.
0x76a9 is code for OP_DUP OP_HASH160.
0x14 is 20 in decimal and it is interpreted as OP_PUSHBYTES20 which pushes the next 20 bytes into the stack.
0x88ac is code for OP_EQUALVERIFY OP_CHECKSIG.
See the analysis of a Transaction with a similar output here.

The scriptCode seemed to be the P2WPKH address scriptPubKey, but the signatures were wrong, why?

After logging the actual utxo.scriptPubKey, I discovered that it was:

OP_0 OP_PUSHBYTES_20 <pubkey_hash>

This wasn't what I expected. I had forgotten something crucial about P2WPKH. SegWit only requires that you specify the segwit version, which is 0 in this case and the 20-byte pubkey_hash. See the P2WPKH section on BIP141. This is called a forwarding script. See this excerpt from bip-tx-terminology:

forwarding script
[Concept] A collective term for scripts that redirect input validation to another script or data structure. Witness programs and P2SH Programs are forwarding scripts. Forwarding scripts make use of script templates that imply additional evaluation steps beyond the explicitly expressed conditions. In the case of P2SH, the output script in verbatim only implies that the redeem script must be the preimage of the hash in the output script, but the template prescribes that the redeem script must additionally be satisfied. For witness programs, the output script is even less verbose with more implied meaning.

DISCLAIMER: This terminology is still pending and it's not widely used yet. See https://github.com/Xekyo/bips/pull/1

The nodes in the Bitcoin network understand that this scriptPubKey is a Segwit_v0_P2WPKH scriptPubKey and they know the script template to use. All I had to do was create a new script following the scriptCode template and use that to make my sighash.

In Rust, using the bitcoin crate, the scriptCode and sighash can be made like this:

let script_code = bitcoin::script::Builder::new()
        .push_opcode(OP_DUP)
        .push_opcode(OP_HASH160)
        .push_slice(&pub_key.wpubkey_hash().unwrap())
        .push_opcode(OP_EQUALVERIFY)
        .push_opcode(OP_CHECKSIG)
        .into_script();

let message = SighashCache::new(unsigned_tx)
        .segwit_signature_hash(input_index, script_code, value, sighash_type);

The Bitcoin crate also provides a function to do this for you.

let mut sighash = SighashCache::new(unsigned_tx);
let (message, _) = psbt.sighash_ecdsa(0, &mut sighash).unwrap();

Conclusion

While the scriptCode looks like it should be the scriptPubKey for P2PWPKH addresses, it is not. P2WPKH uses a forwarding script specified by BIP141 and the actual scriptCode must be constructed using the defined script template.