Oracles

In previous chapters you were introduced to the challenges and dilemmas faced by transactions that rely on some sort of reference data to attest to their validity and you were introduced to some options to resolve that problem:

• Attachments, zip or jar files, can be used when large non-object data, e.g. Excel or pdf files, that doesn’t change frequently is embedded in the transaction.
• Reference states are a good option when the data exists on-ledger, is (mostly) immutable and benefits from the notary guaranteeing that you’re looking at its most recent version.

But, what if the data changes frequently? Are you going to keep sending attachments back and forth between nodes? And, most importantly, what if the reference data exists off-ledger? In this case, how will 2 nodes reach consensus about this data?

Any node will provide some data and claim that it got it from a trusted 3rd party. Different nodes will provide different values. That would lead to non-deterministic transaction verification.

An answer

An answer to the above questions is oracles. Oracles are services that provide facts for which they are ready to vouch:

• Facts: It can be some reference data like the winning team in a basketball game, imagine having a sport events betting CorDapp. Or some sort of calculation that the oracle computes before returning it to your node.
• Vouched for: The oracle will attest to the validity of that fact by signing it.
• Services: This is how they are typically implemented in Corda, and, by amazing coincidence, you learned about how to implement a service in the previous chapter.

You have broad latitude when it comes to implementing your oracle. However, it has to implement at least two functions, in the business sense of the word:

• Query: which lets other nodes learn about valid values. For instance, a node that can expect payment when a temperature gets high will poll, or query, the temperature oracle at regular intervals to see if time has come to be paid.
• Sign: which provides a way for nodes to have their transactions signed by the oracle, or rather those the oracle deems worthy of its imprimatur.

Just like flows, this service piggybacks on the peer-to-peer network of Corda in order to organize a business workflow.

Oh, if you ask the oracle to sign your transaction, will the oracle not learn about your private dealings?

Transaction structure

If you sent the oracle your whole transaction, then yes, it would learn about the whole thing and that would probably be an unacceptable compromise of confidentiality. To remedy this issue, there is what are called transaction tear-offs, whereby parts of the transaction have been removed so only the necessary part remains.

Before looking at the implementation, it is worth pointing out that in Corda a transaction is a Merkle tree. And the transaction id is none other than its Merkle root hash. Additionally, signing a transaction really means signing its Merkle root hash.

What are the leaves of this Merkle tree?

The above image is taken from the documentation.

The leaves are not individual commands or inputs, but component groups. You see:

• Inputs are grouped together into the inputs group component, to form the inputs leaf.
• Outputs into the outputs group component, to form the outputs leaf.
• And similarly for the others: references, command data, attachments, notary and time-window.

For absolute reproducibility, the order in which component groups are added as leaves to the Merkle tree is crucial and therefore clearly codified. By the way, you can infer from their position in the list that references came late, in fact in Corda v4, same for parameters. Putting new groups at the end ensures a fair degree of backward hash compatibility.

As it happens, each component group is itself organized as a Merkle tree. So by convention, there is one so called top-Merkle tree, and one so called sub-Merkle tree per component group, whose roots are violet in the diagram above. Unsurprisingly, the leaves of each sub-Merkle tree are the individual objects of interest: inputs, outputs, etc., plus nonces to thwart brute-force guessing.

Given the properties of Merkle trees that you learned in the first module, you see that you could selectively remove the inputs, or the outputs, or any other component group that you do not want to disclose to the oracle. If you give only the component group hashes, the oracle is still able to reach certainty that the hash is a transaction Merkle root. Such a truncated transaction is called a filtered transaction. With removed parts blacked out, this is how it could look:

Fact in a command

On the other hand, if you filter out everything, with no visible information left, then send it to the oracle, why would the oracle blindly sign the Merkle root? You already know that the required signers are kept with the command, and the oracle is in effect a required signer. So, the convention is to put the oracle fact in a command, and keep this one command visible in the transaction. You can choose to filter out fewer things, or none at all, especially if they are relevant to the oracle. But, beware that oracles are usually coded to reject filtered transactions with more information than is strictly necessary. And, your oracle should reject in this case as well. The reason will become clear shortly.

Adding data to commands is not entirely new. You already saw that being done with the Tokens SDK commands.

With all this, the question is, is it safe for the oracle to sign a transaction with partly hidden information? After all, if the oracle happens to have some valuable tokens owned by its signing public key, you could slip in a token Move command so that the oracle signs off on transferring their own tokens away. This is possible. Remember that:

• When you sign, you sign the whole transaction. Its Merkle root hash to be more precise.
• And, the contract checks that the proper signers are marked as required by the commands. It is easy for someone to add the oracle to the Move command signers.
• Finally, the Corda nodes will finalize this transaction if all required signers have signed. It does not matter on how many commands the same signer is required to sign, a single signature is sufficient.

How can you solve that very real issue?

Separate signers

The Merkle tree transaction is already prepared to protect oracles against this attack. There is, in fact, a separate component group, the signers group. Let’s go back to how the transaction builder adds a command. It needs:

• A CommandData, e.g. your Swap command.
• A List<PublicKey>, i.e. your required signers.

You add them together because they are logically linked. However, when the transaction is prepared for serialisation, a split happens:

• The CommandDatas go into the commands component group.
• The List<PublicKey> go into the signers component group in the same order.

The goal here is to let you filter everything except the different signer lists. So, when you filter out of your transaction, all CommandData except the Swap one, all the required signer lists are still there in the signers component group, in the same order. The oracle can then verify that its public key is present in a single required list, i.e. the one at the same index of the Swap command.

There is in fact a function that does that for you, that you must call as well, like so:

myFilteredTransaction.checkCommandVisibility(myOracleKey);

It assures you that if a CommandData was removed and your key is required for it, then it will throw. So, to protect your oracle fully you should also refuse filtered transactions that have more than the strictly necessary commands.

Admittedly, since by default your filtered transaction includes all signers, this will be perceived as a privacy leak by some.

Allow only the strictly necessary

To achieve that, there is a helper function and you call it like this:

myFilteredTransaction.checkWithFun(element -> {
    if (elementIsStrictlyNecessary) return true;
    return false;
});

Where elementIsStrictlyNecessary is pseudo-code for your complete verification of the element.

Of course all this verification trouble is only worth going through if the partial Merkle tree at least checks out. If it was invalid, it could mean signing a very real, and possibly dangerous, Merkle root hash, after having checked dummy leaves made to fool your incomplete oracle into passing the above tests. So, your oracle should always start with a straightforward:

myFilteredTransaction.verify();

This concludes the introduction of transaction tear-offs. Tear-offs can be used in other settings, for instance to prove to others that you spent a token, and prove only that. Or, other reporting situations.

In fact, there is a common usage that you have been using from the start, it is that of a non-validating notary. The filtered transaction such a notary sees would look like:

Where the inputs are actually StateRef, i.e. they do not disclose the actual content of the inputs.

Example preparation

After all this theory sprinkled with a bit of code, let’s look at an example. You can find the code for it here. Let’s say you want to implement a foreign exchange oracle that gives current quotes for currency pairs Let’s define the FxQuote data type that takes in:

@CordaSerializable
public class FxQuote {
    @NotNull
    private final TokenType base;
    @NotNull
    private final TokenType counter;
    @NotNull
    private final BigDecimal rate;
    @NotNull
    private final Instant expirationDate;

Don’t forget @CordaSerializable, and define a Swap command in your new FxContract:

public class FxContract implements Contract {
    // Implement verify for the right ratios
    [...]
    public interface Commands extends CommandData {
        class Swap implements Commands {
            @NotNull
            private final FxQuote quote;

            public Swap(@NotNull final FxQuote quote) {
                //noinspection ConstantConditions
                if (quote == null) throw new NullPointerException("quote cannot be null");
                this.quote = quote;
            }

            // Plus accessor, equals and hashCode
        }
    }

Since you intend to let the exchange of tokens be controlled by the token contracts themselves, you need a dummy state to attach the FxContract in a transaction.

@BelongsToContract(FxContract.class)
public class FXState implements ContractState {

    @NotNull
    private final List<AbstractParty> participants;

    public FXState(@NotNull final List<AbstractParty> participants) {
        //noinspection ConstantConditions
        if (participants == null) throw new NullPointerException("participants cannot be null");
        this.participants = participants;
    }

    @NotNull
    @Override
    public List<AbstractParty> getParticipants() {
        return participants;
    }
    // Plus equals and hashCode.
}

Filtered transaction example

What type of FilteredTransaction does the developer of the oracle intend to receive? Let’s assume that the developer made public this utility function to inform you:

public class FxOracleUtilities {
    @NotNull
    public static FilteredTransaction filter(
            @NotNull final WireTransaction tx,
            @NotNull final AbstractParty oracle) {
        //noinspection rawtypes
        return tx.buildFilteredTransaction(element -> element instanceof Command
                && ((Command) element).getSigners().contains(oracle.getOwningKey())
                && ((Command) element).getValue() instanceof Swap);
    }
}

That’s how you build a filtered transaction. You pass a predicate that tells whether the individual element makes it in.

Example query flows

The developer of the oracle also made public an example of flow pairs for you to both query and sign. The handlers of both run on the oracle’s node. First the ones to query for a new quote. Here is a CDL of what it does:

Inside an interface Query for encapsulation, the requester:

@InitiatingFlow
class Request extends FlowLogic<FxQuote> {
    @NotNull
    private final TokenType base;
    @NotNull
    private final TokenType counter;
    @NotNull
    private final Party oracle;

    [ constructor ]

    @Suspendable
    @Override
    public FxQuote call() throws FlowException {
        return initiateFlow(oracle)
                .sendAndReceive(FxQuote.class, new Pair<>(base, counter))
                .unwrap(it -> it);
    }
}

Simple. It sends the currency pair and expects a quote in return. That’s a query indeed. On the oracle node’s end, the responder:

@InitiatedBy(Request.class)
class Answer extends FlowLogic<FxQuote> {
    @NotNull
    private final FlowSession requesterSession;

    [ constructor ]

    @Suspendable
    @Override
    public FxQuote call() throws FlowException {
        //noinspection unchecked
        final FxQuote quote = getServiceHub().cordaService(FxOracle.class)
                .getQuote(requesterSession.receive(Pair.class).unwrap(it -> it));
        if (quote == null) throw new FlowException("Unavailable pair");
        requesterSession.send(quote);
        return quote;
    }
}

It’s not overly complicated either. It receives the pair, retrieves the FxOracle singleton instance from the system, asks it for a quote and sends it. That’s a lot of data passed along. We still have to define FxOracle obviously, with:

• .getQuote as the oracle’s query function.

Example sign flows

Now that query, the first function of an oracle is covered by flows, the developer of the oracle made public an example of flow pairs for this sign function. Here is the CDL of what it does:

Inside an interface Sign for encapsulation, the requester:

@InitiatingFlow
class Request extends FlowLogic<TransactionSignature> {
    @NotNull
    private final Party oracle;
    @NotNull
    private final WireTransaction tx;

    [constructor]

    @Suspendable
    @Override
    public TransactionSignature call() throws FlowException {
        return initiateFlow(oracle)
                .sendAndReceive(TransactionSignature.class, FxOracleUtilities.filter(tx, oracle))
                .unwrap(sig -> {
                    if (sig.getBy().equals(oracle.getOwningKey())) {
                        tx.checkSignature(sig);
                        return sig;
                    }
                    throw new IllegalArgumentException("Unexpected key used for signature");
                });
    }
}

It starts to get interesting again. The flow:

• Sends a filtered transaction, FxOracleUtilities.filter(tx, oracle), not the full one, thanks to the utility.
• Receives a signature in turn.
• Verifies that the signature is from the oracle indeed, and applies to the transaction.

What about the responder:

@InitiatedBy(Request.class)
class Answer extends FlowLogic<TransactionSignature> {
    @NotNull
    private final FlowSession requesterSession;

    [constructor]

    @Suspendable
    @Override
    public TransactionSignature call() throws FlowException {
        final FilteredTransaction partial = requesterSession.receive(FilteredTransaction.class)
                .unwrap(it -> it);
        final TransactionSignature sig;
        try {
            sig = getServiceHub().cordaService(FxOracle.class)
                    .sign(partial);
        } catch (FilteredTransactionVerificationException | ComponentVisibilityException e) {
            throw new FlowException(e);
        }
        requesterSession.send(sig);
        return sig;
    }
}

Back to buck passing, with a touch of exception catching. Here, you need to implement an aptly named sign function in your oracle:

• .sign as the oracle’s sign function.

Before moving on to the oracle proper, it is worthwhile to have a CDL for a typical end-to-end oracle workflow:

If your business requires that you query the oracle, scrutinize the returned value, then decide whether to move on to a signed transaction, then, by all means, make 2 flows, a la SalesProposal offer and accept.

Example oracle

The oracle runs on a dedicated server. When you, as a user have access to the above state, contract, utility and flows, you can work with it. The oracle code itself need not be public. Anyway, here goes for the oracle:

@CordaService
public class FxOracle extends SingletonSerializeAsToken {
    private final AppServiceHub serviceHub;
    private final PublicKey oracleKey;

It is a service after all. For the exercise you are going to make it simple and suppose it keeps a simple map of quotes:

public static final Duration VALID_DURATION = Duration.ofSeconds(20);
private final Map<Pair<TokenType, TokenType>, BigDecimal> rates = new HashMap<>(10);

Because it will be asked to sign on previously sent quotes, it needs to remember what was sent:

final Map<FxQuote, Boolean> sentQuotesMap = new HashMap<>(10);
[constructor]

private final LinkedList<FxQuote> sentQuotes = new LinkedList<>();

private void popOldQuotes(@SuppressWarnings("SameParameterValue") int countLeft) {
    while (0 < countLeft) {
        final FxQuote old = sentQuotes.peekFirst();
        if (old != null && old.getExpirationDate().isBefore(Instant.now())) {
            sentQuotes.removeFirst();
            sentQuotesMap.remove(old);
            countLeft--;
        } else {
            countLeft = 0;
        }
    }
}

public static final int POP_COUNT = 2;

private void loadRates() {
    this.rates.put(createPair("USD", "CAD"), BigDecimal.valueOf(1.39));
    this.rates.put(createPair("CAD", "USD"), BigDecimal.valueOf(0.72));
}

@NotNull
private Pair<TokenType, TokenType> createPair(
        @NotNull final String base, @NotNull final String counter) {
    return new Pair<>(FiatCurrency.Companion.getInstance(base),
            FiatCurrency.Companion.getInstance(counter));
}

As seen in the flow, it needs to be able to inform:

@Nullable
public FxQuote getQuote(@NotNull final Pair<TokenType,TokenType> pair) {
    final FxQuote quote = new FxQuote(
            pair.getKey(),
            pair.getValue(),
            rates.get(pair),
            Instant.now().plus(VALID_DURATION));
    popOldQuotes(POP_COUNT);
    sentQuotes.addLast(quote);
    sentQuotesMap.put(quote, true);
    return quote;
}

Notice how it does a bit of cleaning up before sending something back. It is not aimed at being threadsafe here. It also needs to be able to sign:

public TransactionSignature sign(@NotNull final FilteredTransaction ftx)
        throws FilteredTransactionVerificationException,
        ComponentVisibilityException {
    // Check that the partial Merkle tree is valid.
    ftx.verify();

    // Is it a valid Merkle tree that the oracle is willing to sign over?
    if (!ftx.checkWithFun(this::isCommandWithCorrectFXRate))
        throw new IllegalArgumentException("Oracle signature requested over an invalid transaction.");

    // Check that the sender of the transaction didn't filter out other commands that require
    // the oracle's signature. I.e. don't be stolen.
    ftx.checkCommandVisibility(oracleKey);

    // Sign the transaction.
    return serviceHub.createSignature(ftx, oracleKey);
}

Take note of the 3 validations performed before the signature is created and returned. There remains only to disclose isCommandWithCorrectFXRate:

private boolean isCommandWithCorrectFXRate(@NotNull final Object elem) {
    if (elem instanceof Command && ((Command) elem).getValue() instanceof Swap) {
        final Swap cmdData = (Swap) ((Command) elem).getValue();
        final FxQuote quote = cmdData.getQuote();
        final Boolean isMyQuote = sentQuotesMap.get(quote);
        // Check that the oracle is a required signer.
        return ((Command) elem).getSigners().contains(oracleKey)
                // Check that the enclosed quote was indeed provided by the oracle.
                && isMyQuote != null
                && isMyQuote;
    }
    // We don't want to jinx checkCommandVisibility.
    return false;
}

You see, it only accepts a Swap command, and it must contain a previously queried quote.

Further ideas

Of course, your oracle can choose not to keep track of what it sent. In the case of a temperature oracle, see example here, it could instead:

• Check that the temperature in the command is the actual temperature, within some tolerance.
• Check that the time-window’s untilTime is no more than, say, 5 minutes in the future.

Your oracle could also require proof of payment before signing the transaction. That would involve:

• Confirm that the visible outputs cover the fee.

Reference Links

• Oracles overview.
• Writing oracle services.
• Transaction tear-offs: Explains Merkle trees and filtered transactions again.
• Oracle example.