 |
algotradelib 0.0.0.9000
C++ header-only library for algorithmic trading
|
|
algotradelib 0.0.0.9000
C++ header-only library for algorithmic trading
|
This is the operational half of the order journal: what to turn on before you leave a runner unattended, what the file looks like at four in the afternoon when something has gone wrong, and how to read the five verdicts without talking yourself into the wrong one.
Read it before the first unattended session, not after the first incident.
An empty indeterminacy ledger does not mean nothing is owed. It means nothing is remembered.
runner::IndeterminacyLedger is what stops a runner sending a second order for an instrument whose first order might exist. It lives in memory, so a restarted process starts with an empty one — and an empty one is indistinguishable from a clean slate. Worse, an order that was in flight at the moment of a SIGKILL never reaches runner::outcome_of at all: no error path runs, so nothing was ever marked, not even in the ledger that then died.
The journal exists to make that difference visible on disk.
Four steps, and two of them are easy to get subtly and silently wrong:
Step 2 is the one that fails silently. live::bind constructs its own SessionExecutor internally, so wrapping one yourself and leaving it in a variable beside the runner does nothing at all — the runner keeps sending through the plain executor, the journal stays empty, and nothing anywhere reports a problem. bind_journaled exists so that mistake is unexpressible.
Step 4 is the one people forget. Miss it and the journal records faithfully and resolves nothing: every recovered instrument stays withheld forever, and you will conclude the feature is broken when it is merely deaf.
Step 3 does more than mark the ledger. Three kinds of damage cannot be expressed as a mark on one instrument, so recover_runner halts the runner outright for each — local_failure() is set and the loop gates on it:
Clear any of them with resume_after_local_failure() once you have read the broker's list yourself. That call also acknowledges the executor's failure count, which is what lets the runner tell a new storage failure from the one you just fixed — without it, the next send would re-read the first error and halt again immediately.
A backtest journals nothing, because it wraps nothing. There is no flag to remember to turn off.
One JSON object per line, appended, flushed and fsynced before the call returns. tail is the intended interface:
At least two lines per instruction, never an edit of one. The first is written before the send is attempted; the second says what happened, and a serialized runner (the default with a journal) adds a third when the placement is accounted for. A pending line with no companion is the signal.
at is the record and at_text is the caption: the integer is exact on every platform and is what matching reads, the text is for you.
crc is always last. It is the CRC-32 of the line as it would read without it — the whole JSON object, closing } included — after which that brace is replaced by ,"crc":"…"}; to check a line by hand, cut the member and restore the brace. A line whose bytes changed after they were written — a bad sector, a partial overwrite, an editor — fails it and is not read at all: it counts as an unreadable line, and recovery halts the runner outright. That matters for the damage no reader of values could see, such as 005930 becoming 005931, a real instrument. So do not edit the file: a hand-edited line fails its checksum, and so does a key this library never wrote.
A journal this release begins says so in its header ("integrity":"crc32"), so every entry in it must carry a checksum. A journal begun by an earlier release keeps its checksum-less lines readable as they always were; once a checksummed line appears, a checksum-less one after it is damage. The one line this cannot protect is the first checksummed line of such a file, which could lose its whole crc member and pass as an older release's. It is a checksum against corruption, not a signature against someone rewriting the file on purpose.
Rolling back across this release is not supported on a live journal. An earlier release reads a checksummed line (it ignores crc) and appends checksum-less ones; this release then halts on them, as it should on a line it cannot vouch for. To roll back, or to roll forward again afterwards, start the other release on a new journal file, and only once the old journal may be given up. A resolved instruction is not enough for that: a sweep that found nothing writes resolved while a serialized runner still owes the order, and one that found the order names it while it may still be working. Every order the journal tracks must have ended (filled in full, or cancelled with its history row saying so) and holdings must show its final fills, as for rotation under "What this does not do yet".
| What you see | What it means |
|---|---|
| pending then accepted | the order exists, and order_id names it |
| pending then rejected | the instruction was refused — by the broker, by the binding or client before it sent, or by local machinery that failed. The order does not exist. Nothing owed |
| pending then indeterminate | the outcome is unknown (a failure in transit, an unreadable reply, an HTTP 5xx, or a broker refusal reported as PROTOCOL, which is most of them; Binding a broker names the refusals that are rejected instead). Still owed |
| a later confirmed | a serialized runner accounted for the placement: holdings showed its fill, or its history showed it ended with nothing filled, or an operator released it. The placement is no longer owed; an operator's release does not clear a ledger hold, and the next order on the instrument waits for a holdings read received after it |
| a later held | evidence contradicted the placement (holdings moved past what it can explain); only an operator can account for it, with Runner::release_obligations (see the operator procedure below) |
| pending then resolved | a sweep read the broker's lists and closed the question. order_id says what it saw: a number, or nothing found. A second resolved line for the same id, with a number, is a later sweep finding the order |
| pending alone, at the end of the file | the process died between the write and the answer |
| a last line with no newline | the process died during the write |
| the runner halted on an unreadable line | a line changed after it was written — its checksum or an unknown key says so — or a checksum-less line follows a checksummed one |
The last two are the reason the format is line-oriented. A line and its newline are written in one call, so the newline is a commit marker: a torn write loses the tail of the buffer, and the missing newline says so. A file that was rewritten in place could instead be corrupt as a whole.
A torn line is not discarded. Where the symbol is still legible it is salvaged into an entry carrying requires_operator, which no sweep may decide — its id is rewritten so it pairs with no terminal line, its outcome is forced to pending whatever the text said, because outcome may itself be the field that was cut off, and the flag stops the matcher running window arithmetic on a timestamp that was never recorded. (Without the flag that arithmetic resolves it instantly: the missing timestamp reads as the epoch, so the window is long closed and no real row can fall inside it, and the first sweep would report not_placed.) Where the symbol is gone, recover_runner halts the runner outright, because there is no instrument to withhold and the startup gate is not enough — that gate lifts after one order-list read, and a read cannot answer a question nobody can pose.
Opening the journal repairs a torn tail before writing anything. Appending straight onto a line with no newline would glue the next record to the fragment; the combined text would then carry a newline, so the next replay would see one unreadable complete line and lose both. So opening terminates the fragment and writes the salvaged record beside it — and if that repair itself fails, open() is false.
After recovery, each owed instruction is matched against the broker's own order list on every listing the source reads. runner::match_verdict_name prints these:
| Verdict | What it means | What happens |
|---|---|---|
| matched | exactly one row corresponds | resolved; the ledger lets the instrument go, and the runner counts the order until holdings account for it |
| not_placed | no row, and the window has closed | resolved: no order was found. A runner that overlaps orders forgets the placement; a serialized runner keeps it owed (below) |
| ambiguous | more than one row could be it, a row that fits is one the journal records for another of your placements, or a row that fits lacks the evidence to be told from another order | nothing is resolved; the next sweep asks again, and you decide if it stays |
| undecided | no row yet, window still open | wait for the next sweep |
| needs_operator | the entry came from a damaged line, or its line cannot be matched (a cancellation with no order id, a placement whose price does not fit its type) | only you can resolve it |
A resolution is written back to the journal before the ledger is cleared — the same discipline as writing before a send, applied to the other end. Without it the instruction would stay open on disk and every restart would withhold the instrument again for a question already answered. If that write-back fails, the hold deliberately stands and the runner halts.
The record's outcome is resolved, not accepted or rejected. Those are answers the broker gave to the instruction; a sweep learns something different — what the listing shows now. On a cancellation they can be opposites: a matched row means the target order was in the lists read, and still resting when no history was read with them, so accepted would claim a withdrawal that may plainly not have happened. What was observed is in order_id (set when a row was found, empty when none was) and spelled out in detail.
not_placed is a finding, not the broker's word. It says that no row of the lists read matched within the window. A list can lag an order it accepted, and no broker here publishes "every order accepted so far is listed". A serialized runner (Runner::serialized, the default for one that journals) therefore keeps a placement found not_placed owed, unnamed: the journal's question is closed, the instrument still places nothing, live::SessionSource looks for the order's dated row again at every order-list read, and failing that only Runner::release_obligations ends it. A loop stops placing altogether once such an order has been owed for ten minutes (PollingLoop::set_owed_order_limit; running a strategy live has the procedure). A runner that overlaps orders forgets the placement at not_placed, which is why its window must be generous.
not_placed and undecided are the pair to get right. The absence of a row says anything about the order only after enough time has passed for the broker's listing to reflect a write it accepted; read too early, it says nothing at all, and even then it is the finding described above and not proof. That is what the window is for, and it is three durations rather than one tolerance because it absorbs three unrelated things:
Which clock says the window has closed. The later of the runner's decision instant (the period end of the last bar fed, or a quote's time) and the loop's last reading of its own clock. So under a loop a restart can resolve not_placed at its first read instead of waiting for a bar, and on daily bars, where the decision instant is the previous session's end, a placement left in doubt is decided as the loop's clock passes the window, not at the next bar. A source driven without a loop has only the decision instant, and the wall clock stands in before anything was fed. Both have to be wall time on the journal's clock: one that runs ahead shortens the window.
ambiguous is a stop, not a retry. Nothing is written for it, and every sweep asks again, so an ambiguity that comes from missing evidence can end by itself (a Kiwoom order listed before its history row arrives matches once the row does). Two rows that could both be your order stay ambiguous however many times you sweep; no heuristic turns them into one answer. Meanwhile the instrument stays withheld, and a serialized run stops at the owed-order limit. Go and look, then clear the ledger entry yourself with ledger().resolve_after_reading_orders(instrument) — named for what it requires, because calling it without having read that list defeats the entire mechanism. If the order was this run's own, you must also tell the runner what you found: Runner::resolve_indeterminate_placement(instrument, side, order_id) with the broker's number, or with no number if it was never placed (a serialized runner takes a number only with the order's journal id, and ignores "never placed": use Runner::release_obligations). Otherwise the run keeps counting it as a reserved row with no reference until holdings show it (a resting order is then counted beside its own listed row, and a target may withdraw it), and if they never do, it withholds the instrument again once it is past the confirmation timeout from its send, at the next holdings read (counted in orders_unconfirmed_timed_out()).
It cannot manufacture idempotency. A journal narrows the window and makes ambiguity machine-visible. Only a broker can promise that a duplicate is discarded, and none of the three supported brokers offers that on a recovery path:
So placements are matched on attributes — instrument, side, type, quantity, price, and a timestamp inside the window — at every broker. That is a heuristic, and ambiguous is how it tells you so.
A row the journal already gives to another placement is not this one's. Two orders of the same shape sent within the window — the first accepted with its number, the second's request lost — leave the first order's row inside the second's window. The sweep does not name the lost order with that number: the row is the other placement's by its accepted line (or by an earlier sweep's name), so the verdict is ambiguous, never matched and never not_placed, and a serialized run stops for the order at the owed-order limit. A number counts as the other placement's only on a row inside that placement's own send window, since some brokers reuse order numbers on another day. A row with no time (every KIS real-account working order) cannot be placed outside that window, so where a number is reused, an older placement of the same shape that recorded it makes such a row ambiguous: you reconcile it.
A row whose day nobody stated proves no day. Kiwoom's working-order listing gives a time of day and no date; the adapter dates it today and says so (OrderSnapshot::ordered_date_assumed_), and the demo server has listed the previous day's order after midnight. Where the history a sweep reads has the order's own row (the same number at the same time of day, nothing either row states disagrees, and the price stated alike on both), that dated row stands for it, on its own evidence. Until then the listed row neither matches an order nor lets one be found absent (ambiguous), and the later search for an order still owed ignores it.
A placement is matched against order history, not against the open-orders listing. This is the subtlest thing in the whole design. Each broker's working-order inquiry returns only what is still resting: Toss maps it to its OPEN scope, KIS to the modifiable set, Kiwoom to its unfilled-order list (ka10075). A market or aggressive limit order can fill and leave that listing within seconds — well before the first sweep — so matching against open orders alone would find nothing, and once the window closed that absence would read as not_placed. The hold would clear for an order that existed and executed, and the runner would place it again. So any sweep with a placement owed reads the history too, with the calendar range KIS requires, the cursor paging Toss offers and one date at a time for Kiwoom, which refuses a range, and deduplicates by order id because a still-open order appears in both listings.
History is read only when something is owed, so an ordinary sweep costs no request of its own. And a history read that fails concludes nothing at all: absence from a listing that never arrived is not evidence, so every instruction stays owed.
A cancellation needs no history. It names an order id, and whether that order is still resting is exactly what the open listing answers (a sweep that also owes a placement reads history too, and then a matched row says only that the order was listed).
Cancellations are matched by identity, because a cancellation names an order that already exists, so its journal line carries the broker's own order_id from the moment it is written. ambiguous comes only from two rows carrying that id (an order number a broker reused on another day).
Two identical orders in the same second are already prevented, which is why second-granularity timestamps (KIS reports ord_tmd as HHMMSS) are not the problem they look like. Once an instrument is marked, the ledger withholds everything for it, so a second identical order cannot be sent while the first is unresolved.
examples/example-order-journal.cpp runs the whole story with no credentials, no network and no signal raised: a send that never returns, a file holding one pending line, a fresh ledger that would happily trade until recovery stops it, and two sweeps that turn undecided into matched.
1.18.0