The numbers we invented belonged to no one

Three days went into looking for the mistake in the file we were exporting. On day 4 it turned out the mistake was not in the export but in the numbers we had made up for our tests. The tax office was not rejecting our XML. It was rejecting a counterparty that did not exist.

Some background. When you file a withholding certificate to Coretax in Indonesia, there is a field for the counterparty's tax number. That field is not satisfied by a number of the right shape. The receiving side checks whether the number is in the registry. What our screen can check is the digit count and where the separators go; whether anyone owns that number is not something we can answer from here.

What we were trying to build was a nightly automated test. Every night at a fixed hour, the machine generates data, produces the whole set of certificates and runs it through submission, so that in the morning we only have to read the result. With that in place, the day we break something is the day we find out. The numbers we generated were correct in form: 16 digits, separators in the right places, check digit computed and attached.

On 18 August, 12 of the 41 test cases stopped at the same point. The message that came back was short, and it read as though the content we had sent was malformed. It did not say which item was wrong. I suspected our own export. The ordering, the namespace, the way we wrote the dates. For 3 days I looked only at the XML.

On day 4 I put the 12 that stopped next to the 29 that passed. The ones that passed carried numbers taken straight from documents V had actually handled. The ones that stopped carried numbers we had made up on the spot. The difference was not in how we wrote the file. It was whether the number existed. The office was checking: is this number in the registry, and is it still valid.

I told V about it. She is an accountant in Jakarta and has tested KAZENA Books more than 20 times with the real data of her own practice. Her reply was short. A real invoice always has a real counterparty on the other side. Something too obvious to say out loud in actual work was the thing we had forgotten, and we had forgotten it only inside our tests.

The same goes for faktur serial numbers. The serial a PKP puts on an invoice is not a number you decide for yourself; it is allocated to you by the tax office. A number thought up at a desk is either allocated to someone else or allocated to nobody, and in neither case is it yours. So a plausible-looking number failing to go through is the system working. If it did go through, that would be the real problem.

The shape of our mistake is clear. We had made a passing test the goal. When you try to force through something that should not pass, your hand always reaches toward weakening the check. I did in fact look once for a setting that would skip the registry lookup. That I looked at all is an error of direction. What a test should be asked for is not that it passes, but that it stops correctly when it ought to stop.

Here is what we changed. We split validation into 2 layers. The first layer is what we can answer ourselves: digit count, separators, check digit, required fields, whether the line items add up to the total. All of that is checked here, before anything is sent. The second layer is what only the office holds the answer to: whether the holder of that number exists, whether it is valid now, whether that serial was allocated to us. That cannot be known until we submit. We stopped pretending we know it.

We changed the words on the screen as well. It used to say only that submission had failed. Now it says which number, and who rejected it. When the screen says the counterparty's number could not be found in the registry, what the user does next changes. Reviewing your own input and asking the other party to confirm their number are 2 entirely different actions.

How we run the tests changed too. What runs nightly is the first layer only. The path that submits to the office is run deliberately, on the machine of someone who holds real numbers. Before a release, my colleague E goes through it on his own device. He is in a town some way outside Jakarta, and he checks on the connection and the handset that are actually used there. We drew a clear line between the part a machine can run every night and the part only a person can.

We replaced the test data itself. Our work records still held numbers copied from real documents. I had thought using them in tests was harmless, but they are someone else's numbers. They are now replaced with numbers you can see at a glance belong to nobody, and no real number sits inside our tests. The first layer runs fine that way. The second layer was never meaningful with anything but the real thing.

I should say this plainly. I am not a tax accountant. Digit counts, forms and the way lookups are done all change, and which number a given document needs varies with the nature of the transaction. What is written here is a record of how we fixed our own way of testing. Please confirm the actual judgment with a professional.

A request. If you have hit a screen in KAZENA that says only that something could not be sent, and you could not tell what to do next, please tell me. Stopping can be the correct behaviour. But whether the reason for stopping lies with us or with the other party in the document is something we should be writing down. If it is not written, that is our failure.