Tuesday, 8 April 2014

Flash Boys' sinister ending: who owns the new evil microwave tower?

At the conclusion of Flash Boys, Lewis holds out a challenge to find who owns the tower with the mysterious FCC license plate number 1215095.

"The application to use the tower to send a microwave signal had been filed in July 2012, and it had been filed by ... well, it isn't possible to keep any of this secret anymore. A day's journey in cyberspace would lead anyone who wished to know it into another incredible but true Wall Street story, of hypocrisy and secrecy and the endless quest by human beings to gain a certain edge in an uncertain world." 
 Part of the last paragraph of Flash Boys

This final piece of the narrative paints this particular microwave New New Thing as the next, somewhat sinister, step in the evolution of speed. Spread Networks was one piece of secretive infrastructure used to help rig the markets. Now it is the turn of evil microwave. Who owns this new form of evil to help the HFT pirates continue to rig the financial markets?

The internet has reacted! A bit of frenzied internet sleuthing has subsequently taken place over a number of blogs, tweets and web sites. A number of parties have reached the same conclusion.

Tradeworx! Mr Narang confirmed it to the WSJ. They're the new bandits!

Sigh... I don't think so. Read on.

I found the ending of Flash Boys one of the saddest bits of the Lewis book. I feel it was sad for journalism, unless you support deceptive journalism.

Mr Lewis goes to great pains to weave a narrative around Spread Networks. I think you'd have to agree it is a key part of the plot. He then throws out the New New Thing of microwave at the conclusion of the book with clear sinister overtones. You could almost hears the Jaws theme in the background. Good people have been hunting down who could it be doing this New New Thing of evil that was the evolution of speed after Spread Networks? A participatory cliff hanger to ferret out the perpetrators was a neat literary device.

Surely it would have been responsible journalism for Lewis, especially when writing on the scale of a book, a Lewis book nonetheless, to “google” the topic so he could have found out that his narrative was wrong and RF microwaves had been used on the Illinois – New Jersey link since 2009, BEFORE spread networks went live.

It took me less than thirty seconds to get a reference to a credible microwave 2009 story. Here's an article I've tweeted previously from the Chicago Tribune from 2012.
"He [Benti] said the microwave network starts at 350 E. Cermak, ends at another telecom hotel at 165 Halsey St. in Newark, N.J., and went live in the fourth quarter of 2009."
Did he ask Spread Networks? Lewis can remember hand movements and idle moments from conversations of meetings with enough clarity to include much minutiae in the book. Maybe Spread Networks didn't know about microwave when Lewis talked to them?

From page two of the Chicago Tribune article,
"Spread Networks CEO David Barksdale dismissed such criticism. Yes, data travels faster through the air than through fiber-optic lines, but severe weather events can disrupt microwave signals, and those signals can't carry as much data at one time as fiber can."
I think they knew.

Is it shoddy journalism from Mr Lewis? Perhaps he didn't want these particular facts to get in the way of his significant plot device. Now, if Lewis had included the microwave links from 2009 in the story, it wouldn't have necessarily changed his message. He should have been more aware and not so inaccurate. If Lewis can't get the basics of his main Flash Boys narrative right, you need to take everything else with a pinch of salt.

Also, why is Tradeworx copping flak for this? This fact of Tradeworx owning microwave was been public for sometime. Wired ran a high profile story in 2012 that even had a big coloured box with the Tradeworx name on it. There was even a price tag! Hardly a complete secret.

From the wired article with Tradeworx hidden in plain sight
In the words of the great Lleyton Hewitt, “Come on!”

There is no news here, just scuttlebutt and harmful innuendo. Lewis is implying Tradeworx is rigging the market. They are not. They're just collateral damage in Lewis' missteps.

How can we trust the story telling if the main plot device is so twisted out of shape?

Ought Lewis have known more about microwave?

Jim Barksdale is the investment force behind Spread Networks. Is he also an investor in IEX? Can't be sure, but his fellow ex-Netscape CEO Jim Clark is on the IEX board. Obviously these guys are acquaintances of Lewis from the New New Thing days. A Barksdale, David, remains CEO of Spread. Lewis could have asked them. Any of them. Based on the Chicago Tribune story, they ought to have known.

Lewis didn't even need his contacts here. He could have clicked "search."

Incompetent or deceptive?

You decide.


An example of the disclosed use of Spread Networks fibre:


Getco use of Spread from their S-4:
"Colocation and data line expenses increased $18.9 million (52.0%) to $55.2 million in 2010 from $36.3 million in 2009 primarily due to the introduction of Spread Networks, which is a fiber optic line that transmits exchange and market data between Chicago and New York, and the build out of GETCO’s Asia-Pacific colocations and data lines."

Monday, 7 April 2014

HFT - two choices: making money every day or oblivion

If an HFT makes money every day the system must be rigged!

It's not rigged.  It's just math!

I see since Flash Boys came out, HFTs have been copping flak for making money consistently.

Virtu had one down day in 1200 or so days. It must be rigged.

Well, the simple fact is that if you don't make money every day, then you're doing it wrong!

You do, or you die. It's that simple.

HFT implies that you do a lot of trades. If you do a lot of trades with a smidge of favourability, you should always be winning. It's just simple mathematics. The law of big numbers. A systematic screw-up for a day may cause an odd loss but that is a different risk category. You seriously wouldn't be doing a zillion trades of anything without an overall positive expectation. The law of big numbers also works against you. If you're on the wrong side of the fragile line you're practically guaranteed to lose.

This is not to say that you have a "cooked" system where every trade is a winner.

Let's look at some numbers:

Say you have a trade that makes you $0.01 per trade when you get it right and you lose $0.011 per trade when you get it wrong.

Let's now assume you do 10,000 trades a day.

What winning trade % do you need to win at for almost every day to be a winner?

Let's look:
  • 51% means roughly 99.7% of days are losers, a bit over 0.2% of days are winners
You probably wouldn't be happy with that.
  • 53% means 89% of days are winners
  • 54% means 99.9% of days are winners - you lose one day in four years
  • 55% means you really shouldn't lose
The percentages are tight. It is a matter of getting the profit / loss distribution for your trade outcomes right so a lot of trades make it hard to lose money.

Fundamentally, if you're a high frequency trader, you're doing a lot of trades. That's pretty much the definition of high frequency, otherwise you're a low frequency trader. A lot of trades are silly if you don't have a positive expectation. If you have a positive expectation then a lot of trades means you really should have a very high chance of winning on every day even if each trade's win chance is a bit like the toss of a coin.

Let's look at the same situation if you only did 100 trades a day. A hundred is quite a few but not really high frequency.
  • 51% trade win probably = about 38% of days are winners
  • 54% = about 62% of days are winners
  • 55% = about 69% of days are winners
Not quite as good as having a lot of trades at 55%. However at 51% you were previously pretty much guaranteed to lose with lots of trades but you may live a few days longer with only 100 trades per day. This is also why if you want to bet on red or black at roulette, where the house has an edge thanks to zero, you should just make one large single bet to maximise your chances of not losing.

You can see that lots of trades help improve the guarantees. You win or lose more consistently.

The hard thing about HFT is that if it was easy, everyone would do it. There are no easy trades with the level of sophistication that exists in mature markets. It is challenging and you are constantly being pushed to the margins as someone else is always prepared to make a little less and take all your edge if they can. HFT market makers fight compulsively over the scraps and win by making less than another team. It's a bare knuckle fight against other market makers where your reward is to make less money than your opponent if you win. You in turn get beaten by someone prepared to make even less. You are scared of large trades or better informed traders that will trade through you and screw with your distribution of returns. There is little room for error.

It is simple. An HFT should be making money every day or they will be going out of business. There is money or there is oblivion. There is never any forgiveness, just paranoia.  HFTs are terrified of adverse selection. Worse still, other market makers are trying desperately to replace you by being a fraction better. Lots of smart firms are always shutting down in HFT land. It is a tough business.

So, what's up with traders making money every day?

It's not rigged.  It's just math.


PS: Survivorship bias for HFT types means that all should have consistent profits if they remain in business for an extended period of time. I find that an interesting thought.

Earlier description of how HFT market makers make money
Earlier reasoning as to why investors and speculators can both win in markets


Footnote:  I spent quite a few years as a positioning style trader with a global portfolio of currency, short and long interest rates, equity indices and some commodities.  I made money every year but it was a battle.  Each day was a 48% chance of a win. So, overall I was usually wrong. I was successful because the average winning day was 1.6 times the average losing day. It was pretty scary though as just missing your dozen best days in a year would be the difference between an annual profit or an annual loss. It was also quite funny when people asked for comment on the markets as it was hard to explain that with my trading I was more likely to be wrong that right for any given day.

HFT is pretty similar. It is also just a game of profit and loss distribution. It is a lot easier on the psychology as instead of waiting for a whole year to see if things work out, you should know at the end of each day with a large enough number of trades. Even worse for the psychology is that as a position style trader normally the profits are positively autocorrelated so that if you are losing you expect to continue losing. Worse, you can't do anything about because if you over trade, resulting in extra frictional costs, you're only guaranteeing an eventual loss. HFT has better daily psychology but higher paranoia as there is a lot that can go wrong.


Footnote: This is vastly simplified but correct in essence.  Profits and loss tend to be positively autocorrelated which makes things more volatile in practice. Distributions are nowhere near as simple which is why there are PhDs running around with computers attached to their fingers.

Friday, 4 April 2014

Flash Boys - Misleading information

Flash Boys?  I read the book last night.  Appalling.  I found it a well wide of the mark. It vilifies and accuses inaccurately. The pulsating vehemence of its message makes it a singly dangerous book.

I love Michael Lewis' story telling. He really knows how to write a page turner for a geek like me.

Working in an investment bank in 90s, Liars Poker had a cult status with the currency note based serial number poker he made infamous being pervasive with many traders in Sydney too. Without Moneyball there wouldn't have been a Soccernomics book to read. Soccernomics made watching football richer and, as a Liverpool fan, I'm especially liking soccer just now. So thank you Mr Lewis.

The Big Short. Terrific read.  Hmm, but what happened to Paulson? A good story but not quite the full picture. No doubt about it though, Lewis writes books that many people find fun to read.

Nonetheless, Flash Boys is a stunningly dangerous piece of misinformation. The truth is quite a bit different to the words you'll read beyond its red cover.

I know just enough to be dangerous about this as I am a former HFT. I've traded a million index option contracts in a day in Korea on system built from the ground up.  Back, in 2010, when people still used Blackberries, I got a call from Joanne, my broker in Toronto, half a world away, surprising me with the news that I had traded a bit over 13% of RIM at TMX for the day. Really? Some little tin pot firm on the other side of the planet mixing it up with big boys? Lewis is right there, you never know what strange firm may be responsible for a bunch of trading. The little firm I worked in had quite a few years of trading with only a few down days. Not quite as impressive as those firms with one or zero down days, but I'll take it. A team of mine also built a prototype sub 2 microsecond exchange matching engine and I'm the inventor on a somewhat dubious hardware matching engine patent. I like matching engines. I left HFT to do some cool tech but got trampled on spectacularly badly in court. In summary, I think I have just enough knowledge to know that there is quite a lot I really don't know. Though maybe just a enough to call Mr Lewis out on some facets of his book. Just enough to be dangerous.

The IEX guys sound like great lads but let's start by looking at how to win on their platform. Their price is delayed fairly. OK. You are lucky enough know the price is going to be always delayed and thus stale at match point. The stale pricing is guaranteed by many miles of fibre in a shoebox. Great! Try to use all your inputs to deduce an impression of a better price in real time. Throw it at them. Send in limits or IOCs at your biased price and see if you land a trade in their time warp.  Better information exists on the outside of the delayed world.  Use it.  It's still a race. Slower exchanges are always problematic as the world knows better. IEX is no different. You can be played by better information or decisions. You don't eliminate the speed race. Hmm, it's not really much different to any other venue in that regard. There is always an arb in a world that has space and time as a feature.

In theory, there is no difference between theory and practice. Unfortunately, the real world does intervene on our clever ideas. It is wrong to think that there is not a continuum of probabilities and latency where views that seem absolute can't be replaced by a bit of educated guesswork with a virtual latency gain. Things are just not as simple as Lewis makes out. Some people think bunching trades into chunky time slices may be the solution. Such time point based auctions are not the answer. Even if you traded with an auction to the minute there is still a game of maximising your information at the deadline with latency tricks. So even slowing down trading to minutes becomes a latency game. A stupidly inefficient one, but still latency sensitive. Unintended consequences abound. You really do have to be careful what you wish for.

The clever thing IEX seem to have done is simply make a co-location space the size of New Jersey that includes all the other exchanges. It is still a game where speed matters and variances exist. It's just a bit different. Other market structure trade techniques still apply to the IEX context too, it just seems those get ignored in the story as inconveniences.

The IEX guys mean well. I just believe the ultimate basis for their thinking is unfortunately a bit wrong. Not a lot wrong.  Just a bit wrong. They are smart guys but I feel they have been skewered by their misunderstanding of Thor and HFT. When you have a hammer, everything looks like a nail. (Sorry, couldn't resist.) The result is they have run off on a bit of a tangent. It sounds a nice system, but I suspect it is not quite want they intended. There are many good features and reasons why it may make sense to use IEX but you need to get over the idea that it is perfectly fair and cannot be "gamed."

Clearly, "Thor" was a great tool. It was effective. However, I don't think the SEC guys were really dishonest in questioning Thor's goodness relative to the idea of over provisioning liquidity at multiple venues. Poor form from Lewis in that regard. Tearing down ideas without proper consideration is never a good thing. The liquidity at various venues was real and could be hit individually. It wasn't fake. You must remember that the HFT market makers are scared rabbits fighting against adverse selection but needing to be stoic and fearless to maintain priority and sufficient size. It is dangerous picking up pennies in front of the market steam roller in the name of efficiency. The market maker may dare to put out more liquidity than they are comfortable with in an effort cover all the bases. Odds are it is safe enough as they shouldn't get hit all at once. Thor! Thwack! Ouch! HFT market making is a thankless and tough job. Perhaps only exceeded by the thanklessness of being an SEC official that is wrongly ripped into by Mr Lewis.

Think about it. If some dude or dudette comes along and cleans your over-provisioned liquidity clock, then, as a market maker, you'll have to adapt and put out less liquidity or otherwise change style.  Is the gaming of the simultaneous orders with Thor really better than the over provisioning of liquidity? Thor's net result is to force there to be less simultaneous liquidity at venues and more bias toward particular venues. Is that a good result? I can see both arguments. There is certainly a ying and yang there.  It is naive to consider just one point of view.

IEX should be congratulated for their endeavours and especially their impressive ethics recounted in the book.  They have certainly tried very hard and it looks a decent enough solution as an exchange. There are also plenty of other ideas about making better exchanges. Then again, perhaps Lewis hasn't really explained it properly.

There are certainly great points in the book, such as the reference to the silliness of some of the inane order types. Some order types do seem to border on the ridiculous. As always, hindsight is a wonderful thing. However, reading the book you also wouldn't know that microwave wireless on the Chicago to New York link pre-dated Spread's cute fibre perhaps as far back as 2009. You wouldn't know that BATS was also the fastest exchange back in 2009 with a round trip of about 443 microseconds.  That was an important feature. Fastest makes a difference. You wouldn't know that there are natural advantages causing liquidity attraction in a game theoretic sense for the fastest exchange. I was horrified by the depiction of the flash crash. Frankly, Lewis' depiction of the May 2010 flash crash circumstances bordered on negligence. It's all a bit loose in the fact department.

There are lots of myths and folklore in trading.  Some are frighteningly unchallenged.  Did you know solely investing in index funds is a really dangerous idea? If everyone did it, there'd be no price discovery. The market would fail. That is an obvious statement but many would find it confronting on a first read. For some ideas you don't need data, just as Einstein didn't really ride a light beam. It is wrong of firms like Nanex to ignore theoretical arguments, just as it is wrong to also ignore the data from the system. Many of the ideas that need confronting are indeed already well understood. It is important to remember a couple of key ideas: markets are best left alone to be the wonderfully efficient mechanisms they are, despite the motivations of their participants; and, complex systems, including human beings, need controls and limits. These principles are in conflict and need constant re-balancing but such ideas _and_ data can guide us.

Why has Lewis gone so wrong here? What's the agenda?

When I was 14 I read a story in the newspaper that I wanted to do Information Science at university. I didn't know what that was. I'm pretty sure I said I wanted to be a pilot.  Wrong message. I learnt at an early age that the truth of an article is often inversely proportional to how close you are to the story. Likewise, I find that I've been at least close enough to the HFT story such that Flash Boys doesn't stand up well to scrutiny.

Flash Boy's message was clear, but I don't know what the Lewis agenda is.  Perhaps he's just got it wrong? Against that, it reads like he has an agenda. Lewis needs to tell us how stuffed up the whole US equity trading system is and how you are being ripped off at every turn. Controversy sells. Hopefully Lewis is just honestly wrong and not really being Machiavellian and trying to sell a boatload of books. He has been accused of just trying to whip up a frenzy to sell his book but I for one don't believe that. He is just pretty badly wrong in my view. I'm sure he'd disagree. Though it was annoying, I found it a fun read. Just don't take it too seriously. I had similar thoughts reading both Lance Armstrong's autobiography and Flash Boys, "Should it be in the fiction or non-fiction section?"

The investor, big and small, insto and retail, needs to remember that there has never been a better time in history to trade and get great pricing on executions.

Equities trading in the US is far from perfect. The system has a lot of good features and quite a few things that need some fixing. It always will, but it's pretty damned good. Flash Boys uses many woeful anecdotes to wrongfully undermine confidence in the national market system. Proceed with caution. Flash boys is a dangerous book.

Friday, 24 January 2014

Is the NSA dragnet an unconstitutional secret success?

I have a theory that the NSA, or their data, is actually successful at stopping many crimes. Perhaps not the Boston Bombers, but many others. The trouble is they have to keep it secret.
NSA mass surveillance: a failure or just foxing?
Billions of dollars, many years, project restarts and still failure? Perhaps not. A few tech people I know concur with my thoughts that it must be really hard to fail at connecting the dots with all the data the NSA collect. How can all of those nice graphs and pools be not useful? Private companies do just fine on profiling us with less data. You'd think there would have to be a successful pre-crime unit hidden away somewhere slurping up all that good data.  I've made a living at getting signal out of noise and though their problem is large, it should not be that difficult.

Q: So why would the agency paint a picture of failure?
A: They have to.

Why? That's easy, I think.  Let me meander through.

Professor Geoffery R Stone and now the US Government's Privacy Board agree the program is unconstitutional. That makes it illegal. We all wait for the Supreme Court to make the final adjudication.

Let's say the programme is successful.  It is used by many, including the FBI, DEA, to make connections. The agencies have been caught out handing over information and requiring the case when at trial not to use that information so the source or method may remain protected. Stories abound of PIs giving their mate at the FBI a call to get information on someone. Law enforcement agencies would be tremendously excited at having such data at their finger tips.  Imagine finding all the people within a radius of a crime from their cell tower records. Now find all the people within a radius of the use of a stolen credit card. Intersect.  Call up the person's address, track them via their cell, and then get the warrant to "confirm" the cell phone records. That would be pretty useful.

So just perhaps the data is very valuable indeed.

Now the agencies have a problem.  If it is declared unconstitutional and therefore illegal then any dependent legal cases would be subject to being revisited. What a mess that would be.

The prospect of terrorists, murderers, rapists, kiddy fiddlers, kidnappers, conmen, etc all walking the streets is indeed frightening. Perhaps the government is in a bind.  You can't say it is successful otherwise those cases are endangered.

If mass surveillance by the Ministry of Truth was truly useful, as I suspect it is, then I understand the dilemma. I'm not sure I'd like to be responsible for releasing a lot of criminals onto the street. I'd be looking to solve the constitutional problem but without letting the cat out of the bag that would free the criminals.

Maybe this is why the powers that be really want to keep the programme in place and make it pass constitutional muster.

Ex-NSA staffer, William Binney of ThinThread points out that the follow on project to his did spend years and waste billions before the project was killed. Perhaps the NSA and friends are truly incompetent, but I personally doubt they are really so incompetent that they have continued to fail up to 2014.

So has the NSA wasted billions and many years on a failed project?  Is there secret value in the project that we can't be told about?

If the programme has value then perhaps Obama's speech makes more sense than we are allowed to know. Shameful indeed, but perhaps they have no choice now that they have dug their own grave.

Not committing the constitutional crime in the first place is the trite solution. The mess is perhaps a lesson in that open government rather than secret courts is the best path.

If this mess is the case, how would you fix it?

If you get linked to this story by a government official then perhaps you know the answer ;-)

Monday, 20 January 2014

Federal Court of Australia - food for thought

Honourable Justice
Michelle Gordon:
A poster child for
FCA reform?
My recent trials and and ongoing tribulations have made me think about the Federal Court of Australia quite a bit, mainly out of utter frustration. I see a system of good intentions that is quite broken. The work of many good judicial officers and staff are let down by an ambitious few or those who curry recognition via patronage to outside interests. Perhaps the people who consider reform are simply too close to the forest to see the trees? 

Let me meander along with some of my thoughts below.

1. Judges below the High Court should not have tenure or there should be a mechanism that makes it simpler than current practice for a judicial officer to be dismissed.

It may be appropriate for the Chief Justice of the Federal to have tenure but the process of appointing Federal Court judges is not sufficiently robust to support tenure.  Hon Justice Murphy, Hon Justice Davies and Hon Justice Gordon of the Victoria Registry are prime examples of judges for whom tenure was perhaps premature.

An alternate mechanism for dismissal by peers, the Chief Justice, or some other non-political approach should be available. It could be that some kind of contract or probationary role should be in place for new justices for a substantial period.

Australia is not alone here. There is recognition of this the world over. It is hard to see how appointing a solicitor without prior experience of judicial office to a tenured position at the Federal Court of Australia makes sense. This was the case is Murphy J.  It further does not strike a chord of fairness given the former working role of Murphy J & former Prime Minister Gillard and the hint of scandal around the union trust fund abuse. Our experience of justice at the hands of Murphy J was certainly not complimentary relative to the intellect shown by other good judges.

Whilst tenure allows reform without fear it may also encourage hubristic ambition as a judge may push their own agendas to promote their own political causes.  For example, Gordon J wrote a treatment of competition law and then used a novel approach against Bradken and Greiner promoting this treatment.  Gordon J wrote a treatment on fast tracking litigation then used such techniques, which may have been unwise, in Zomojo v Hurd & Others.  For any powerful entity to take ambitious action without any fear of consequence is dangerous.

2. Judges, except in the High Court of Australia, should not be the first point of call with bias or apprehended bias claims against themselves.

It is widely recognised around the world that self-assessment for bias and the perception of unfairness perhaps should not sit with the judicial officer overseeing the case.  In our particular case we have seen Gordon J show terrible bias and on looking at some of Her Honour’s other cases, for example Bradken/Greiner, Her Honour has shown poor self-assessment.  The SMH reported on one of her cases that she was surprised that a defendant thought it improper that she should sit on a case where she had previously prosecuted the gentleman as a barrister on behalf of ASIC.  In the ANZ bank fees case, Gordon J sits on a school board chaired by an ANZ senior executive.  A higher standard needs to exist for public confidence in the court system.

3. Mediation is a good idea but flawed. It should be completely optional without recourse if shunned.  It should come with warnings regarding the abuses that may occur.

Our own experience in court mediation has been terrible. The applicants treated the process with contempt and simply used it as an abuse to fire outrageous claims to attempt to hone an argument. It also seemed that it was used as a mechanism to leak information through back channels to the court system as the relationships between the registrars and judiciary is close, perhaps too close at times. However, if a party is to resist mediation today they would be ill considered by the judiciary before even stepping into the court room. There is not an understanding of the dangers and abuses that parties may carry out in mediation.

4. The Federal Court of Australia rules disallowing self-representation by a corporate are flawed and they should be altered, at least to the equivalent of NSW Supreme Court.

Abuse by litigation is easy as litigation is expensive. More so in the Federal Court of Australia.  Millions of small businesses are potentially subject to bullying and intimidation by legal threat. Firms with money can overpower others in Federal Court as firms may be denied adequate representation or any representation at all.  If you cannot afford a lawyer you cannot even represent your own small company which means you simply get trampled on.  Even when our company had a barrister in court, we were denied representation as having a barrister in court was insufficient legal representation.  That makes no sense.  The system needs repairing.

5. Lawyers and barristers should be more accountable for their correspondence and statements to the court and opposing parties.

We and our various legal representatives have been subject to many lies, purposefully misleading statements and intimidation from the other side.  Aggressive legal behaviour when grounded in truth is much different to unwarranted threats, out right lies, feigned memory loss in the stand, and intimidation toward smaller legal offices, all of which we have witnessed first hand.  Unfortunately such actions are almost never punished which creates a popular market for such firms. Two particular Australian firms are renown for such bad behaviour. Firms that will seemingly take great personal risk by illegally stepping of the line forcefully are appreciated by clients looking for aggressive legal action.  However, it turns out there is virtually no risk resulting from such behaviour as almost nothing is ever and far from being a risk to those firms it is just a great marketing benefit which attracts more clients and perpetuates the court condoned behaviour.

Part of the reasoning for the FCA not allowing self-representation of corporates is that legal representatives are meant to be held to higher account, as it is their profession they are supposedly risking if they were to be inappropriate.  Our experience of lies and deceit from the opposing legal representatives make a mockery of this. The system is broken. 

Our first complaint is before the Legal Services Commissioner in Victoria but I would note the Commissioner’s own words that larger firms are almost never before the commissioner as they behave properly. I don’t think our experience is particularly exceptional so I believe this perception from the Legal Services Commissioner to be a distortion simply due to the power and influence of such firms allowing avoidance of consequence.

6. Procedural fairness and natural justice claims to be heard during a trial, not after.

When a judge makes gross errors in procedural fairness and natural justice it is wrong that a party must wait an inordinate amount of time to have that heard especially when a participant goes to great lengths to delay the possibility of review. In our case it will be years before we may be heard. Also, due to a terribly wrong decision by the Davies J to put solvent companies into insolvency by a party with no proper standing, the claims of the corporates will now never be heard.

It is wrong that a judge can change the allegations and the case after the trial has been run as Gordon J did. A person has a right to know what the case is that they are actually answering otherwise how can they present evidence and a defence?  It is wrong that a judge can ignore submissions from a defendant and quote irrelevant submissions from the other side.  It is wrong that a judge can throw out evidence wholesale without proper consideration. It is wrong that a judge can complain of evidence not being presented in a written judgment when the transcripts show they themselves denied it being presented. It is wrong that a judge may declare evidence is not to be heard until a second part of the trial then rule on that part of the case before the evidence is heard and then dismiss a stay or appeal. It is wrong that a judge can selectively ignore evidence in judgment when it does suit their purpose in a judgment.

When a judge, such as Gordon J, goes off the rails so outrageously, it is wrong that years can pass before any recourse is possible. It is wrong that other legal mechanisms can be further used to delay and derail having an avenue to appeal. The system is broken.

7. Officers of the court should be held to a higher account with respect to apprehended bias from outside employment or directorships.

Davies J and Gordon J both work in the same tax faculty at Melbourne University Law School.  An applicant’s wife also worked at Melbourne University as a Professor.  The applicant and his wife had donated at least $200,000 over several years to the University of Melbourne.  That year of the trial, 2012, a relative of the applicant was awarded an academic prize in the Masters of Law and there are many other relationships between the applicants and the judicial officers, including at least one of the registrars. 

Gordon J is deciding an important ANZ bank fee class action case.  Gordon J’s boss on a school board Her Honour sits on is an ANZ executive.

Such relationships bring into the question whether a reasonable lay person may perceive that a judge may or may not be impartial in such circumstances.  There should be more stringent guidelines to what is acceptable and when a judicial officer may not sit on a case.

8. Back channels and informal communication prejudicial to or with the potential to pervert a proceeding need to be eliminated.

Over the last few years of this case running it has become clear that there are back channels of communication between lawyers, registrars, judges, judges chambers and applicants.  We have overheard inappropriate murmurings in the registry whist standing at the registry counter.  We have been subject to onerous behaviour from a registrar time and time again only for that registrar to later recuse himself when he would have had to make a decision in our favour as he then declared a friendship with the applicant. We received a response to an application before it had even been served. Certainly not all our experience has been this way but there as been enough to be very disconcerting.  There needs to be higher standard and also, if we are aware of it in our occasional flirtations with the court process it indicates a larger problem as our reference points are few and far between.  The majority of the iceberg would be underwater.

This rampant communication and back channel gossip seems particularly endemic to Victoria.  Legal representatives we have spoken to in other states, particularly those that occasionally practice in the Federal Court in Victoria, shake their head in disbelief at the general closeness of legal community to Victorian Judges, Federal or otherwise, and remark that such behaviour would never be tolerated in their state, whether it be Qld, NSW or SA.

Perhaps in Victoria there should be a review of policies and procedures and retraining, where appropriate, of courts with respect to professional policies in this regard. There needs to be a higher standard.

9. Defamation by the media during a trial happens too easily without recourse.

During our case, Fairfax have published defamatory articles in The Age, SMH, AFR and BRW without regard to the truth. It seems there are one or more relationships between our adversary and parties within Fairfax. Fairfax has only been too willing to abuse their position and yet the current defamation laws only offer a long and torturous path with little hope of getting any recompense on winning such obvious defamation. As has been explained to us, defamation is only for rich people. There is much to fault with Fairfax’s behaviour, but equally the regulations and laws of the land are to blame for allowing such abusive patronage in the media.

10. Defamation by the judiciary is too common.

A judge is protected from being sued for defamation. A judge is expected to make reasonable conclusions. A judge may interpret the demeanour of a witness on the stand and make conclusions regarding their character, as this is part of the job.

However, we have seen outrageous character assassination in judgments that seem solely to serve a purpose of allowing the judge in question to support the conclusion they wish to make. Gordon J is particularly adept at writing fanciful story lines in judgments where Her Honour has to exclude facts that do no support the prosecution Her Honour makes.  We have also seen such assassination of character where corroborating evidence from other witnesses is ignored as it would be perhaps inconvenient to assassinate multiple witnesses’ character over the same factual contention.

With great power comes great responsibility.  Judges should be required to show much more care in their pronouncements on character. Witnesses should have some recourse to claim damage or seek correction or apology from the court rather than be published eternally as people of poor character. The collateral damage of judges, whether rogue or not, is unforgiving and everlasting. A person may spend a life in charitable works and be a pillar of the community only to be casually destroyed by a judge without regard to proper merit. There needs to be a higher standard and recourse for such parties especially when the parties are not direct participants but witnesses, employees or otherwise connected to the case.

For some judges it seems to be part of their standard toolkit.  Gordon J was harsh in our case on numerous people from our side.  Davies J was unacceptably and outrageously hard on one particular witness who is a pillar of his community.  Then you look at other Gordon J cases and you see a pattern of behaviour, for example, Bradken and the character assassination of former Premier of NSW, Nick Greiner AC.

11. Expert reports should be able to be challenged on entry rather than waiting for their use.

A tainted and deeply flawed expert report, where the expert Associate Professor Leong from the University of Sydney prior to writing the report had raised a $200,000 grant in conjunction with the applicant, was entered unchallenged into the court records. As the companies have, wrongly in our view, fallen into the hands of a liquidator, that flawed expert report will live in the public court records unchallenged.  This does not seem appropriate. There should be some minimal review before an expert report can be accepted into the court files to handle such obvious objections.

Also it begs the question of whether or not a provocative and flawed report from someone such as Assoc. Professor Leong should be able to be put forward without review from the University of Sydney as he basks in the aura of their good name and likewise does damage with his ill-conceived patronage.

12. The appointment process for Judges needs to be reconsidered.

In Victoria, at least, politics, hereditary relationships and friendships seem to have too great of an influence in the appointment of judges.  The appointment of Gordon J with her husband Hayne J sitting on the High Court of Australia has created an environment where legal professionals fear criticising her actions.  The appointment of a solicitor with no previous judicial experience, Murphy J, to the Federal Court of Australia, was curious especially with respect to the former relationships to Prime Minister Gillard. The appointment of Davies J to the Federal Court of Australia is curious due to her unremarkable Supreme Court experience and her relationship to existing FCA judges, especially Gordon J, and the status of her father, all of which paint a picture of cronyism.

It would appear that, at least in Victoria, better minds and diversity in court appointments should be considered as there seems to be too much patronage in the appointment of some of the judiciary in recent times which in turn taints those meritorious judges who deserve their commissions and serve the community well.


The unfortunate circumstances of my recent experience have confirmed that getting too close to the sausage factory and seeing how the sausages are made is never a good thing.  My experience, the experience of an outsider with limited previous experience of the court processes, may offer some fresh insights.  The problems I have seen seem to be more than just a squeaky wheel.  

Real reform is needed.

Friday, 17 January 2014

January 17th - Will today be memorable?

President Obama has curiously decided that 17 Jan 2014 was to be the day he would make a speech regarding justice, liberty and the role of the intelligence communities, including the NSA.

President Obama has preempted the final report from his privacy board, so the date appears carefully chosen, but why?

If you meander around the 17th of January in history a few topical items pop up.

In 1964 on the 17th of January, the President's wife, Michelle Obama was born. Happy 50th to the First Lady. In a few years my wife will also be 50 and I know I'd better have a damn good reason, and something my wife would be proud of, if I was to do something else that wasn't focusing on her big day.  Will the President make his wife proud and stop the spying on all citizens, all citizens' children and Michelle Obama's children?

In 1961 on the 17th of January, President Eisenhower, a general, a Republican, gave his farewell address which presciently warned of the vigilance required to keep the Military Industrial Complex in check to avoid freedoms being crushed. Will Obama now reward Edward Snowden who at great personal risk served the people by being the vigilant citizen that Eisenhower prayed for?

If Eisenhower's timeless speech was day 1, then President Obama was born on day 200 in Hawaii.

In 1893 on the 17th of January, the monarchy of Hawaii was overturned in a coup d'├ętat by the Citizen's Committee of Public Safety.  This paved the way for President Obama to be born into a state that had joined just the Union not quite two years prior.

Painted by William Ranney in 1845, this depiction of the Battle of Cowpens shows an unnamed black soldier (left) firing his pistol and saving the life of Colonel William Washington (on white horse in center). {Source Wikipedia}
In 1781 on the 17th of January, a fledgling nation had a decisive victory over a tyrannical oppressor at the Battle of Cowpens as part of the American Revolutionary War. Two US Navy ships have carried the title of USS Cowpens in honour of this feat.

During the Battle of Cowpens, a heroic unnamed black soldier was depicted in Ranney's painting of the battle to have saved the life of Colonel Washington, George Washington's second cousin.  Will President Obama be as heroic as the unnamed soldier on the 17th of January 2014?

Will President Obama heed the warnings of President Eisenhower from the 17th of January 1961?

Will the President honour his wife's 50th birthday on this January 17th by doing what is right and showing respect to Mr Edward Snowden who risked it all to answer the historic clarion call of President Eisenhower for all citizens to be vigilant and protect freedom from oppression by the military industrial complex?

Let's hope President Obama's speech echoes through history with sufficient pride for future generations to venerate the date of 17th of January 2014.

Wednesday, 15 January 2014

Russian Constitution - Articles 23 & 24 - Plus Some Other Countries

Article 23

  1. Everyone shall have the right to the inviolability of private life, personal and family secrets, the protection of honour and good name.
  2. Everyone shall have the right to privacy of correspondence, of telephone conversations, postal, telegraph and other messages. Limitations of this right shall be allowed only by court decision.

Article 24

  1. The collection, keeping, use and dissemination of information about the private life of a person shall not be allowed without his or her consent.
  2. The bodies of state authority and local self-government, their officials shall ensure for everyone the possibility of acquainting with the documents and materials directly affecting his or her rights and freedoms, unless otherwise provided for by law.
I think that beats the rights in the five eyes? The judicial processes and enforcement of such may be another issue. I don't know if the promises of the constitution are kept but the Russian Federation seems quite an unjust place for fair trials from afar. So, add a grain of salt.  Interesting nonetheless in the context of the current NSA affair.
Source thanks to the pipes and tubes of the Internet [1] & [2]

Now for any colour as long as it is black: here are the pertinent articles from the Chinese Constitution which carve out juicy exceptions for State security and censorship that enshrine Orwellian (or NSA-like) capabilities:

Article 39 

The residences of citizens of the People’s Republic of China are inviolable. Unlawful search of, or intrusion into, a citizen’s residence is prohibited.

Article 40

Freedom and privacy of correspondence of citizens of the People’s Republic of China are protected by law. No organization or individual may, on any ground, infringe upon citizens’ freedom and privacy of correspondence, except in cases where, to meet the needs of State security or of criminal investigation, public security or procuratorial organs are permitted to censor correspondence in accordance with the procedures prescribed by law.

In Australia there is no constitutional right to privacy nor improper search and just a limited form of freedom of political communication.  The privacy laws vary across the states and privacy law is complex as described by the Australian Law Reform Commission with particular considerations in place for intelligence gathering.  Gigamon, Endace, Splunk, etc are used by the Australian agencies to trawl.

The Egyptian Constitution candidate currently being voted on is also interesting in the context of the Arab Spring:

Article 57 Private life

Private life is inviolable, safeguarded and may not be infringed upon.

Telegraph, postal, and electronic correspondence, telephone calls, and other forms of communication are inviolable, their confidentiality is guaranteed and they may only be confiscated, examined or monitored by causal judicial order, for a limited period of time, and in cases specified by the law. 

The state shall protect the rights of citizens to use all forms of public means of communication, which may not be arbitrarily disrupted, stopped or withheld from citizens, as regulated by the law.

Article 58 Inviolability of homes

Homes are inviolable. Except in cases of danger, or if a call for help is made, they may not be entered, searched, monitored or wiretapped except by causal judicial warrant specifying the place, time and purpose thereof. All of the above is to be conducted in cases specified by the law, and in the manner prescribed. Upon entering or searching homes, those inside shall be notified and informed of the warrant issued in this regard. 

Article 92 Limitations clause 

Rights and freedoms of individual citizens may not be suspended or reduced. No law that regulates the exercise of rights and freedoms may restrict them in such a way as infringes upon their essence and foundation.

And Brazil?  I'm not sure I understand "anonymity is forbidden" in IV as it reads strangely. Nor is "the confidentiality of the source shall be safeguarded" in XIV below clear to me.  Brazil likes its paperwork and at 435 pages the English version of their Constitution is not short.

Title II - Chapter 1 - Article 5

IV – the expression of thought is free, and anonymity is forbidden;
X – the privacy, private life, honour and image of persons are inviolable, and the right to compensation for property or moral damages resulting from their violation is ensured;

XI – the home is the inviolable refuge of the individual, and no one may enter therein without the consent of the dweller, except in the event of flagrante delicto or disaster, or to give help, or, during the day, by court order;

XII – the secrecy of correspondence and of telegraphic, data and telephone communications is inviolable, except, in the latter case, by court order, in the cases and in the manner prescribed by law for the purposes of criminal investigation or criminal procedural finding of facts;
XIV – access to information is ensured to everyone and the confidentiality of the source shall be safeguarded, whenever necessary to the professional activity;

Perhaps the following could be interpreted to allow your private property (email?) to be used by the state to stop terrorism?  Seems a bit of a stretch though.

XXV – in case of imminent public danger, the competent authority may make use of private property, provided that, in case of damage, subsequent compensation is ensured to the owner;

If that was the case then perhaps you'd be justified in calling on this next right to prevent the "use" of your communications ;-)

XXVII – the exclusive right of use, publication or reproduction of works rests upon their authors and is transmissible to their heirs for the time the law shall establish;

Tuesday, 14 January 2014

Professor Geoffrey R Stone concludes NSA activity illegal

Professor Geoffrey R Stone is an important person.
Professor Geoffrey R Stone: Edward H. Levi Distinguished Service Professor of Law
The University of Chicago - The Law School

Professor Stone has at least Top Secret clearance. He hired President Barrack Obama in 1992 to teach Constitutional Law at the University of Chicago. Professor Stone is one of the five trusted wise men to be part of the President's Review Group on Intelligence and Communication Technologies. The recent public report, Liberty and Security in a Changing World, recommends sweeping changes though it does not stop the data collection but rather seeks to improve controls.

During an interview and debate on DemocracyNow, Prof Stone said [about 8:40 into the you-tube clip] words to the effect that there was "absolutely nothing illegal or criminal" about what had been revealed about the NSA programs. He said the claim that they are "unconstitutional and illegal" was "wildly premature."

At the Huffington Post, Prof Stone has concluded a series of articles with the headline and conclusion that, "The NSA's Telephone Metadata Program Is Unconstitutional."

It shows, to me at least, how far legal thinking can get from reality. It would seem obvious to the average Joe that collecting and storing hundreds of millions, if not billions, of meta-data records on citizens every day is obviously against what the founders had in mind when they wrote the Fourth Amendment. Whilst Prof Stone's series of articles leading to the same conclusion are interesting, well reasoned and insightful, it is, to me, a tortured indictment of bottom-up legal reasoning for something so obvious. What is wrong with top-down reasoning starting at the Fourth Amendment itself?

Prof Stone's article's final paragraph fires a warning at those claiming obviousness:
"There are those who maintain that this program is obviously constitutional and those who maintain that it is obviously unconstitutional. They are both wrong. There is nothing "obvious" about this. If this ever gets to the Supreme Court, it will be interesting."
And herein lies the fundamental problem. That which is obviously wrong to a citizen is not obviously wrong to those with smart and strong legal minds such as Professor Stone. It is the world of court-facts versus real-world-facts.

It is nice to see Professor Stone conclude the existing program is unconstitutional:
"In conclusion, then, in my judgment the existing program is unconstitutional. As currently structured, it violates the Fourth Amendment's requirement of "reasonableness." On the other hand, it should be possible for the government to correct the deficiencies in the program in a manner that both preserves its legitimate value and substantially mitigates the risks to privacy that it currently poses."
I guess he is saying that from his perspective an unconstitutional verdict is now less "wildly premature."

Professor Stone is one of the saner voices in the debate and his new "considered" view is a welcome evolution. The US Supreme Court ruling he points to regarding wholesale collection of GPS data is particularly enlightening. The intricate arguments I just can't buy as there should be no need and such arguments smack of post justification and integration with existing laws from an activity that is an obvious abhorrence. Professor Stone's claims that there is nothing obvious about this may be true in the sense of a bottom up legal post-justification but surely not in a principled adherence to the US constitution where the Fourth Amendment rights are fairly succinct and clear.

There is another larger story here pointing to the introspective nature of the legal system, its growing distance from reality and its perversion of purpose where it is becoming more polarised into a system of contract resolution and further away from being a justice system. As a recent victim of court injustice, I'm keenly aware of the lack of justice present in the modern legal system where dollars, patronage and hubris rule over a contract resolution system without proper justice. President Obama needs to consider the equitable course of action here, borrow from the purpose of the Court of Chancery, and thank Mr Edward Snowden properly by giving him the relief of a homecoming without persecution.



PS: Without much thought, I think the controlled collection and warrant-based search procedures may be a good advance but I'm not sure I understand how something that is illegal due to unconstitutionality becomes legal, but the good Professor's articles hint at that reasoning via meandering thoughts on what is "reasonable" and what constitutes a "search." A state sanctioned protected passive collector of data, the Data Guardian, that only lets targeted court warranted searches or algorithms to run without the agency having their own ability to examine or investigate is perhaps possible with careful cryptographic means. Perhaps this is where encrypted local government CCTV footage should also be stored? How can such mass surveillance ever be made safe from not just abuse but future abuse? Collecting mass data just seems wrong. Whilst the ability to hunt down terrorists and criminals retrospectively via their metadata is noble, the debate on balancing security, privacy, anonymity and freedom needs more work. Mass surveillance and storage of data on children is gut wrenchingly evil. Unwitting mass surveillance is a gross violation of the Fourth Amendment. Humanity needs boundaries to protect itself against its own behavioural boundaries.

Thursday, 9 January 2014

Good people from the NSA come forward

An article from former NSA staffers that is worth reading.

It reads a little bitter, but as you read it you understand why.  The Fourth Amendment violations of the NSA were well understood by some of the NSA staff, such as the authors of the above article.

It is also a good reminder of why Mr Snowden has taken the course of action he has. The expose is a reminder that there was no chance of being able to sanely blow the whistle from within as others had already failed.  Pardon Snowden and bring him home.


Sunday, 5 January 2014

NSA solution - learning from HFT

It is kind of interesting to think about about potential solutions for the NSA and the other participants in the five eyes.

You have gotta feel some sympathy for many good engineers and developers at the NSA and elsewhere. They have built a very impressive global spying network as part of the five eyes that provides amazing capabilities with some very cool technology. However, those staff that have completed their mission well, that have a moral conscience, will be shifting somewhat uncomfortably in their chairs. It is easy to get caught up in the mission and creep into doing the wrong thing. You find yourself all-in on the grey zone which makes it darker and darker until it turns black but looking around it all looks the same and you don't recognise where you have found yourself. The pendulum needs to swing back.

It is wrong to collect all such data on your citizens. It should be obviously wrong. In the US it is no doubt illegal. Eisenhower was right to worry. Hopefully the US Supreme Court will do its duty properly and say such when the cases in motion percolate up the system. The situation is less clear with respect to legality in the other countries participating in the five eyes that don't have the equivalent of a Fourth Amendment. However, the moral imputation is clear. Pressure from the people outside the US may make illegal such mass oversight on their own citizenry. I personally doubt it. Most people I ask don't seem to care. I put it down to the apathy you find in the comfortable lifestyles of modern society. However, vocal minorities can and do swing elections. The agencies should care as they may need a new plan if their government finds self-interest supports appearing to care about their voters.

Much of the NSA collection is pretty obvious.  Tap comms lines everywhere, use deep packet inspection, etc.  One of the surprising things is that much of this seems to have happened much later than you would have thought. The NSA and others were sucking on satellites since satellites existed. The stories about firm's financial trading in Japan suddenly improving when switching from faxes to other means abounded in the early 1990s. The Japanese have been good at satellite interception and disseminating economic information for many years. You would have thought the NSA would have been a bit better prepared for the tubes of the internet and slurping up the data earlier than Snowden's leaked documents showed.
Source: Guardian: PRISM starts 14 years after the release of the Mosaic web browser in 1993
Perhaps the NSA was more competent and PRISM was just a newer program replacing an earlier slurper.  We'll never know. Also, the use of Hadoop for analysis is surprising as you would have thought they would have been better prepared with their own tools. Just goes to prove that government's are not efficient, just good at getting there eventually. As I pointed out earlier, the NSA's system latencies are not great, being a couple of generations behind best practice in finance at a similar point in time. The thin film transparent resonant RF tags remain secret ;-)

These things point to some potential strategic thinking deficits at the five eyes. They should have been aware that collecting all of the data was not a sustainable position and thus have plans for when their taps are turned off. It also potentially points to a lack of foresight in system design and architecture to end up relying on tools such as Hadoop. Hadoop is a good tool, not tremendously efficient, but really, they should have been at the forefront, not playing catch-up. Full credit for using open source to cover your deficits though.

Perhaps the five eyes are great strategic thinkers and already have plans for losing their citizens' data but you suspect not. They have certainly created a rod for their back as the world will start demanding much better physical, policy and cryptographic security. Vulnerabilities to so called tailored access operations will always be there but they will become less easy over time. Spies are a great alternative to war. Intelligence gathering is not about to go out of fashion.

There is the Hail Mary option of each eye simply getting one of the other four eyes to collect in their country on a quid pro quo basis. This already happens to some extent. The full extent we don't know. Such transitive interposing of responsibility may be illegal and if not, certainly legally unsustainable due to its simple ethical and moral repugnance. The eyes need to plan on not being able to store mass data from their own citizens.

There is perhaps a solution.  Just as the financial world has provided a treasure trove of domestic spying opportunities via Sars-Ox data retention policies, the financial community could help again. Finance could provide some lessons to the five eyes in how to make the hunting of bad guys palatable and possible without the NSA collecting massive amounts of metadata. Finance, via scripts written down in the evolutionary war dictated by capitalism's policy of creative destruction, is ahead of the NSA on latency, so it is not unreasonable there may be more lessons to learn.

That solution is algos.

Leave the data where it is.  Require it to be stored, like Sars-Ox. Use distributed algorithms where you search across the data in the vendors' own data storage.  Send the code to the data. If you used a "warrant-id," or some other authority token, then accountability could also be baked into the system in an auditable way. It has similarities to the way high frequency trading applications may use algorithms to distribute trading responsibilities to various venues or market places.  Some exchanges and brokers are allowing or beginning to allow the embedding of algorithms into their systems, such as through FIX Algorithmic Trading Definition Language (FIXatdl) [Wikipedia]. You wouldn't want to do something quite so clumsy, but the point is that such an approach is feasible.

It's not quite as easy as all of that.  Many NSA algos, such as the CO-TRAVELER system, require correlations across systems. These would be slower due to the there-and-back-again nature, but the broad outcomes would still be achievable.  Ignoring the future potential of homomorphic encryption for the time being, you can imagine various mechanisms where vendors' systems could anonymously co-operate by cryptographic methods to improve the efficiency of what may be achieved in a complex query. The vendors keep the metadata for billing and analysis anyway. The state could pay for extra resources, such as systems, data centres and energy as required. However, the use of the data would become much more accountable.

It does also make you think that vendors, such as an AT&T or Verizon, should be restricted in the ways they can make use of their data so they are not mini-spy agencies, but that is a separate curiosity. Hopefully a political lobbyist for such a telco does not have unlimited powers of omniscience. Yes, you should be worried.

You could even imagine a world of co-analysis protocols where like minded countries co-operate on crime and terrorism by opening up all the vendors' data in their own countries to such "warranted" algorithms or "search agents" to nationally approved algos originating from approved agencies. This could be done quite quickly you'd imagine. It is not technically hard. There is certainly a lot that could go wrong. It could even get ridiculously out of control. I don't think I'd like a mandated algo entry point for warrant approved searches for my cell and all my equipment at my home. It is a matter of balance. The pendulum needs to swing back to the citizen. Distributed algorithms are one way that freedom and privacy could be improved with the fight against crime and terrorism also being improved.

It does remind me of a kind of strange perversion in the finance world. I was once told it was required for a US bank to collect information on account holders so you could prove that your institution did not discriminate. In Europe it was required that a bank didn't collect such information so you couldn't discriminate. I'm not sure if that is a bad joke hanging around in my memory or if it was the real situation, but the point is clear. It is obviously better if the information is not collected. For that data that is indeed collected, it is better not being centralised. Subject the data to properly authorised search in situ.

Eventually life for the intelligence community will become more difficult. Encryption and anonymity will improve. The immorality of the mass surveillance of your own citizens will be more clearly recognised and policed without a rubber stamp. Perhaps a global legal framework or treaty for such mass surveillance disdain, regardless of nationality, will one day make the planet a better place.

At my geek level, I like the idea of more extensive hardware, firmware and software integrity checks. RF analysis and tempest security should become more widespread. I also like the idea of new kind of virtual mobile network operator where phones not only have improved encryption on traffic but they shuffle their identities in the system to improve protections against some types of data extraction and tracking attacks. Imagine an integrity check that maps a binary executable to software source or behavioural description to ensure Ken Thompson has not influenced your system. Simpler servers, network gear, operating systems and software are all required so we can have more certainty about their operations.

There is a lot to be done.

Perhaps the intelligence communities shooting themselves in the foot by over reaching and being too effective could indeed make the world a better place. Thank you Mr Snowden for making us take our security, privacy, identity and, most importantly, freedom, more seriously.

A pessimistic optimist

[Bring Snowden home]