Sunday, 23 November 2014

SC14 view from afar

Supercomputing 14 in New Orleans has wrapped up. Despite the stasis in the top 500 echelons some meanderings of interest to traders continue to emerge from HPC.

DE Shaw's Anton 2 - click to enlarge
(The Anton 2 Chip: A 2nd Generation ASIC for Molecular Dynamics Simulation Hot Chips 2014)

The BIG announcement for me for the week was Xilinx's SDAccel. Every latency focused trading firm is going to have to evaluate this framework from a functionality and productivity point of view. There have been quite a few different toolkits for doing this kind of work over the years but this feels a game changer.

I've previously been involved with Impulse-C and found it an interesting and useful product from a good company. However, it was largely not used in my shop as the types and pragmas required in the code essentially removed the C'ness which made it yet another C-like foreign language in practical terms. We found it more productive to just go straight to VHDL code at the end of the day, so the Impulse-C, though impressive, became shelf-ware. Perhaps if we had more of a computational burden it may have been more useful, but much of the work at the time was tricky I/O and thus VHDL mattered more.

The nicest implementation of C-like hardware languages I've reviewed over the years was Handel-C. Lovely CSP / Occam / Pi-calculus like constructs. Quite elegant code. Not C though, but this was not that important for me at the time. The killer was that the platform implementation targets were limited and the performance of the code was somewhat, er, um, challenged. Limited performance is not what you're looking for when biting off hardware implementations. Impulse-C targeted VHDL or Verilog so it had the advantage that the specific platform compiler could optimize. Handel-C tried to do more and targeted the RTL level directly but it didn't quite do the terrific job the native compilers could do. I think they lost an opportunity there.

There have been a bunch of others over the years. For example, Mitrion-C was bundled with Cray's hybrid FPGA supercompute platform for a while (Cray XD1(TM) supercomputer) and SGI's RASC. No stand outs for success. C-like programming for FPGA platforms is hard.

Hopefully, Xilinx's SDAccel can pick up some success. I think the big game changer is both the support and weight of Xilinx behind the platform and the availability of partial reconfiguration on the fly. It remains to be seen whether the intrusions of specific types, code styles and the dreaded pragmas overwhelms the benefits a high level language brings, but it sounds quite the advance to me. I'm optimistic. An impressive aspect from the demos was the occasional improvement possible over raw VHDL from both clock and area perspectives. When combined with SDNet and their QPI toolkit, extremely compelling architectures may be possible with small development teams. Let's hope the C-like approach works for FPGAs this time. All trading firms will need to have a look at this kind of framework if truly low latency is what makes them tick.

Some other tech from SC14

Intel had some interesting announcements, not all at SC14. The further development of the Knights platform to Knights Hill is very compelling, especially when combined with HMCC memory. With the progression from Corner, Landing to Hill, there is plenty of geographical growth possible. I can't wait to see what the Mountain platform or 2030 Galaxy platform looks like ;-)

The release of HMCC 2.0 and the support of various vendors, including Xilinx, is exciting for a few reasons, but the main one for me is the indication of continued growth in 3D, or 2D stacking, manufacturing. Intel was also talking about this for Flash memory, just as Samsung has recently done, and Intel presaged packaging solutions with around 10TB of Flash in SSDs in only a couple of years. NVidia also previously announced stacking of memory with their Pascal GPU platform. NVidia's Volta, announced for Summit and Sierra, will continue that. Continued development of innovative packing will drive densities higher and costs down though at a cost of accessibility as it seems this is a bit like the new PCB but only accessible to the very few.

D E Shaw's Anton 2 chip was an exciting platform to read about. I'm not sure if it won the Bell prize, but if it didn't, I think it probably should have (* update: it did). One notable takeaway from reading their paper is that so much more is still possible with further overlap, out of order speculation and processes beyond 40nm and 2B gates. I can't wait to see Anton 3. It is good to see a firm be prepared to build a custom ASIC. Their last one, the original Anton, was done by an e-beam house in Japan but the markings on this chip are from Korea, so I wonder who they are using to fab there? Whilst it is for molecular dynamics, I'm sure every trader would love to have such awesome power at their finger tips for "Hardware Support for Fine-Grained Event-Driven Computation." Though I'm sure their research motives are altruistic, I'm also sure a clever trading firm would find a use for that ;-)

100Gb connectivity continues to mature with announcements from Mellanox of a sub-90ns 36 port 100Gbps IB switch and with the Invea-tech platform, previously mentioned, being demonstrated at SC14. I must say I find the 25G Ethernet movement, also on demo at SC14 running "under 100ns" platform to platform, a little more practical as a future path for now.

Mellanox, in addition to their 100Gbps IB/Eth card, also joined SolarFlare in having an FPGA NIC solution. It is an alluring solution that looks like it may be better than the SolarFlare for latency as it supports FPGA fabric on either or both of the network and PCIe facing sides. The press doesn't declare the flavour of FPGA nor the tools available. We'll have to wait to understand it better to see if it can compete with a vanilla network connected FPGA NIC for low latency trading.

The Power8 CAPI platform had its first official platform release with Nallatech at the show. A good latency reduction technique to be aware of. I think I'd be focusing on the Xilinx / Intel QPI solution myself though.

Some of my old friends at Metamako were at SC14 next to the Xilinx booth with their compelling platform. If you want the fastest way to get two packets from two wires onto the one wire at 10G, say for facing an exchange, risk gateway or other financial choke point, such as a mandated firewall, then their new MetaMux 32 reigns supreme. It is the fastest way at ~100ns to get two packets onto the same wire at 1G / 10G speeds. This is in additional to their fancy-smancy layer 1 reconfig tricks. I think the biggest use case for me for Metamako is simply using their MetaConnect platform just for timestamping though. You could save a bunch of money using one of their MetaConnects, or perhaps even the MetaMux, to timestamp and collect packet information rather than using expensive timestamping cards but I'm not sure people realise how cool that is from a non-intrusive tap reconfig and cost saving point of view. I'd really like a MetaConnect to script up for dynamic network reconfig and running performance unit tests for financial apps in the test lab.

A mystery for me is the intriguing Intel Omni-Scale platform. Just enough details have been released to know that it is going to be interesting but not enough to really know what it is all about. So, keep your eye out for developments there. Intel have long had the opportunity to kill off much of the NPU, custom processor and even perhaps the entire NIC market by incorporating compelling networking interfaces into the coherent level of their microprocessors. It is not clear when and how this may happen with Omni-Scale. Vendors relying on NIC card revenue should be starting to shift a little uncomfortably in their seats. A big leap in network latency reduction could be at hand but there remains an opportunity for a firm like Cavium or AMD to gazump Intel on the low latency network and compute combination. Direct fabric integration, not the kludgy PCIe SoC kind, is long overdue for mainstream processors. Maybe this is at last on its way.

Perhaps not so relevant to trading apps, the announcement of Summit and Sierra was intriguing with their reliance on nVidia's Volta with stacked RAM and NVLink. It is certainly a nice win for the IBM Power 9 platform even if the headline performance is really reliant on nVidia rather than the Power 9. It also just shows how hard an ExaFLOP is, at least for LinPack, as both of don't quite get to ExaFLOP performance, so the ExaFLOP race remains on.

Happy trading,


{NB: Scant public details from Intel on Knights Hill (optical connect, stacked memory, faster) and Omni-Scale/Path.}

Monday, 17 November 2014

Alpha Data - PCIe 3.0 Xilinx UltraScale 10/40G

I was browsing the exhibitors at SC14 which is on this week in New Orleans. I noticed that I had missed this nice looking Xilinx Kintex UltraScale ADM-PCIE-KU3 from Alpha Data which was news way back in April 2014.
Source: Alpha Data - Data sheet for ADM-PCIE-KU3
(click to enlarge)
Kintex UltraScale is now going mainstream so perhaps just noticing it nowish is cutely serendipitous. I'm not sure about the position of the SMA connector for timing as it looks a little inconvenient in this picture though the picture was an early release. Hopefully it has moved. PCIe 3.0 allowing direct access to Intel uP cache memory (data direct), from Sandy Bridge onwards, would allow lower latency than competing PCIe 2.X solutions. PCIe 2 solutions really should be avoided in this day and age if you're going to the trouble of eeking out the little bits of latency you can.

Simple is good. It is indeed a pretty simple board with up to 8 x 10G lines via break out cables or 2 x 40G, plus additional RAM. Not sure too many in trading would find the SATA ports useful but some may for direct recording to disk but I can't imagine too many bothering.

It looks a good choice and worth checking out at their booth at SC14 this week if you're after a pretty clean Kintex UltraScale design for PCIe 3.0 goodness.

I still think I'd prefer a Zynq solution, such as the Hitech Gobal board HTG-Z7-PCIE-HH or one of the not yet released Arria 10 ARM SoC based boards (e.g. BittWare’s A10PHQ). Still, I'd think a Zynq 7045 would be preferable to the 7100 on the HTG as I'd feel better trading better transceivers for less FPGA fabric, so I'm yet to find the goldilocks board for me. I'm interesting in hearing about boards that you might have found that are just right for your network oriented application. Drop me a mail.

Happy trading,


PS: Nearly two year old FPGA summary from this blog
PPS: Terasic FPGA board previously mentioned


The mystery regarding the PPS placement mentioned above is solved. Here is a picture of the Alpha Data card from SC14 via Xilinx's blog:

Picture of Alpha Data card from Xilinx Forum
(Click to enlarge)

This shows the SMA being exposed to the outside world via the PCIe bracket. It's a bit of an ugly hack but, hey, it works. Overall the card looks a pretty satisfying solution especially when teamed with the particularly awesome SDAccel framework Xilinx announced at SC14.  This particular card was one of the launch devices. SDaccel is a game changer, especially for trading and HFT. All low latency trading firms will need to review SDAccel from both a functionality and productivity point of view.


Saturday, 8 November 2014

Don't buy an ExaNIC from ExaBlaze

I've seen more dubious PR from the nasty people at Zomojo/ExaBlaze recently.

I thought it worthwhile warning people once more before they get sucked into ExaBlaze's reality distortion field.

Specifically: why their ExaNIC cards are not a smart idea; and, why you should steer well away from such wasteful expenditure.

It would be easy to be seduced by the pitch for the ExaNIC but the pitch is hollow with parts simply straight out lies. It's not your fault if you've been deceived into purchasing one.

There are two potential reasons for looking at an ExaNIC:
   1) A low latency network card, and
   2) An even lower latency FPGA trading solution.

Neither make sense. Let's meander through why.

Is it a good NIC?

In the first case, the ExaNIC card is feature poor and it has little advantage over a much better solution from Mellanox or SolarFlare. Mellanox[2] and SolarFlare[1][3] are quality NICs that have rich feature sets, kernel bypass and trusted implementations with open source drivers. Also, Chelsio and Myricom are worthy of some consideration if you have specific features you need.

You should be concerned about the signal integrity of the cards from ExaBlaze as the initial versions had poor to little signal integrity on the networking side which is a difficult problem to spot and solve. It's fine when it works but potentially devastating when you're having to solve problems. ExaBlaze push silly claims about latencies quoting 60 byte packets when the minimum frame length on 1G and 10G is 64 bytes. A 60 byte frame is illegal and referred to as a "runt" frame. To top it all off, the ExaNIC cards are expensive. Sure, buy one to play with if you must, but it makes no sense to invest in the card as a NIC solution.

If a quantum of a hundred nanoseconds is really important to you then, regardless of any NIC, you should be considering an FPGA solution to avoid the PCIe latency cost.

So, what about the ExaNIC as an FPGA solution for trading?

Just don't.

As an FPGA solution it is a much simpler and clearer case to weigh up. It is one of the slowest 10G or 1G Xilinx FPGA solutions you could possibly use. Yes, one of the slowest. If you are going to go all the way to an FPGA solution, which is an expensive way to go, then make sure you do it right.

The fundamental limiting factor in the ExaNIC x4 is that the network facing SERDES are off chip and slower than using native on-chip Xilinx transceivers. ExaNIC as an FPGA path is a dumb solution for low-latency. There are plenty of other excellent FPGA boards out there so you have no reason to waste your money. If you have been fooled into purchasing an ExaNIC on the mistaken belief that it was a fast way to connect to the market and trade then you may be entitled to your money back. They are lying to their customers about their high-latency architecture by referring to it as low-latency.

"It hung in the air in exactly the way that bricks don't."

Look at the following picture from the ExaBlaze web site:

(click to enlarge)
The standard way an FPGA NIC card works, except an ExaNIC doesn't. The SERDES for ExaNIC x4 are not on the FPGA, they are external, an extra hop, and higher latency. Don't be deceived.
The diagram clearly shows the network SERDES to be on-chip with respect to the FPGA. This is a lie for the ExaNIC x4 and it may also be a lie for the ExaNIC x2 if it is a similar design. The SERDES on the ExaNIC x4 is an external mux-demux chip and a slower solution than an on-chip FPGA transceiver. Why would a vendor use a higher latency architecture? To use a cheaper FPGA and save money. Don't be fooled. If you have fallen for the ExaBlaze misrepresentation, go get your money back is all I can suggest.

In summary, as NICs go, there is better quality, functionality and more reasonable prices from Mellanox and SolarFlare NICs, so avoid ExaNIC. As an FPGA solution, it is just a dumb idea to invest in an architecturally slow solution. You can't make it fast. It is slow by design. Get a real FPGA board from a trustworthy vendor as there are plenty of good alternatives. The combined ARM & FPGA SoC solutions as NICs are my favourites at the moment even though the ARMs are a little underwhelming. The SoC FPGA fabric to uP hop beats PCIe latencies for interesting hybrid solutions.

You should try to buy equipment from a trustworthy vendor. Matt Chapman annoyed a lot of people with his little video at Inside HPC where he dances around a product that Zeptonics developed and he claimed as his own. What a schmuck. Greg Robinson used stolen IP from ITG when he was sacked as CEO of ITG Australia to compete on the trading front at Zomojo with ITG's prop trading (Canada / NY interlisted arb specifically) and told multiple staff (so it's not just my claim) not to talk to Bill Burdett about it as Bill was still on the Board of directors of NYSE listed ITG Inc. Bill is a Zomojo beneficial shareholder. Then you have my little continuing dispute with them where I hope for a miracle one day that will see my faith in justice restored with some kind of judicial remedy that passes the smell test. Exablaze / Zomojo lied to either the court or their customers claiming Chapman was no longer associate with the trading at Zomojo to customers and yet at the same time claiming to the court he was an officer of Zomojo. Those statements are mutually exclusive, so both couldn't be true. Zomojo/Exablaze lied to the court many times. Zomojo and Exablaze are nasty people. Do you want to buy from a vendor that not only trades against you but misleads you? Caveat emptor.

If you've bought an ExaNIC, you should use their misleading representations as a reason to return the NICs and get your money back. Fortunately for you, there are plenty of good people in the industry making better product. It's simple. Exablaze serves no useful purpose. Use better products.

Happy trading,


[Update Sunday 2014-11-23]

PS: My speculation the ExaNIC x2 works the same way as the ExaNIC x4 with an external _SLOW_ mux-demux SERDES requiring an extra hop is confirmed. Xilinx point out the x2 board uses a Kintex 7-160T FPGA which has a maximum of 8 high speed transceivers. As the card has 8 x PCIe 2 (yeah, no PCIe 3.0), it has no transceivers left for facing the network.  So, it is confirmed, the x2 has a similar architecture to the ExaNIC x4 and ExaBlaze are lying to their customers about the design on their public website. The architecture is slow by design. Use a better product.

[1] Some Solarflare 10GbE Server Adapter Features

• LSO, LRO, GSO large packet offloads • TCP/UDP/IP checksums offloads • Line rate packet
filtering • Receive Side Scaling (RSS) • Accelerated Receive Flow Steering (RFS) • NetQueue,
VMQ, SR-IOV • 256 Multicast filters • Jumbo Frames (9KB) • 4096 VLANs/port • PXE boot,
iSCSI boot • IEEE 802.3ae – 10 Gigabit Ethernet • IEEE 802.3an – 10GBASE-T • IEEE
802.3ad – Link Aggregation and Failover • IEEE 802.1Q, 802.1p – VLAN tags, priority • IEEE
802.3x – Pause

Operating Systems: RHEL 5, 6, 7; MRG; SLES 10, 11; SLERT; other Linux; Windows Server 2003, 2003 R2, 2008, 2008 R2, 2012, 2012 R2; OS X v10.6, 10.7, 10.8, 10.9; Solaris x86 10, 11; ESX 3.5, 4.x, ESXi 5.x; KVM; Windows Hyper-V; XenServer 5.x, 6.0.All server adapters support: SR-IOV, 127 VFs per port,1024 vNICs per port.

[2] Some Mellanox NIC features

– IEEE Std 802.3ae 10 Gigabit Ethernet
– IEEE Std 802.3ba 40 Gigabit Ethernet
– IEEE Std 802.3ad Link Aggregation
– IEEE Std 802.3az Energy Efficient Ethernet
– IEEE Std 802.1Q, .1P VLAN tags and priority
– IEEE Std 802.1Qau Congestion Notification
– IEEE Std 802.1Qbg
– IEEE P802.1Qaz D0.2 ETS
– IEEE P802.1Qbb D1.0 Priority-based Flow
– IEEE 1588v2
– Jumbo frame support (9600B)
– VXLAN and NVGRE - A Framework for
Overlaying Virtualized Layer 2 Networks over
Layer 3 Networks. Network Virtualization
hardware offload engines 
– Single Root IOV
– Address translation and protection
– Dedicated adapter resources
– Multiple queues per virtual machine
– Enhanced QoS for vNICs
– VMware NetQueue support
– RDMA over Converged Ethernet
– TCP/UDP/IP stateless offload
– Intelligent interrupt coalescence
– Remote boot over Ethernet
– Remote boot over iSCSI
– MPI, Platform MPI
– PCIe Base 3.0 compliant, 1.1 and 2.0 compatible
– 2.5, 5.0, or 8.0GT/s link rate x8
– Auto-negotiates to x8, x4, x2, or x1
– Support for MSI/MSI-X mechanisms
– Interoperable with 10/40GbE Ethernet switches.
Interoperable with 56GbE Mellanox Switches.
– Passive copper cable with ESD protection
– Powered connectors for optical and active cable
– QSFP to SFP+ connectivity through QSA module
– Citrix XenServer 6.1
– RHEL/CentOS 5.X and 6.X, Novell SLES10 SP4;
SLES11 SP1 , SLES 11 SP2, OEL, Fedora 14,15,17,
Ubuntu 12.04
– Windows Server 2008/2012/2012 R2
– FreeBSD
– OpenFabrics Enterprise Distribution (OFED)
– OpenFabrics Windows Distribution (WinOF)
– VMware ESXi 4.x and 5.x

[3] Some specific SolarFlare features

Product Number
Standards & Compliance
IEEE 1588 v2
IEEE 802.3ae
IEEE 802.3ad
IEEE 802.1Q
IEEE 802.1p
IEEE 802.3x
RoHS Compliant
5.9W (typical)

Precision Packet Time Stamping
7.5ns resolution
Stable Precision Oscillator
Stratum 3 compliant; short term drift < 3.7*10-7 in 24 hours
Server Clock Synchronization Accuracy
Sub 200ns
1PPS-input circuit
Rising edge active, TTL into 50Ω
1PPS-output circuit
Rising edge on-time, TTL into 50Ω
I/O Virtualization
2048 guest OS protected vNICs; 240 virtual functions;
16 physical functions
PCI Express
PCIe 3.0 x8 @ 8.0 GT/s
SFC9120 10G Ethernet Controller
Supports high-performance 10GbE
SFP+ Support
Supports optical & copper SFP/SFP+ modules; Direct-Attach,
Fiber (10G or 1G), 1G/10G combo
1000BASE-T SFP Support
Supports 1G 1000BASE-T SFP modules
Low Latency
Cut-through architecture/intelligent interrupt coalescing
Receive Side Scaling (RSS)
Distributes IPv4, IPv6 loads across all CPU cores;
MSI-X minimizes interrupt overhead
Hardware Offloads
TSO, LRO, GSO; IPv4/IPv6; TCP, UDP checksums
Adapter Teaming/Link Aggregation
LACP for redundant links & increased bandwidth
(compatible with MLAG)
Jumbo Frames
9216 byte MTU for performance
Enhanced Tuning
Adaptive interrupt moderation
IP Flow Filtering
Hardware directs packets based on IP, TCP, UDP headers
Advanced Packet Filtering
4096 multicast filters; 4096 VLANs/port; adaptive TCP/UDP/IP,
MAC, VLAN, RSS, RFS filtering; Accelerated Receive Flow
Steering (RFS)
Intel QuickDataTM
Uses host DMA engines to accelerate I/O
Remote Boot
PXE, iSCSI boot; unattended installation
Virtualization Support
VMware ESXi; Microsoft Hyper-V; XenServer; Linux KVM;
Operating Systems
RHEL 5, 6, 7, MRG; SLES 10, 11, SLERT; other Linux;
Windows Server 2008 R2, 2012, 2012 R2

Wednesday, 1 October 2014

Zomojo / Exablaze's appeal comment

I'd like to thank Zomojo or Exablaze for their legal analysis of my appeal at this link.

If you're perhaps thinking there may be something interesting about trading or tech in this post, please stop reading now. Insomniacs need only apply. It is probably only tangentially interesting to those who have been following the three year old Zomojo/Exablaze v Zeptonics saga.

Without getting too carried away, I must agree with Exablaze's general thrust that my chances of success are dim but perhaps not for the same reasons. I certainly believe and hope that my view of justice will be agreed to by the Appellate Court but my personal history suggests that my likely outcome at the FCA will be yet another defeat. Lay self represented litigants are at a considerable disadvantage and have very little track record of success. Even if legal aid was available, I can't imagine a pro-bono lawyer getting too worked up about a fight between two HFT types regardless of the injustice involved. Lawyers serve a good purpose but they have to eat and can not represent poor people like myself. I am left to do what I can with the support of family and friends.

Some statements in the Zomojo / Exablaze communique are dubious at best. Let me meander though my thinking as to why.

One option of remedy being put to the Appellate Court is a constitutional writ of Certiorari. If this was granted by the court the entire case, including the Zeptonics bits, could be tipped out. I hope this happens and believe that would be just, but, yes, the historical record of my legal defeats suggest this is unlikely. The court is not a justice system. It is a contract resolution system where $ assist greatly in providing supportive argument.

This writ of Certiorari is different to the US use of the term where is it seems to be an order from an appellate court requiring the case to be forwarded. In Australia such a prerogative, or constitutional, writ is a remedy, quite a heavy handed one, that may result in the decision being overturned and details struck from the record:
"Certiorari is an order setting aside a decision (technically, the record of the decision-maker is removed to the court and the court then quashes the decision and expunges it from the record). An order for certiorari would be sought where a decision has been made unlawfully and the decision should be set aside." [Source: The Law Handbook web site.] 
Zomojo argue in their communique that Zeptonics is not represented thus decisions that have affected Zeptonics cannot be altered. This is not true as such a writ could result in the case being wholly undone. Also it is important to note that virtually any party, even a party unrelated to the original case, may attempt to appeal a court's decision by seeking such a writ if meritorious. For example, if the London whale had of been found guilty in the FCA in Melbourne, Greenpeace may have been entitled to run an appeal based around such a writ as an unrelated and very confused party.

More technically, the standing of a person seeking such a writ should be that of an “aggrieved person” which, according to the Queensland decision of R v Knyvell; ex parte Weber ([1929] St.R.Qd. 16; and see De Haber v. Queen of Portugal [1851] 17 Q.B. 171) could be as wide as “any person” following the tradition that unlawful dealings of tribunals under the cloak of law should be allowed to be challenged by any subject. The history of this relatively loose requirement for standing culminated in the decision in Attorney-General of Gambia v N’Jie ([1961] A.C. 617.), where the interpretation of the term “person aggrieved” was effectively “any person.”

Maybe my legal thoughts are off base and I'm barking up the wrong tree. Perhaps, just maybe, there is some legal merit to this line of reasoning. The court will adjudicate in due course if I can cross the t's and dot the i's and make it that far.

Zomojo argued vigorously to Honourable Justice Middleton that the appeal not be allowed to continue. The process was allowed to continue. Their QC did not win that argument, at least yet. The Full Bench of the Federal Court of Australia will hear the leave to appeal and appeal on a single day in 2015, likely to be February or March.

If you're having trouble sleeping, you can read my draft notice of appeal and a further submission from myself as they are public documents registered with the court.

I believe Zomojo is being a little misleading in their communique. They didn't really need to be as I acknowledge that, regardless of the merit, a self represented litigant is unlikely to succeed. I fervently hope to be an exception to that general rule as the scores of people harmed by the injustice of this mess deserve better.


Friday, 19 September 2014

Zomojo v Zeptonics appeal going to the full bench of the FCA

I've not had much success in the Federal Court of Australia but a small ray of hope broke through the gloomy clouds yesterday. A very small ray. Honourable Justice Middleton permitted [though maybe didn't prevent is a better synopsis] my appeal to move forward to the Full Bench of the Federal Court of Australia with a likely schedule of February or March 2015.

No, it's not the appeal I wished I could present, but at least the Court has had the good grace to allow this piece of due process. Even though it feels a bit like a lamb to the slaughter, perhaps, just perhaps, the Judicial Officers of the appellate court will be able see the underlying truth and find a way to rebalance the scales the justice.

Winning the appeal would indeed be a Pyrrhic victory as the harm that has come to pass cannot be undone. With the good ship Zeptonics destroyed, scores of people and their families affected, there can be no proper accounting to undo the evil. I do hope Zeptonics can get the set of its intellectual property back that was ordered by the Court to be assigned to Zomojo, some of which is being used by Exablaze, and that dependent customers are not overly affected as they were in the sinking of Zeptonics. Obviously I believe I have a good case and hope for the best, but my personal court history suggests my chances are slim.

Being broke and trying my best to be an orderly self represented litigant, I have much to do over the next little while to fight a proper appellate battle. I do suspect I'll have a busy Christmas and an unhappy wife. If there is any one out there with suggestions or formal legal advice, apart from “Are you completely mad?”, please drop me a line on
“Never give in. Never give in. Never, never, never, never—in nothing, great or small, large or petty—never give in, except to convictions of honour and good sense. Never yield to force. Never yield to the apparently overwhelming might of the enemy.” 
― Winston Churchill, Never Give In!: The Best of Winston Churchill's Speeches


Wednesday, 6 August 2014

Historic US microwave links and NY to Chicago in "34 Jumps"

Recently Jump Trading was in the news with their NATO tower story. It seemed to be well read with quite a few different brokers around the world mentioning it to me.

However, not many people really realise how old the news is. If you had of searched for a story with the words "Jump", "microwave", New York and Chicago, you may have come across this link as one of the first:

Click to expand. Source[1]: Originally September 1949 Long Lines Magazine
Yes, that's right. September 1949. A whole sixty five years ago next month. The second page of the article is below at [2].

A New York to Boston link went up in 1948 [3]. East joined West with the "Skyways" intercontinental link, with obligatory White House telephone call christening on August 17, 1951. It referred to operations at four billion vibrations a second[4], so presumably it was 4GHz microwave technology.

The network grew in size and here is the footprint of the US National "Broadband" network in 1960:

Click to expand. Source[5]: US 1960 Broadband Network
Microwave travels at around 3.3 microseconds per kilometre and light through optical fibre at around 5 microseconds per kilometre. AT&T's first "slower" long line fibre link went live in 1983 / 1984:

Click to enlarge: AT&T's first long fibre link
We have to be careful about books like "Flash Boys" that twist history into falsehoods to make catchy plot points. I guess we'll be hearing about new TransAtlantic high frequency trading stories soon, but remember that the first cable went across in 1858. There is a long, rich history of incredible stories in telecommunications that more people should be aware of.

Click to enlarge: Source [7]

I'll let you into a little known secret: that early transatlantic cable may have been 3 to 4 milliseconds faster than the current fastest fibres for signals crossing the ocean. At least for the three weeks of operations it had. Ask an engineer why that may have been the case...

Happy trading,



[1] Sourced from a cool site about the old long lines:

[2] The follow up page from [1]

Click to enlarge. Source
[3] From the Boston NYmicrowave link article from 1948:

[4] Nice pamphlet on the 1951 Skyways originally from

[5] Original 1960s map from long-lines:

[6] An earlier long-lines map via long lines from the Bell System Technical Journal, September 1952:

[7] Early Transatlantic Cable from from

[8] Here are a couple more maps from [7].

Tuesday, 15 July 2014

Mellanox UDP latency = 1.1 microsecond

Mellanox's VMA release notes for VMA version 6.6.4 have updated Netperf UDP RR benchmarks. The table below is  for one of their Ethernet cards.  1.1 microsecond is a good result for a well priced and well supported card with a comprehensive feature set.

Not quite Dolphin PCIe territory where they claim 0.74 microseconds but Dolphin's socket implementation on top of PCIe (not Ethernet), if you chose to use it, would make the solution a little worse than Mellanox according to their Dolphin's supersocket reporting (1.26 microseconds).

Has anyone some public SolarFlare or Chelsio numbers? I'd love to compare.

Happy trading,


Monday, 14 July 2014

Zomojo / Exablaze developments - firing staff

Some interesting, to me at least, news about Zomojo / Exablaze the last couple of weeks.

I hear a bunch of staff have just been fired and are interviewing around town. In fact, I've been told by more than two sources now that Phil Manuel, the SA, is the only full time staff member left at Zomojo with some of the older crew working two days a week with the rest terminated. One of the guys I hired back in 2009 has a one month old baby and a two year old and is now looking for work. Harsh. He is a good guy. Zomojo is evil but there are some good, perhaps deceived, people involved I have considerable sympathy for. Good-luck to them and their families.

I guess it is not surprising to hear how Exablaze are now having yield problems with manufacture as quality problems might be more commonplace with a device you didn't conceive of that requires modifications (the management board was end of lifed, for example). Quality control, tsk, tsk. I can't imagine you'd be feeling comfortable if you were an Exablaze customer.

The two day a week part-time slavery for some staff is interesting. I wonder if it so they can avoid the long service leave entitlements most of them are due? Perhaps it is just a hook to entrap them and keep them from plying their craft elsewhere?

I understand Zomojo capitulated / settled with a trading firm they were suing last year in the first half of 2013. As far as I can tell from the court files, that firm, a completely innocent party, still hasn't been paid their due costs of some hundreds of thousands of dollars for their loss as agreed. That has been drifting out for over a year now. There is another court question this Thursday regarding paying for an IT firm Zomojo had work done last year as part of their legal case against me and that firm has not been paid for over a year now for work done. PPB Advisory should have been aware that if they were going to lie with dogs they may get fleas.

Makes you wonder if Zomojo is solvent?

Then, just now, this article turns up in Korea: FUTURES AND ZOMOJO?! (A better English translation is now below at [2]) I think they may have the wrong broker but the gist of the story may be true enough. The broker mentioned is a solid broker and it would be a shame if they are suffering from an unwarranted Zomojo contagion.

Zomojo lie in court. Zomojo mislead their customers. Greg Robinson was terminated by ITG. Robinson's Bellwether Share Fund investment and directorship didn't turn out too well. Now he has driven Zomojo off the side of the road. He is indeed a man with a black thumb for the corporate garden. If you hear his footsteps behind, you may be justified in quickening your pace.

I do feel a bit sorry for Robinson as one day his family will wake up and realise he is just a pseudo-intellectual fraud.  He is good to have a beer with, pleasant to talk to, have a round of golf with, but deeply flawed nonetheless. A person of no substance. Taking my wife's super from a decades of teaching. Vilifying my father unjustly after went to great lengths to assist them. Robinson is trying to bankrupt the good citizen in his twilight years. Causing my family enough hardship and stress to put one of my daughters in hospital. Nasty. I guess I'm a little biased, so make up your own mind. Caveat emptor.

The second director, Dean Boyle, was just a lap dog sniffing Robinson's butt. However, it is Ian Heddle who is the true despot in that place.  It is not really worth any energy or thought wasted as whereas Robinson is more akin to a knife carrying sneaky fart, Heddle is just a plain sh.... Obvious on sight.

I still hear ex-Zomojo people refer to Heddle as Bubble. I thought the nickname was just because of his bulbous head which is out of proportion with his already large frame. Being above the fray (can't you tell) I never questioned the origin of the nickname. After about a year, I found out they called him that because he was like the boy in the bubble (with apologies to David a heroic lad). Heddle came in to work, never did anything, never interacted and left before anyone else. He lived in his own little bubble. He'd vilify staff and not talk to people for weeks on end, including myself. This is why I left. It was a soundly unpleasant place to work and it got to the point where you just didn't want to turn up to work. Leaving was both the hardest and the easiest decision.

I think the funniest story I've heard about that horrid time is the game some of the staff used to play, "Guess what Bubble is doing today." Each day they'd have a sweep on what he would be doing at exactly 11am, as he didn't do a great deal at all. It got a bit monotonous as each day it turned out to be browsing the web. Every day. All day even. Not just 11am. Worse. The guys couldn't break it down much further as it turned out to be almost always cartoons. Not the graphic novel, mature, artistic kind. The simple kiddie stuff in the main. The game was retired as it turned out to be a closed ended problem and of no scientific merit or wagerable outcome.

I can't imagine why Zomojo no longer makes money and have reduced to a skeleton staff . Can you?

Happy trading,



PS:  I still own a third of Zomojo Trading Pty Ltd. No annual reports. No information at all. It is the vehicle through which all the revenue comes in, or at least it used to be that way. They seem to be violating a bunch of ASIC and ATO rules. Not sure what to do with the holding as I certainly can't afford legal advice any longer.  If you have any creative ideas, let me know ;-)

“Hyundai futures providing unfair advantages via FEP?” HF states ”this is a false claim” Claims of Hyundai Future providing its foreign clients with methods of unfair advantages of FEP and lease line, giving them an advantage and harming normal investors, have been circling the market. But Hyundai Future has stated that this was a false claim.

“A” securities firm director “B” stated, “Hyundai Futures has provided a FEP with direct connection to KRX’s order system to Zomojo for a monthly fee of KRW 200 million”. The director also said “Zomojo directly manages the servers and even install their own programs and place unfair trades”

Zomojo is known to be an algorithm trading firm using their programs. FSS regulation states that firms including Hyundai Future are prohibited to provide clients like Zomojo with special services, information or equipment.

It is also prohibited for the clients to install their own programs as Director “B” has stated. This gives the special clients the advantage by having higher speeds than KRX’s standard compared to normal investors.

On the other hand, Hyundai Futures stated that they have never provided Zomojo with the servers or even opened an account with them. HF representative said, “we are aware that Zomojo is a specialist in the FEP field, and we have met with them, but it was because Zomojo proposed to sell their software to us.”

“We have declined Zomojo’s offer, and the claims that HF is using Zomojo’s software is false.” It has been identified that Zomojo has reached out to other firms as well as HF. Therefore, it is a possibility that other firms also provide services in an unfair manner such as HF.

Person in the brokerage industry stated, ”it is tough to identify if the securities or futures firm is utilizing these programs in the servers without visiting the sites.” “foreign algorithm trading accounts in Korea has reached over 1200 accounts since March”

A KRX personnel said, “only way to identify these illegal algorithm trading is to inspect their IDC directly”, “but even if we charge the firms with suspicion, the firm will simply decline to provide information stating it is their confidential business information.”

FSS personnel said “we have sent guidelines as this has been a problem before”, “instead of criminal charges, it would be easier for the government to address disciplinary actions through administrative measures”

Tae Woong Ryu Reporter (Aju News)

Sunday, 15 June 2014

Nice Terasic PCIe 3.0 FPGA board

I do like the look of this board, the Terasic TR5-Lite even if it is not quite as sexy as a Triumph TR5 from my birth year. Any feedback on the actual use of one of these would be most welcome.

Stratix V, half height, half length with a firewire (1394) shaped 422 connector for timing input. Direct connectivity to the dual SPF+s. 36MB QDRII+, 2GB DRAM and PCIe 3.0 capable from the x8 edge connector makes it look good for HFT.

It uses a Altera Stratix® V GX FPGA (5SGXEA7N2F45C2) so you get 622k logic elements. Dunno what it is, but the board just looks nice and elegant. It only seems to have 4 user controllable LEDs  and five status LEDs, so Mike Dini wouldn't approve[1].

Whilst an Arria V or Kintex-7 solution would be better value for 10G, Terasic seems to have put together almost the perfect minimal FPGA HFT package here. I'm still thinking a Zynq-7100 based board with its useful ARM is a better idea overall though.


Friday, 6 June 2014

Finra ATS stats

Finra has made a stab at releasing ATS / dark pool stats.

It makes for some interesting reading. I've put the stats into a table below as I've found the Finra web site has been a bit up and down. Is that ITG and IEX running neck and neck with 100M shares a piece?
Finra ATS / Dark Pool Stats - Source: Finra
(Note the % are bunk, though indicative, as the dates are different)

Name Shares
Shares/Trade Date
CROS CREDIT SUISSE SECURITIES (USA) LLC 376,207,777 12.1% 2,189,977 13.5% 171.8 06/05/2014
LATS BARCLAYS CAPITAL INC. 307,354,920 9.9% 1,701,372 10.5% 180.7 06/05/2014
UBSA UBS SECURITIES LLC 282,217,511 9.0% 2,307,746 14.2% 122.3 06/05/2014
MLIX MERRILL LYNCH, PIERCE, FENNER & SMITH INCORPORATED 215,734,880 6.9% 1,099,830 6.8% 196.2 06/05/2014
MSPL MORGAN STANLEY & CO. LLC 207,196,400 6.6% 780,930 4.8% 265.3 06/05/2014
DBAX DEUTSCHE BANK SECURITIES INC. 189,134,200 6.1% 1,117,870 6.9% 169.2 06/05/2014
SGMA GOLDMAN SACHS EXECUTION & CLEARING, L.P. 182,887,381 5.9% 1,094,422 6.7% 167.1 06/05/2014
FLOW LAVAFLOW, INC. 154,595,446 5.0% 862,489 5.3% 179.2 06/05/2014
KCGM KCG AMERICAS LLC 135,074,012 4.3% 979,475 6.0% 137.9 06/05/2014
JPMX J.P. MORGAN SECURITIES LLC 113,052,036 3.6% 422,060 2.6% 267.9 06/05/2014
ITGP ITG INC. 106,841,795 3.4% 415,319 2.6% 257.3 06/05/2014
IEXG IEX SERVICES LLC 101,722,565 3.3% 304,784 1.9% 333.8 06/05/2014
BIDS BIDS TRADING L.P. 96,849,500 3.1% 221,827 1.4% 436.6 06/05/2014
KCGB KCG AMERICAS LLC 65,498,506 2.1% 574,660 3.5% 114.0 06/05/2014
ICBX INSTINET, LLC 64,843,400 2.1% 269,894 1.7% 240.3 06/05/2014
EBXL LEVEL ATS 60,250,900 1.9% 378,885 2.3% 159.0 06/05/2014
CXCX CITIGROUP GLOBAL MARKETS INC. 57,960,326 1.9% 300,479 1.9% 192.9 06/05/2014
DLTA DEALERWEB INC. 55,129,108 1.8% 103 0.0% 535,234.1 06/02/2014
XSTM NATIONAL FINANCIAL SERVICES LLC 42,356,635 1.4% 113,495 0.7% 373.2 06/05/2014
LTPL CREDIT SUISSE SECURITIES (USA) LLC 34,167,610 1.1% 287,279 1.8% 118.9 06/05/2014
IATS INTERACTIVE BROKERS LLC 33,835,637 1.1% 97,090 0.6% 348.5 06/05/2014
LQNT LIQUIDNET, INC. 33,593,400 1.1% 811 0.0% 41,422.2 06/02/2014
APOG CITADEL SECURITIES LLC 30,638,504 1.0% 180,977 1.1% 169.3 06/05/2014
NYFX CONVERGEX EXECUTION SOLUTIONS LLC 26,367,524 0.8% 60,292 0.4% 437.3 06/05/2014
MSTX MORGAN STANLEY & CO. LLC 23,245,500 0.7% 141,446 0.9% 164.3 06/05/2014
PULX STATE STREET GLOBAL MARKETS, LLC 19,573,064 0.6% 1,743 0.0% 11,229.5 06/02/2014
VRTX CONVERGEX EXECUTION SOLUTIONS LLC 14,949,562 0.5% 69,232 0.4% 215.9 06/05/2014
PDQM PDQ ATS, INC. 14,866,289 0.5% 79,138 0.5% 187.9 06/05/2014
LQFI CITIGROUP GLOBAL MARKETS INC. 13,779,198 0.4% 242 0.0% 56,938.8 06/02/2014
INCA INSTINET, LLC 13,059,700 0.4% 16,985 0.1% 768.9 06/05/2014
WELX WELLS FARGO SECURITIES, LLC 11,996,652 0.4% 68,802 0.4% 174.4 06/05/2014
LQNA LIQUIDNET, INC. 9,010,800 0.3% 3,586 0.0% 2,512.8 06/02/2014
BOOK BLOOMBERG TRADEBOOK LLC 8,655,811 0.3% 39,562 0.2% 218.8 06/05/2014
MLVX MERRILL LYNCH, PIERCE, FENNER & SMITH INCORPORATED 5,508,700 0.2% 13,224 0.1% 416.6 06/05/2014
AQUA AQUA SECURITIES L.P. 5,072,599 0.2% 305 0.0% 16,631.5 06/02/2014
LEHM BARCLAYS CAPITAL INC. 2,913,308 0.1% 20 0.0% 145,665.4 06/03/2014
RCSL RIVER CROSS SECURITES, LLLP 1,330,693 0.0% 7,266 0.0% 183.1 06/05/2014
MSRP MORGAN STANLEY & CO. LLC 1,115,700 0.0% 2,112 0.0% 528.3 06/02/2014
FNBR FOLIOFN INVESTMENTS, INC. 379,218 0.0% 11,191 0.1% 33.9 06/05/2014
WDNX WEEDEN & CO.L.P. 308,590 0.0% 514 0.0% 600.4 06/02/2014
CURX CURIAN CLEARING, LLC 184,094 0.0% 3,787 0.0% 48.6 06/02/2014
JEFX JEFFERIES EXECUTION SERVICES, INC. 6,700 0.0% 56 0.0% 119.6 06/02/2014

Total 3,119,466,151 100.0% 16,221,277 100.0% 192.3