Astea Keynote Address

I was honored to give this talk, addressing Astea people and the students and faculty of the FMI (the Sofia University Faculty of Mathematics and Informatics, “Bulgaria’s MIT”)

We turned my limited bandwidth into a benefit by pre-recording videos for each of the talk segments, and keeping the “full story” videos to compliment the ones we trimmed to fit the keynote’s 40-minute timeslot. Here are those longer segments, with the points each discusses.


P.S. I won’t be offended if you watch at a 1.25× speed (I do!..   ; )   [gear->Playback in Chrome; lower-right 1.0× in VLC]

P.P.S. If you’re wondering about the 100× speedup in development I ascribe to Paper Prototyping, or why I rigorously mean “Extending the Mind,” please see this omissions page.



· “I’d like to change how you think…the functioning mechanisms of your thought processes.”
· The mind has an “input/output protocol” and we don’t currently engage with it well
· If we do so at the non-recurring engineering phase we add hugely to a delivered project’s value
· Cognitive Engineering needs to be called out as a separate discipline (& why)
· Cognitive Engineering applies perceptual and cognitive sciences, but software engineering patterns apply
· We read and apply the real science, but our partitioning of the mind is not the scientist’s; they care about how things work, we care about how to optimally use them.
· Our first approximation to the mind’s i/o protocol:
    Perception -> Recognition -> Association -> Understanding -> Analysis -> Insight -> Action
· We co-evolved with things; things help us thing
· J.J.Gibson identified affordances in the ’50s; we’re good at recognizing affordances but not at storing them; good at recalling uses & behaviors when we see them
· A thought experiment: look out the window and see millions of things easily; trying to show millions of things in a standard UX toolkit is impossible
· We can tailor our work to fit the mind’s input protocol and show not millions of affordances, but many more than the dozens in contemporary UX designs



Let me show you interfaces where we tried to do that tailoring

TextArc, a nerdy tool to show distribution of words in text (but written up in the NY Times as art, and shown in MoMA)
· Thousands of words shown, but not overwhelming because of perceptual layering

Does Cognitive Engineering replace graphic design? Absolutely not: Tora Trading EMS
· Information Layering
· Does the additional visual richness actually help us think?

Steps building a single slide (a block architecture of the Pool)
· Ecological Psychoacoustics: we don’t hear sounds, we hear the physical properties of the object making the sound.
· Current UX work does not allow this: every reference exhibits indirection, crippling our ability to work
· We can draw things so that people see the actual point of the object: direct perception of the data



Goldman Sach’s NYSE Broker Handheld, first breakthrough in the Cognitive Engineering Design Methodology
· Physical artifacts (like brokers’ paper execution pads) evolve to fit their ideas under the direction of experts
· We can copy their result and they’ll recognize it
· Or we can invent a new object that transcribes the defining intellectual characteristics of their thought objects
    —and they’ll recognize things they’ve never seen before

NYSE’s own Broker handheld
· Sped up 10-15 seconds for filling out a form to 6 seconds; not good enough because electronic executions were done in a second
· Created “blueprint” executions for “refills,” their most common operation
· They stored all execution details except the key degree of freedom brokers needed to change: price
· Price was determined by where you dropped it; high for an aggressive buy order, low to wait for a price dip
· Independently tested, the best brokers now did the same thing in 1/2 second—this kept brokers relevant on the NYSE floor for a couple more years



Putting recognized objects together for more complex ideas

NYSE Specialist’s Workstation
· Five to six times the information on the screen if you count characters
· But more like 50 times the information if you count the number of inferences and actions you can make: a harder-to quantify metric, but the only one that matters

Relationships Among Scientific Paradigms
· Associate hundreds of groups of papers to make higher-level insights
· A static print can be more interactive than a computer program

· Draw a node-link digraph, usually an uninterpretable hairball after a few dozen links
· We draw 5,000+ nodes representing companies and 20,000+ links representing supplier/customer relationships for supply chain analysis
· But can still make dozens of inferences when seeing the entire graph
· And can still zoom in to compare two companies’ details, like how one supplier’s failure could strongly impact two companies



XTL (an Extreme Trade List EMS)
· Redesigned the “blotter” to follow mental associations among information types: though chunks
· But don’t just dump all the information in front of someone, forcing them to do the meaningful integration
· Make one tool for each task—optimized around that task, like:
· The Alert Monitor, optimized to find unusual orders
· The Active Order Landscape, optimized to find trades to fix
· The Order Handler, optimized to select the best person/firm/algo to execute the trade
· Allowed a qualitatively different, better selection—thought to be impossible before: based on all factors (not just the two or three one could look up before)
· In general, making a tool for each task refactors the trading environment around the true scarce resource: the trader’s time/attention, eliminating risk, errors, and increasing effectiveness in less time
· Refactoring diagram (appendix A) & deeper case white paper study here
· The only way to do effective systems architecture and engineering for a set of real-world tasks is to understand how they’ll be used
· Cognitive Engineering does this at the right level of granularity: workflow tasks, and mental task steps

A major bank’s Energy Trading System led us to “invent” composite fields: combining several data fields on one line
· One of the clearest distinctions between the (unacceptable, here) results of decent design by a salient Global markets UX vendor and the results of the Cognitive Engineering Design Methodology
· Composite fields map to the (Psycholinguistic) cognitive chunks actually used in the domain of practice
· The screen was 1/3 to 1/4 the size, but clearer
· This allowed them to generate over 30,000 different input screens that were nonetheless instantly recognizable because the chunks were themselves defining characteristics for the higher-order energy products
· Recognizable without system training because it tapped directly into the domain-of-practice training the experts already had internalized
· Unsolicited quote from project manager: We’ve rolled out most of your designs into production and the feedback has been positive, people booking complex products without having to understand the application or been given training


Analysis, Insight, Action

Demonstrated by building three simple data circuits in the Pool
· Data flow in an IoT context
· Direct perception of data (no indirection)
· And direct manipulation of the data
· Integration, visualization, and simple analysis in a portfolio management context
· Slicing and Dicing a data cube in an insurance portfolio
· Using a special-purpose console, tuned to completely satisfy one element in a task—we could make more complex consoles and enable entire tasks
· Self-documenting circuits can be easily shared with peers & other roles
· Encapsulation: can make a circuit into a module that can be re-used wired into other higher-level workflows
· Strongly-typed visual programming language, yet we expect it to be useable by more people than can current use spreadsheets; truly democratizing access to data

Case study: Cogentity

· Applying Cognitive Engineering to the complexities of revenue/cost flows and attribution
· Shows the contribution of every person’s efforts toward the firm’s goals
· Rolls them all up into company valuation metrics, where underperformance of any metric can be traced back down to the functions/people responsible
· One view replaced a 55-slide deck used in an actual board meeting at a top-100 software company, and contained 3 times the information
· All in one place, so complex or distributed problems can be understood and fixed
· Marketing funnel solution walk-through, a counterintuitive solution to serious problem (from a real-world application of Business Mechanics)



Can Cognitive Engineering be learned?
· Yes, it’s meant to be like an engineering handbook: easy to apply
· Extract & capture a Mental Model
· Support tasks entirely in a Schematic
· Map the schematic to available/affordable tech to create Candidate Designs
· Test before developing with Paper Prototypes

If user interfaces are 2-5× more effective since the early ’70s, that still compares poorly to the 50,000,000,000× improvement in computer hardware

We need to start focusing on the only thing that makes many systems meaningful: their connection to the human mind