Opinion Knowing when the universe is trying to tell you something is a core competency for many paths through life. You can’t pass an exam in that, sadly, but perhaps we should consider what links these three stories: Microsoft moving to open-book exams for certification, ChatGPT passing law and other exams, and people are tearing their hair out trying to recruit an IT skilled workforce. Could it be that the way qualifications work doesn’t pass muster?
In some cases, this is demonstrably true. If your law school exam can be passed by ChatGPT, your law school exam is broken. Not because a computer passing an exam is an automatic red flag – some multiple choice exams can be aced by a ZX Spectrum running a BASIC program, but they’re still useful in context. However, when that context involves screening for professional ability in a field that affects other people’s lives, you don’t want to be testing them for things that can be done by an hallucination-prone text predictor with no concept of reality.
So let’s move onto our own turf. What are IT qualifications for in 2023?
They can certainly save time and effort for recruiters. If you set an automatic filter on submissions searching for an appropriate degree, or a suite of industry-recognized letters, you’ll have fewer resumes to read. You will also miss out on a ton of talent, and if that seems a reasonable compromise then that talent will certainly be better off elsewhere.
This effect, the filter effect, may go some way to explain why the larger a corporation gets, and the more rule-based its hiring policy becomes, the more the place becomes a useless collection of business tailoring. It can be positively alarming to see this in action, especially in companies whose creation stories include founders who dropped out of college.
At the other end of the spectrum, a good degree in science or math is going to give you a hearing at any number of places who see those as evidence of high quality analytical abilities. There’s nothing wrong with that, except that it’s its own ecosystem, one that does little to address the capacity of most people to do useful work for the IT requirements of most places.
Exams also demonstrate that a person’s capacity to do a certain class of work is not necessarily dependent on having explicit training in that area. The complication arises because exams have to test something concrete. In the days when knowledge was hard to access and slow to change, that was a good thing to test for. Now that all of the knowledge is in all of the places, all the time, being able to search it, sift through it, and synthesize it is what matters.
At all levels of IT, that’s important. The rate of change can be rapid and unpredictable, with a comfortable specialty in one set of skills at risk of becoming superfluous overnight if there’s a change in platform or vendor. The smarter interviewers have long known to ask candidates what sort of resources they use to solve problems or rapidly learn new requirements – from Slashdot through to Stack Overflow and Reddit, knowledge of the most relevant places for the best information is key. Against that, a formal qualification may have led the candidate to a long period in a comfortable niche.
Microsoft’s open book tests are something of a recognition of this, although the books that can be opened are simply those that Microsoft provides. This is great news if the most important industry requirement is planting and weeding inside Microsoft’s walled gardens, but less good if you’re looking for broader awareness.
You can think of your entire career as an open-book exam, just one with gratefully fuzzy grading. Open source is even closer to that idea, where project ownership or participation is a very good signal for the switched-on recruiter.
Let them in
Open source isn’t so good culturally, and it can be an off-putting place if you don’t fit in or don’t have the confidence to cope with its less nurturing aspects. For a tradition that should be purely meritocratic, it’s curiously lacking in diversity. That’s a sign of a lot of people not signing up for a sector desperately short of people.
We need an on-ramp for talent that leads to a lifetime pipeline of skills acquisition, replenishment, and documentation. A combination of living resume and exploration guide, it needs to be vendor and platform agnostic, it needs to be appealing to the young and curious as well as the older, more determined careerists. Think of it as an open framework for curating humans as they do IT, with room for curated content, communities, pathways and tools, sandboxes and showcases. Built around humans with the best of technology in the service of teaching that technology, it might sound expensive. It really isn’t.
We’re a rich industry where Meta can burn tens of billions on vapid VR and me-too social media, and nobody can count how much Google has splurged on failed projects of no detectable merit. For a tiny percent of the world’s tech revenues, a new, universal, and frankly fabulous education system could be built and deployed with the help of the finest pedagogues available. The result would solve the industry’s most pressing need, for good people doing good work, and through expansion into other areas benefit us more than AI will ever manage. Although whether it will ever produce a legal system capable of telling decent lawyers from insane robots – well, we can hope. ®
