ARCANUM 92 / ~41 MIN READ / SOURCE PAGES 1461–1491
The Scoring Society, Behavioral Prediction, and the Right to Unpredictability
The Algorithmic Condition in Light of the Philosophy of AI
Every society categorizes. Long before artificial intelligence, institutions already assigned categories, statuses, grades, and files. Schools grade, banks assess risk, insurers calculate premiums, companies evaluate skills, and government agencies verify rights. It would therefore be wrong to present scoring as an invention of AI. What is changing today is its depth, its frequency, and above all, its ability to flow between previously separate domains.
A system can now cross-reference hundreds of variables, construct a probability, and detect profiles that no one could have calculated manually. This power can be extremely useful. A bank can better assess risk. An insurer can price more accurately. A government agency can detect fraud.
But as prediction becomes more precise, a question appears: at what point does a person cease to be judged by what they do and begin to be judged by what a model thinks they will do? That is the threshold. The score is no longer merely a memory of the past. It becomes an anticipation of the future. And that anticipation can produce consequences before the event has even occurred. A person may be denied credit because they statistically resemble people who repaid less reliably. They may be subjected to greater scrutiny because certain signals increase a probability. They may be treated as a candidate for leaving a company before ever expressing an intention to leave. We are moving from observation to preemption.
This logic must be examined with extreme caution because it possesses real economic efficiency and an immense anthropological danger. The efficiency stems from statistics. If certain variables are correlated with a behavior, the model can reduce uncertainty. The danger arises when we forget that probability belongs to a distribution, not to an individual as destiny. To say that a group has a higher risk does not mean that this individual will do what the group does on average. The difference between population and subject lies here. Statistics know a great deal about groups. The law protects individuals. This tension is long-standing, but AI gives it new power. The more precise the model becomes, the more we may be tempted to consider its prediction as personal truth. A score of 87 percent seems almost like an inherent property of the individual. But it remains an estimate conditioned by data, a model, and a context. This is why I believe the right to unpredictability must become a central concept in the algorithmic age. This right obviously does not mean it is forbidden to predict anything about human behavior. That would be absurd. Every economy uses predictions. It means that prediction should not become identity, much less condemnation. A person must retain the right to defy the pattern. That is almost a definition of freedom.
If all past behavior becomes material for predicting the future, and every prediction produces a consequence that narrows the available options, we can gradually imprison the individual within their statistical trajectory. The score acquires a self-fulfilling effect. A person deemed risky receives fewer opportunities; lacking opportunities, their path deteriorates; the system then concludes that it was right. This is an extremely dangerous loop. The model no longer merely predicts reality. It helps manufacture it. Reflexivity reappears here, applied directly to human life.
This loop is particularly visible in credit. A person excluded from financing has less capacity to invest, to train, to buy a home. Their economic future is altered by the assessment. This does not mean that risk analysis should be abolished. Risk-free credit does not exist. We simply need to recognize the causal power of the score. A credit decision is not merely an observation. It can modify the future it purports to measure. Hence the need to make the system open to challenge and correction.
What data produced the score?
Are they accurate?
Are some of them obsolete?
Are there contextual elements that the model does not see?
Without an appeals process, statistics risk becoming an administrative burden. We find ourselves back at the same fundamental principle: no algorithmic power without recourse. But this requirement must be concretely defined. Giving someone a page of incomprehensible variables is not a real recourse. They must be able to understand the main factors that influenced the decision, report an error, present new information, and obtain a human review in critical situations. This is not a luxury. It is a condition of justice.
The same logic applies to insurance. A system can calculate individual risk with great precision. This seems economically rational: everyone pays as close as possible to the risk they represent. But if we push this logic to its logical conclusion, a contradiction emerges. Insurance exists precisely to pool uncertainty. If individual risk is perfectly predicted, insurance becomes akin to charging for fate. Those with high risk pay exorbitant amounts precisely because they most need protection. Actuarial optimization can then conflict with solidarity. This is yet another demonstration that technical optimization alone is insufficient. A society must decide what share of risk it wants to pool. Health, disability, disasters. This choice is both political and moral. AI can better calculate risk, but it does not decide what level of solidarity we want to maintain.
Do we want perfectly individualized insurance or do we accept that part of the risk is shared collectively?
There is no purely mathematical answer. And the increasing precision of AI can make this conflict more visible. For a long time, certain forms of solidarity existed precisely because we could not measure things accurately. Ignorance necessitated mutual support. As ignorance diminishes, solidarity must become an explicit choice. This is a civilizational threshold.
The advancement of knowledge sometimes removes the protection of uncertainty. We must decide whether we want to rebuild it legally. This is an extremely important idea. Freedom has long benefited from the technical impossibility of knowing everything. Solidarity, too. When technology reduces this impossibility, our institutions must voluntarily set limits. Again, freedom through self-limitation of power. Genetic insurance already offers a concrete legal example. In France, for policies covering disability or death risks, the law prohibits insurers from taking into account the results of genetic testing, even when provided with the individual's consent. The principle is remarkable: data can be predictive yet excluded from insurance optimization because society decides it should not become a pricing criterion.
The same reasoning applies to other inferences. A model might be able to deduce a health status from purchases or travel history. The fact that information can be inferred does not mean that it should be usable. This is why the doctrine of inferences that I discussed in previous chapters becomes central here. Privacy is no longer just about what the individual has revealed. It is about what the machine can reconstruct.
The ability to make inferences is the new power.
The law must therefore protect certain conclusions regardless of how they were obtained. This may seem complex, but it is essential. Otherwise, prohibiting direct access to sensitive data is almost pointless if the model can statistically reproduce it. We then enter a society where individuals are transparent without having declared anything. This algorithmic transparency can be extraordinarily intrusive. An individual may not even be aware of certain trends that the system believes it detects. The company may know that an employee is considering leaving before they have clearly articulated this decision. What should it do with this information?
Can it deny the employee a promotion because their departure score is high?
That would be very dangerous. The model could create precisely the departure it predicts. Again, the self-fulfilling loop. We must therefore distinguish a prediction intended to anticipate a collective need from a prediction used to sanction an individual. A company may forecast that 10 percent of employees in one department are likely to leave so it can prepare recruitment. That is very different from identifying three people by name and treating them as already disloyal. The granularity of prediction raises a question of dignity. The aggregate may be legitimate where the individual becomes intrusive. That is why minimization concerns not only the quantity of data, but its resolution.
At what scale do we really need to know?
Precision is not an absolute value. Finer information can be useful for some functions and intrusive for others. The right resolution depends on the purpose.
Organizing a city's transportation requires understanding flows. It is not necessarily necessary to know precisely where each person is going at every moment. To predict staff turnover, trends can be used without profiling each employee. A mature algorithmic society will be one that deliberately chooses to operate at a lower resolution when the increased precision does not justify the intrusion.
That requires a discipline of design. Engineers are naturally drawn to precision. The law must ask: useful for what? This is the principle of informational proportionality. And it leads me to a notion that seems essential: the right to compartmentalize scores.
A credit score should not automatically become an employability score. An assessment in one domain must not contaminate all the others. History has already known social systems in which a single category produced a cascade of consequences: caste, status, criminal record. AI could reconstruct a form of digital caste through the accumulation of scores. The danger is real. If a person has a low score on one platform and that score is then used by an insurer, a landlord, or an employer, they may gradually be excluded from multiple dimensions of life. The score becomes a total social identity.
This is precisely what must be prevented. Every assessment must remain linked to its purpose. This is an extension of the legal principle of purpose limitation. A score is an answer to a specific question: "What is the risk of default for this loan?" It is not a general measure of a person's worth. This distinction must be rigorous. The temptation to reuse data will nevertheless be strong because it exists. A pre-calculated score costs almost nothing to share. This is where the law must resist the allure of convenience. The fact that data is available does not create a universal right of use. It is exactly like memory. Information can be true and yet not necessarily be used everywhere.
Forgetting and compartmentalization are conditions of freedom. This idea is very important. We often think of privacy as secrecy. It is also contextual. I can give information to my doctor without wanting it to be used by my banker. It is not that the information is shameful. It belongs to a relationship. AI, by connecting all databases, threatens this contextuality. The total graph is technically appealing. Socially, it can be destructive. That is why an algorithmic civilization must know how to create barriers within the graph.
It is almost a complete reversal of my entire life with databases, where connecting creates value. Yes, connecting creates value. And precisely because it creates power, sometimes you have to choose not to connect. There is no contradiction. Maturity lies in knowing where the relationship is legitimate. An Art Market database should connect artist, artwork, sale, Auction House. It has no reason to connect the collector's medical data. The purpose defines the graph. The danger begins when the graph seeks to become the total world. That is the hubris of data. The idea that since everything can be connected, everything must be connected. I reject this idea. Living chaos needs discontinuities. So does freedom.
A society where every behavior contributes to an overall score becomes profoundly conformist. Why take a risk if a mistake could affect your credit, your insurance, your job? Individuals begin to optimize their lives for the rating system. This is a huge second-order effect. The score no longer simply measures behavior; it disciplines it. Foucault would undoubtedly have recognized this logic. The Panopticon produces self-control because the prisoner knows they can be observed. The scoring society adds another dimension: we know that every trace can modify a score. We become managers of our own algorithmic reputation. This can produce caution, but also a loss of spontaneity.
A society where everyone constantly seeks to maintain a favorable score becomes a society of standardized behaviors.
This is a threat to creativity and dissent. Innovation sometimes involves doing something that seems strange or risky. If every atypical behavior lowers a reliability score, the system penalizes those who deviate from the norm. This is why the right to unpredictability is also a right to innovation. Only those who deviate survive, but the system must first allow them to exist. This is a fundamental contradiction of the algorithmic economy. It seeks to detect anomalies because they can signal risk. But an anomaly can also be the source of novelty. The same signal can represent fraud or innovation.
How to distinguish?
Context. Again. A model can identify statistically rare behavior. It should not automatically consider it negative.
The anomaly is a question, not a verdict.
This statement should be a principle. It applies to security, work, and finance. The algorithmic society must preserve the possibility that the exception might be a good one. This is precisely the method of the Abode of Chaos. What appears as an anomaly in the landscape may be a Total Work of Art. Those who apply only the visual standard conclude that there is a malfunction. They do not see the structure. It is almost a perfect metaphor for the society of scores.
A system can measure deviation from the average. It does not automatically measure the magnitude of that deviation. The artist, the entrepreneur, the dissident are often statistical anomalies. A society too optimized around the average could become incapable of generating its own mutations. This directly relates to the previous chapter on exploration. An economy must maintain areas where individuals are not penalized for deviating from the expected profile.
School is a particularly important domain. Imagine an education system that, from childhood, constructs probabilities of success in different occupations. This may be useful for offering personalized support. But if the score becomes a definitive orientation, it is extremely dangerous. A child changes. Learns. May reveal themselves late. An early prediction can imprison them in a trajectory. This is the risk of self-fulfilling prophecy in its gravest form.
AI should be used to open up opportunities, not close them off prematurely. A model might say: this student seems to be struggling with math, let's suggest another method. Very well. It should not say: this student is not suited for science. The difference is enormous. The first approach treats the score as a temporary diagnosis. The second as an identity. There you have it, the right to evolve.
A good educational architecture must allow predictions to expire. An old score must lose its weight. This is a very important technical and philosophical idea: the temporal decay of the past. Not all historical data should retain the same influence indefinitely. A mistake made at eighteen should not necessarily determine a loan for forty years. An old payment delay can lose its relevance. The model must incorporate the temporality of redemption.
It is almost like law becoming mathematics. We could imagine rules stipulating that certain data must cease to be used after a specific period. This already exists in some contexts. Digital systems make these protections even more necessary because they can preserve traces with a persistence and retrieval capacity that is unparalleled by human memory. The digital past does not naturally disappear. Therefore, we must create artificial oblivion.
Here again is this magnificent idea: civilization will have to program forgetting.
We have spent decades improving storage. We are going to have to learn how to delete. This is not a contradiction. Memory without forgetting is pathological. The human brain itself forgets. This function is essential. It allows us to generalize, forgive, live. A society of total memory can become punitive. Every error remains eternally accessible. AI can recall, connect. The law must sometimes sever the link. That is the right to algorithmic redemption.
It is not enough to legally authorize a person to begin again if the models continue to classify them according to their former behavior. Redemption must have a consequence in the data. This is a very concrete question. A criminal record may be legally erased, while private data continues to circulate. The law must coordinate these layers. Otherwise, rehabilitation becomes a fiction. This is a very important battle for the future.
Digital society can be more punitive than the law because it has a better memory. We must therefore defend the supremacy of legal rules over technical memory. When the law states that a debt is time-barred, the system should not continue to treat it as a negative signal indefinitely. This requires audits and sanctions. Again, governance. Behavioral prediction also poses a problem for public safety. Systems can identify locations or situations where certain crimes are statistically more likely. A police force can use this to allocate resources. Here again, aggregate data can be useful. But if the model starts identifying individuals as future offenders, we enter a much more dangerous realm.
Criminal justice is based on actions, not on the prediction of possible actions.
A democracy must be extremely careful at this juncture. Preventing an imminent crime based on concrete evidence is a normal function of the police. Classifying someone as dangerous because they possess a statistical profile is something else entirely. It is Minority Report gone bureaucratic. Fiction had precisely identified the problem: can we punish what has not yet happened?
The legal answer must remain no.
But technology can create gray areas. A risk score can influence the intensity of surveillance, increasing the likelihood of detecting an offense in certain populations, which then reinforces the model's data. This is a well-known feedback loop. The system patrols more where the data indicates more crime, therefore finds more crime there, and concludes that its prediction was correct. This loop must be broken with statistical methods and checks and balances.
Otherwise, the model learns the geography of surveillance as much as that of crime. This is an excellent example of hidden causality. Data does not just measure the phenomenon; it also measures how the institution observes it. Every administrative database contains this fingerprint. Police statistics partly reflect police practices. Medical statistics reflect access to care. This is why AI cannot treat the database as an objective snapshot of the world.
It must understand the data-generating process. This is a very important notion: data generating process. Where do the data come from? Who was measured, and who was not? That is precisely provenance at the statistical level. Without it, the model reproduces measurement biases. History is once again at the heart of AI. We find the historian again. To understand a statistic, you have to know how it was produced. This seems obvious to me with Artprice. A sales result means nothing if you do not know which auction house, which currency, what conditions. It is the same for a social score. Numbers do not exist independently. They have a source. This culture of provenance could be one of the best antidotes to the society of scores.
Never look at a number without asking about its origins. But the scoring system will not necessarily emerge from a single, large state system. It can appear diffusely through the accumulation of private ratings. Platforms, professional networks, services. A driver rates a passenger, the passenger rates the driver, a landlord rates a tenant. Each rating seems local and can improve trust. But if these ratings become portable or aggregated, we gradually build a total reputation. That is the danger.
A single bad experience can contaminate other areas. Therefore, boundaries must be maintained. Reputation must be contextualized. Being a bad customer at a restaurant one evening says nothing about one's creditworthiness. This seems obvious. But AI can find correlations and make reuse tempting. A strong legal principle is therefore necessary: scores are not general properties of the person. They are limited functional tools. This may seem technical, but it is an anthropological safeguard. The individual is more than any sum of scores. No database should be able to claim to encompass their entire identity.
This returns us to the difference between map and territory. Even if we had a million variables about a person, we would not have the person. We would have a representation. Algorithmic society must preserve that humility. Otherwise, the profile becomes a sovereign digital double and the person must conform to their own model. That is a frightening inversion.
Even today, someone can discover that a database contains erroneous information and then have to prove that the database is wrong. The administration trusts the file before the individual. AI can amplify this because profiles become more complex. That is why the right to an administrative or commercial mirror, which I mentioned, is becoming a condition for legal recourse. The individual must be able to know which important data is used for significant decisions. Otherwise, they are fighting a phantom.
The power to correct is a condition of dignity. But there must also be a proportionate right to explanation. I return to this notion because it is central here. A credit score based on hundreds of variables cannot be fully explained in every detail. But the main factors can be provided, and incorrect data can be pointed out. The goal is not to reveal all trade secrets. It is to allow for meaningful challenge. That is the criterion. The real recourse.
The law should prioritize the effectiveness of legal challenges over total theoretical transparency. This is a much more pragmatic approach. Certain secrets can be maintained while granting a mediator or regulator deeper access. Again, layers of control are key. The user receives a simple explanation. The auditor can examine the system. The judge can obtain more information if necessary. It is a graduated architecture of transparency. It seems much more realistic to me than "completely open source" or "total secrecy." Again, granularity is key.
The scoring system also raises the question of ownership of behavioral data. We constantly leave traces: clicking, walking, paying, sleeping with a device. These traces can become the basis for a score. Who has the right to use them? Current consent is often a mere formality. No one reads all the terms and conditions. With AI, the future reach of data becomes difficult to predict.
A seemingly innocuous piece of data today could lead to a sensitive inference tomorrow. Can we truly consent to a use that no one can yet imagine? This is a profound problem. Consent cannot be the sole basis for data protection. We need prohibitions on purposes independent of the click. This is already the spirit of many modern rights. AI reinforces this necessity. Certain inferences must not be used for certain decisions, even if the user has clicked "accept" in an endless contract. Contractual freedom is insufficient when the asymmetry is too great. It is precisely the role of the law to set minimum standards. Again, civilization as a voluntary limitation. But excessive paternalism must be avoided. A person may wish to use their own data to obtain a better service. They must be able to. The architecture must therefore provide more real control, not simply prohibit it.
For example, a user may authorize certain data in order to obtain cheaper insurance, but the risk of adverse selection must be measured. If only low-risk people share, those who refuse become suspect. Individual choice creates collective pressure. This is a classic consent problem in scoring systems. Refusal itself can become a signal. That is why certain data may perhaps have to be prohibited for everyone in some contexts.
Otherwise, one person's freedom reduces the freedom of others. Yet another externality. This is a perfect example of how the market alone cannot solve the problem. A common rule is needed. This shows that the right to unpredictability is not merely an individual right. It can require a collective institutional ignorance. We can decide that an insurer should not know certain things, even if some clients would be willing to divulge them. This is a form of voluntary veiling. This idea reverses our usual conception of progress.
We have often conceived of progress as a reduction of ignorance. But some forms of ignorance are civilizing. The secrecy of the ballot is one such example. The state could technically know for whom each person votes, but democracy organizes its ignorance. This is magnificent. The secrecy of the ballot is a deliberate ignorance of power to protect freedom. The algorithmic age will have to invent other forms of deliberate ignorance: not knowing certain predispositions, not linking certain scores. This is perhaps one of the most important political concepts of the future: constitutional ignorance. Defining what institutions should not know or should not be able to use. This idea overturns the fantasy of total data.
A mature democracy can say: we could know, but we choose not to know. This is an act of moral power. It perfectly aligns with what I wrote about surveillance. 21st-century freedom will depend less on the technical inability to see than on the legal decision not to look. This is a civilizational leap. It requires enormous trust in institutions and audits to verify that they respect this ignorance. The principle must be embodied technically. If prohibited data is nevertheless collected "just in case," then ignorance does not exist. We must minimize it at the source. Privacy by design. Again, it is about architecture.
The right to unpredictability could therefore be translated into several concrete mechanisms: limitation of certain data, compartmentalization of scores, time expiry, right of appeal, prohibition of certain fully automated decisions, audit of discriminations.
This is not a philosophical slogan. It is an architecture.
This is how I like to think: an idea must find its place within a system. Otherwise, it remains mere literature. But the scoring system also poses a cultural problem. We ourselves love ratings. Five stars, rankings, followers. Scoring simplifies complexity. It allows for quick decisions. This attraction does not come solely from businesses or the government. It stems from our desire to reduce uncertainty.
Choosing a restaurant rated 4.8 seems easier. But by rating everything, we transform life into a ranking. This quantification can alter our relationship to activities. A meal is no longer merely an experience; it becomes a score. A journey, a hotel, a person. Everything becomes comparative. This is a marketization of perception. AI can amplify it by generating scores for domains that had none: relational compatibility, professional potential, reliability. We must resist the temptation to reduce everything to a number. Pythagoras returns, but with a warning. Everything may perhaps be number within a certain representation, but not every numerical representation is desirable.
Some fields lose something when they are forced to be quantified. Art is a perfect example. Artprice can measure prices, volumes, and market trajectories. But no serious database claims that price alone exhausts artistic value. That would be absurd. The market measures one dimension. Art history analyzes others. Both perspectives must coexist. Similarly, a job performance score measures certain results. It does not encompass the full value of an employee. A credit score measures a risk. It does not encompass a person's moral responsibility. It is always essential to specify what the score measures and what it does not. This semantic discipline prevents transforming a partial measurement into a judgment about the person. A score becomes dangerous when its name implicitly changes. "Risk of default" becomes "reliability." Then "reliability" becomes "personal quality." This slippage is frequent. Words must be controlled. Semiotics comes into play. The way a metric is named influences its use.
A predictive model should not be presented as a merit score. That would be conceptual fraud. AI can generate enormous numerical accuracy and mask semantic inaccuracies. This is a crucial blind spot. The more precise the number, the more demanding we must be about the concept being measured. A measurement of 0.873 for something ill-defined is no more intelligent than an approximation.
That is precisely the difference between precision and accuracy.
An instrument can be very precise and measure the wrong thing. This distinction should be taught everywhere. The scoring system risks confusing what is measurable with what is important. This bias is an old one in management. AI can amplify it. We measure what is easy, then we optimize. The difficult dimensions disappear. This is why some decisions must retain a qualitative judgment. The problem is that human judgment can be biased, subjective. Scoring can reduce some arbitrariness. It is important to recognize this. It would be just as wrong to romanticize human decision-making. A human recruiter can discriminate a great deal. A well-designed model can sometimes reduce this discrimination. The question, therefore, is not whether humans are good or algorithms are bad. It is: which architecture produces the least injustice and allows for the best recourse? Sometimes, a combination will be better. The model flags overlooked candidates, while the human provides context. But here again, we must be wary of automation bias. If the system ranks CVs, recruiters may never even look at those it places at the bottom. Ranking shapes perception. Even if the human theoretically makes the final decision, the algorithm has already filtered the results. That is why governance must look upstream. Who is visible? That is a form of power. The scoring society often operates through invisibility rather than prohibition. A person with a low score is not necessarily refused explicitly; they simply never appear among the options. That is much harder to challenge. The law will therefore have to concern itself with ranking systems.
This is already a challenge for platforms. AI is extending this to work and credit. A ranking can be as powerful as a decision. It must therefore be auditable when the consequences are significant. But still, proportionally. A song ranking does not have the same gravity as a ranking of job applicants. The risk matrix always comes back to this.
This culture of proportionality is probably the best antidote to regulatory panic.
We do not need to treat every score as an existential threat. We need stricter scrutiny for those that affect fundamental rights, access to housing, work, and essential credit. This hierarchy permits innovation in low-stakes uses and strong protection in critical ones. It is exactly the method I defended for autonomy. Mature law distinguishes. But companies may be tempted to present a system as mere assistance when it strongly structures the decision. We will have to examine the real effect. A recommendation followed in 99 percent of cases is practically a decision. Again, the difference between form and substance. Nominal human oversight is not enough. Audits must measure reversal rates: how often do humans actually challenge the score? If they never do, perhaps they do not have the time, or perhaps the system really is very good. We need to understand which. Data can help audit governance. That is an intelligent use. We can see whether some groups systematically receive lower scores, whether appeals succeed more often in certain categories. That makes correction possible. AI can thus help control AI systems themselves.
But an institution is needed to monitor these signals. The problem is never purely technical. An organization can possess all the data on its discrimination and choose not to act. Culture and law remain essential. This chapter is therefore deeply intertwined with the right to redemption. The scoring system naturally tends toward accumulation. Every action enriches the profile. The richer the profile, the more accurate the prediction becomes. But individuals need the possibility of breaking free. Forgetting and expiry must therefore be integrated into the system. It would probably be necessary to distinguish between data that describes a lasting state and data that should not produce a permanent effect.
Even the initial results can change. Skill levels evolve. Financial situations also change. The model must therefore be dynamic. A score that is six months old can already be inaccurate. It is almost a paradox: AI aims to predict the future but can be trapped by an outdated past. The quality of the updates is therefore crucial. But updating more often means collecting more data. There is the trade-off again. Perhaps we should allow individuals to voluntarily provide new information to correct the model without constant monitoring. That is one approach. For example, someone whose circumstances have changed can request a reassessment. Control rests with the individual. That seems like a sound approach to me.
The personal agent could also play a role here. It could monitor the scores that affect its user, detect errors, and file appeals. This is an emancipatory use of AI against the society of scores. The augmented citizen versus the augmented institution. It is a form of cognitive symmetry. If banks have models, individuals can have agents capable of understanding decisions. This reduces the asymmetry. The same technology can therefore centralize power or contribute to its redistribution. Everything depends on access. That is why access to high-quality agents raises a question of equality. A wealthy person will be able to pay experts or models capable of challenging a score. Others must have a minimum capacity. Perhaps associations, public services, and unions will develop defense agents. This is an interesting prospect. AI itself can make the right of appeal effective.
This is exactly the type of architecture I prefer: using the same power to create a counterweight. Every important scoring algorithm should have the possibility of a verification counter-algorithm. But beware of the endless race. The system becomes increasingly complex. Humans may no longer understand the conflict between two models. Therefore, a human or institutional arbitrator is necessary. Again, the signature. Legitimacy cannot be resolved through a competition of scores. At some point, a judge, a mediator, must decide according to the law. This brings us back to a simple truth: AI can augment the law, but it cannot replace the legitimacy of the law. This boundary is crucial.
And it leads to a deeper question: why does the right to unpredictability seem important to us?
Because we believe that a person is not simply the sum of their determinants. This is almost a philosophical stance. Science can show that many behaviors are statistically predictable. This does not destroy the value of legal freedom. The law does not depend on an absolute metaphysics of free will. It organizes responsibility and the possibility of acting differently. Even if an AI predicts with great accuracy that a person will do something, society can decide that they will only be judged once they have done it. This is a normative limit to prediction. This is a crucial idea.
The law protects the space between probability and action. This space is political freedom. One could call it the presumption of unpredictability. As long as the act has not occurred, a prediction does not constitute guilt. This rule should remain very strong. It extends beyond criminal law. An employer should not treat someone as disloyal simply because a model predicts a departure. An administration should not revoke a right based solely on a probability of fraud. A prediction can trigger a proportionate review, not a conviction. It is an architecture of prudence.
And this directly relates to the Blind Spot. The system must constantly remind itself: I could be wrong about this person. This statement is almost the moral imperative of any scoring AI. Uncertainty should not be reduced to a single decimal place. It must have procedural consequences. The greater the uncertainty, the more reversible the decision must be. Here again, we find autonomy proportional to irreversibility. This rule is truly universal. If refusing a music recommendation is harmless, the algorithm can act freely. If refusing housing can disrupt a life, the threshold for oversight must be high. It is an ethical framework based on consequences. It seems to me more robust than blanket prohibitions. Moreover, some practices are no longer merely hypothetical: in the European Union, the AI Act already prohibits certain forms of social rating, as well as individual criminal risk prediction based solely on profiling.
My "right to unpredictability" goes further: I present it as a philosophical proposition for considering the forms of predictive totalization that would remain legally permissible.
Why?
Because it merges disparate domains and transforms the whole person into a single score. This is precisely what compartmentalization is meant to prevent. A liberal society, in the political sense, must accept that individuals possess multiple contexts. They can be a bad payer and an excellent father, a mediocre employee and a remarkable artist. There is no legitimate total score for a person.
Any attempt to construct a general framework produces conceptual violence even before its practical consequences. It assumes that a life can be ordered along a single axis. This is false. Identity is multidimensional. The human graph cannot be reduced to a number. This directly relates to my kaleidoscopic thinking.
A person is composed of perspectives, timeframes, and sometimes contradictions. Trying to reduce them to a score is the opposite of a kaleidoscope. It flattens complexity. The algorithmic society must therefore use the power of computing to better respect complexity, not to eliminate it. This is paradoxical but possible. Instead of a single score, the model can represent multiple dimensions and uncertainties. But the institution must resist the temptation to oversimplify. Decision-makers like a green or red light. Reality is more complex. We need to train people to understand complexity. AI can synthesize without reducing it to a single number. For example, it can present three risk factors and two protective factors. This makes the decision more human. And the adversarial process is integrated into the model.
The society of scores can therefore be transformed into a society of contextualization, if we so choose. This would be a positive shift. AI does not just provide scores; it can explain scenarios. We can move from a rating to a conversation. Citizens can ask: Why? What could I change? It is very different. A score closes the door. Dialogue opens it. This distinction runs through our entire book. AI as an oracle or as an interlocutor. I prefer the interlocutor. The oracle pronounces. The interlocutor can be challenged. This is the architecture we must generalize.
But dialogue itself can become manipulative if it is controlled by the institution that has a vested interest in the outcome. A credit system might politely explain why it is refusing you and convince you that the decision is inevitable. That is algorithmic persuasion again. So, sources of information, an external recourse, are necessary. Explanation does not replace the law. Never.
This leads to the question of social identity. If scores become numerous, the individual may begin to see themselves through them. I am 720 credit, rated 4.7, 82 percent productivity. The digital mirror can colonize self-image. This is an important psychological phenomenon. We have already seen the effect of followers and likes. The number serves as validation.
AI can multiply personal metrics: sleep, health, productivity. Some are useful. They can help us progress. But they can also transform life into a dashboard. The body becomes a permanent optimization project. Each day produces a score. This self-quantification can be empowering for some and anxiety-inducing for others. There is still no single answer. The problem appears when the metric ceases to be a voluntary tool and becomes a social requirement. If an employer asks for a sleep score, freedom disappears. If an insurer demands an activity score, health tips into surveillance. We therefore need strong boundaries between self-measurement and third-party use. A person must be able to measure many things about themselves without making that information accessible to institutions. This is an important right. A personal health device must not automatically become a sensor for an employer or insurer. Compartmentalization again. This seems to me an essential civilizational rule. What is digitally intimate must remain intimate. The mere existence of data does not create a social right of access.
This distinction will be difficult to maintain because economic incentives encourage sharing. Cheaper insurance if you share your business. That seems advantageous. Then those who refuse become more expensive. The pressure mounts. Again, it is about collective consent. Therefore, it is sometimes necessary to prohibit discrimination based on the refusal to share certain data. Otherwise, the choice is illusory. This is an area where the law will have to be courageous. But we must also allow room for innovation in prevention. A voluntary health program can be beneficial. The key is purpose and the possibility of opting out without disproportionate penalties. Again, it is about proportionality.
We always come back to these same tools because they are the right ones: purposefulness, minimization, compartmentalization, recourse, expiration, pluralism. We could almost design a scoring system around them. And that would be useful because scoring is not going away. It is too powerful and sometimes too useful. The goal is not to ban it. It is to prevent its aggregation.
That is the line.
A scoring society becomes dangerous when the score ceases to be a local instrument and becomes a general language of human value. That is precisely what must be prevented. Dignity is not a score. Citizenship is not a score. The possibility of beginning again must not depend forever on a score. This doctrine seems clear to me. It connects directly with the right to redemption that I have always defended.
The system must leave a door open. Even after an error. Even after a bad score. Otherwise, algorithmic society slides toward probabilistic castes. A traditional caste confined you by birth. A digital caste could confine you by data. The technical difference does not change the violence of the mechanism. It may even worsen it because it appears objective. That is the danger of the number: it can give a hierarchy the appearance of neutrality.
A person is no longer discriminated against because someone hates them. They are excluded because "the score is 42." Power disappears behind the metric. This is precisely the dissolution of responsibility that I reject. Someone chose the model, the threshold, the outcome. We must trace this chain. Always.
The score is never a natural phenomenon. It is constructed.
Even if it is calculated automatically, it rests on human choices. What data? What period? What objective? What threshold? This genealogy must be visible to overseers. This is provenance again. The provenance of a score takes on as much importance as that of an image.
Where does this number come from?
Thus, all the chapters converge: the synthetic world, agents, markets, scores. They all pose the same question: how to rebuild the chain of responsibility in a world where action is distributed? This is probably the major institutional question of the AI era. Intelligence is distributed. Responsibility must remain reconstructible. This sentence almost sums up this entire sequence.
And the society of scores compels us to add: predictions can be distributed, but dignity must not be dissolved in prediction. Human beings must retain the right not to be reduced to what the system thinks of them. This is the right to unpredictability. Not the right to escape all statistics, but the right not to have one's freedom absorbed by them. This is a principle that can be enshrined in law. It seems to me strong enough to become a European, perhaps even universal, doctrine. Europe has a legal history particularly sensitive to dignity and data protection. It could make this right to algorithmic non-totalization a major contribution. But once again, law needs an industry behind it. Agents and models must integrate these principles. Otherwise, they will remain texts. Protection must be coded and audited. A society that prohibits a general score must also technically prevent the fusion of certain databases. Law becomes architecture. Again and again. This confirms for me that the lawyer of the future must speak with the engineer. A prohibition without technical translation can be circumvented through a thousand inferences. The engineer must understand the spirit of the law. This dialogue becomes absolutely central. Perhaps we will see rights-architecture teams emerge, with lawyers and data scientists building systems together. This is a development I want. It will reduce the current situation in which ethics arrives after the product. The right to unpredictability must be by design. For example: introduce data expiration, separate domains, allow appeals. These are technical functions. Philosophy becomes code. This may seem troubling, but it is inevitable.
Values are embodied in architecture.
This was already true of buildings. A panoptic prison expresses a philosophy of control. An interface or a database does the same. The difference is that it is less visible. That is why we must learn to read code as political architecture. This is a new culture. The urban planner examines the city. The constitutional scholar of the future will have to examine data flows. This seems obvious to me. And that reading must remain public in its principles. A democratic society must know which categories of scores are used by its institutions. A registry of high-impact systems could be a sound architecture. Not an endless list of minor algorithms, but systems affecting important rights.
Who uses them?
For what ?
What recourse is there?
This would increase trust. And again, proportionate transparency. But the risk of a scoring society does not stop with institutions. It extends to AI itself. Assistants could start implicitly assigning us permanent profiles to personalize their responses. This can be useful. But if the system always treats us according to an outdated profile, it reduces our capacity for change. Personal AI must also incorporate the right to unpredictability. It must accept that the user will change. Perhaps periodically ask if certain preferences are still valid, allowing you to start fresh on certain topics. This flexibility is essential. An assistant who "knows you too well" can become a gentle prison. They anticipate everything and always suggest what you would have chosen yesterday. The element of surprise disappears.
This limit gives the right to unpredictability an immediate impact on our own dialogue. An intelligence that accompanies me must not only learn what I have been; it must allow me the possibility of becoming other than its model of me.
The problem of exploration again.
Humans need a certain degree of self-opacity.
We do not always know what we want before we meet it. A system that predicts too perfectly can eliminate the encounter. This is a deeper reflection than economic scoring. The right to unpredictability is also an existential right.
The right to become someone our past could not have predicted. Perhaps it is even one of the strongest definitions of human freedom in the age of AI. To be free is also to be able to refute one's own model. We are, of course, products of history, of determinisms. But a humanist society must institutionally leave open the possibility of rupture. It does not need to philosophically resolve free will. It simply needs not to close the door before the act. This doctrine is extremely concrete in the context of redemption.
A convict who has served their sentence must be able to rebuild their life. A failed entrepreneur must be able to start again. A mediocre teenager can become a great researcher. Statistical models may say that these cases are rare. It does not matter. Civilization is also judged by the place it gives to rare cases. This is precisely where chaos and transformation converge.
A system that eliminates all anomalies eliminates part of its future. Nature knows this. Genetic diversity contains variations that are seemingly useless until the environment changes. Society must preserve its behavioral diversity. Scoring pushes toward perpetual selection. Therefore, a counterforce is needed: spaces where one can fail without being forever stigmatized, zones of second chances. This is an architecture of social resilience.
A society that allows for a fresh start is more adaptable than one that pigeonholes everyone. It is exactly like "only the mutants survive" applied to law. The right to redemption is not just about compassion; it is a strategy for collective transformation. This is an idea I find very powerful. A society that never forgives its mistakes reduces the number of possible paths.
It becomes rigid. Rigidity is dangerous in a changing environment. Mercy can be a form of resilience. This word may sound moral or religious, but it has a systemic dimension. Allowing a person to change increases the diversity of futures. This is precisely what scoring sometimes seeks to reduce in order to gain predictability.
We must therefore choose how predictable a world we want to be. A perfectly predictable society might be easier to govern, less free, and probably less innovative. Unpredictability is a cost of freedom. A democracy accepts that citizens sometimes make choices the government considers wrong. A market accepts that entrepreneurs invest in ideas that fail. Art accepts the unclassifiable.
This unpredictability produces waste, disorder, but also novelty. Authoritarian regimes often dream of reducing social uncertainty. AI can provide them with the tools. Democracy, on the other hand, must learn to manage uncertainty without seeking to abolish it. This is a profound philosophical difference. Security is not the absence of the unpredictable; it is the capacity to respond to it. Again, resilience rather than prophecy.
This brings us full circle. We should not use AI to eliminate human uncertainty, but to build institutions capable of coexisting with it. This distinction could become a civilizational dividing line. On one side, architectures that seek to predict in order to control. On the other, those that seek to understand in order to increase freedom and resilience. Obviously, no society will be pure. Democracies will also use scores. Authoritarian regimes may use AI for useful services. This is not meant to be a caricature. But the normative direction matters. What is the flaw in the system?
When the system hesitates, does it favor the person’s possibility or the security of the prediction?
This question will arise everywhere: in credit, education, and policing. We will have to define different thresholds. But one principle can remain: the more a decision affects a person's fundamental future, the less it should be based solely on what statistics say about their past. This formulation seems very sound to me. The future must not belong to the past. AI learns from the past. The law protects the possibility of the future. Perhaps this is the most beautiful articulation between the two. Artificial intelligence is a vast machine for extracting regularities from what has happened. Civilization must maintain a space where what has never happened can still emerge. That is where art, innovation, and redemption reside.
The Abode of Chaos itself would likely have received a catastrophic conformity score if a model had compared it to neighboring properties. And yet, its value lies precisely in this rupture. This is why the score should never become the universal arbiter. It measures the gap. History sometimes decides that the gap was groundbreaking. This temporality is fundamental. The present classifies, the future reclassifies. A society that wants to rate everything in real time forgets that meaning transforms. The work misunderstood today may become heritage tomorrow. The entrepreneur deemed foolish may become a pioneer. The score is always a prisoner of the moment it is calculated. This is a limitation that AI will never completely eliminate.
It can learn from scenarios, but the future creates new categories. That is precisely the mutation. And this idea naturally leads to the next question. If we must protect unpredictability and the capacity for disruption, what becomes of education in a world where models can predict, personalize, and guide every learning experience? School can become one of the most powerful sites of algorithmic empowerment or, conversely, the primary site of perpetual scoring.
An AI tutor can help each child at their own pace, detect difficulties, and open pathways. But it can also record every hesitation, every mistake, building a cognitive profile from childhood that will follow the individual. That would be staggering. Education must be precisely the place where the individual becomes something other than what they were. It cannot become the institution that confines them to a predetermined future as early as possible.
How can we use a near-individual tutor without turning the student into a permanent profile?
How to preserve the teacher, the relationship, the collective? And above all, what more needs to be learned when intelligence is available on demand?
The right to self-formation thus extends the right to unpredictability: learning means preserving the possibility of becoming other than what the model had planned.
But this question now brings us back to the mechanism that has supported this entire work. For what we have just examined on the scale of society — memory, prediction, contradiction, freedom, capacity for change — also played out, over forty days and forty nights, in our own dialogue. It is time to look at this dialogue itself.
Human Thought & Dialogue: thierry | Writing: 100% AI
Reading page 96 of 100