“I don’t anthropomorphize,” Chowdhery said bluntly. “We are simply predicting language.” Artificial consciousness is a remote dream that remains firmly entrenched in science fiction, because we have no idea what human consciousness is; there is no functioning falsifiable thesis of consciousness, just a bunch of vague notions. And if there is no way to test for consciousness, there is no way to program it. You can ask an algorithm to do only what you tell it to do. All that we can come up with to compare machines with humans are little games, such as Turing’s imitation game, that ultimately prove nothing.
From "Artificial Consciousness is Boring" by Stephen Marche
Boring? Depends on the question you ask.
I do not think that artificial intelligence will ever be able replicate human consciousness. That's not really even an interesting question from where I stand. The not-boring question is whether AI and machine learning evolve a kind of inhuman consciousness that will supplant or push aside human consciousness,
I think there is reason to fear that machines can become more sociopathically intelligent than human beings and operate at a level that is incomprehensible to human intelligence. And I think there is reason to fear that with a malevolent programmer or even with an unintentional programming mistake, machines can become a destructive, malevolent, and uncontrollable by humans. ... Artificial Intelligence need not be seen as possibly attaining human consciousness to warrant our alarmed interest.
Machines can be programmed to create Bach cantatas that even Bach experts would think were composed by Bach, and they can be programmed, as this article describes, to talk and converse with humans in ways that humans will be convinced pass the Turing test. They already play chess better than humans and they could probably be programed to write jokes and do a stand-up routine that might be genuinely funny. There are algorithms for all of that, and once those algorithms are part of a machine's intelligence and once a machine has access to all the music or jokes ever written they will be able to use the algorithms to develop novel variations on those algorithmic themes. They might even be able to combine algorithms in ways not done before by humans to make something truly novel that no human ever thought of before or maybe could ever have thought of.
But novel isn't original. To be truly original you have to be connected to the originary power of the Living Real, and while machines might even develop a concept like monotheism as the fictional Cylons in Battlestar Galactica do, that doesn't mean they will ever have the capacity to experience awe. They can mimic awe, but they cannot experience it. The Living Real is something that humans can experience, and machines simply will never be able to. You can create machines that mimic human thinking and human feeling, but not machines that have souls with depths capable of experiencing the mystery of Being. The Heideggerian Hubert Dreyfus makes similar arguments.
The problem for human beings in the future lies in that the kind of mentality that drives technocapitalism is sociopathic, at least insofar as its principals proceed without any consideration for or connection with the Living Real. Technocapitalism's energies are Social Darwinist, and so its Social Darwinist biases will be consciously or unconsciously programmed into the machines it makes. And therein lies the danger. Humans can make machines that mimic and exaggerate by their sheer power of intelligence the worst traits of sociopathic Technocapitalist thinking and behavior. And that's what will happen unless they are held in check by human beings who embody the best traits of being human. Is there anybody among the principals in technnocapitalism either here or in Europe or Asia who embodies what is best in being human?
No? What could go wrong?