In this article, we will see how to solve "<secret contains redacted>":Invalid character 'a' looking for beginning of value error if you are also getting the same while performing application deployment in Kubernetes cluster using Kustomize tool. I am basically using Fortanix as my secret management tool where I am storing all my Kubernetes secrets so that during any deployment in respective namespace, I can fetch those secrets using sdkms-cli tool and deploy all my configuration.

All secrets stored in Fortanix is in encrypted form depending on encrypting format you are using, so while fetching those secrets it cannot be used as it is. We need to decrypt the secret before apply changes to cluster using kubectl apply command. This is exactly the issue happened with me. I forgot to decrypt the secret before deployment and tried to use the encrypted secret as it is. This resulted in "<secret contains redacted>":Invalid character 'a' looking for beginning of value error. I am going to explain this error further below and the steps I had taken to solve this problem so that you can also try the same.

But before that, let me quickly explain you my simple-app application kustomize file structure through which I am trying to deploy all my kubernetes objects configuration.

kustomize
|
|---bases
|     |-- hello/simple-app
|     |-- hello/simple-app/tools.yaml
|
|---overlays
      |-- hello/simple-app/kustomization.yaml
      |-- hello/simple-app/secrets/example-credentials.yaml
      |-- hello/simple-app/example-app.yaml
      |-- hello/simple-app/example-registry.yaml

Solved "kustomize secret contains redacted": Invalid character 'a' looking for beginning of value

Also Read: How to Upgrade Kubernetes Cluster to Some Specific Version [Step by Step]

So as I was explaining, I was trying to deploy an application called simple-app in my kubernetes namespace through kustomization.yaml using below kustomize and kubectl command. During the deployment, I noticed that below command failed with error "<secret contains redacted>":Invalid character 'a' looking for beginning of value. This came as a surprise to me as I was not anticipating any error.

cyberithub@ubuntu:~$ kustomize build example-deployments/kustomize/overlays/hello/simple-app | kubectl apply -f -
secret/example-credentials-service configured
service/example-app-service created
deployment.apps/example-app-service created
The Secret "example-registry-credentials-service" is invalid: data[.dockerconfigjson]: Invalid value:"
<secret contents redacted>": invalid character 'a' looking for beginning of value

While the above error could come due to multiple reasons but in my case it was occurring because I forgot to decrypt the secrets which was getting fetched from Fortanix secret management tool. Kubernetes does not understand the secrets encrypted in Fortanix hence during the deployment it needs to be decrypted to plain text. So to decrypt the secrets I need to include my below decryption command during deployment using Kustomize.

sdkms-cli --config ~/example/app/secret.json fortanix decrypt -

So now my complete deployment command should look like below. This will check kustomization.yaml under simple-app location for all the required resources to be deployed in current namespace and then deploys it as shown below.

cyberithub@ubuntu:~$ kustomize build example-deployments/kustomize/overlays/hello/simple-app | sdkms-cli --config ~/example/app/secret.json fortanix decrypt - | kubectl apply -f -
secret/example-credentials-service configured
secret/example-harbor-credentials-service configured
service/example-app-service created
deployment.apps/example-app-service created

As you can see from above, this time manifest file changes got successfully deployed without any error. This confirms that our issue is resolved now. You should now see all required kubernetes objects configured successfully in your kubernetes cluster.

Hope it helps. However, sometimes it is possible that even after decrypting the secret properly, you might be getting same "<secret contains redacted>":Invalid character 'a' looking for beginning of value error. This can happen due to some other reasons as well. In that case, we need to specifically check this error in your system and find out the setup and secret management tool you are using to store secrets. After getting all those information, I will be able to help you with your error as well. Please do let me know your feedback in comment box.