Simon Krenger

For my k3s installation, I wanted to try out the free tier of Splunk Cloud for storing logs. That means configuring Vector (my log collector of choice) to forward the logs to Splunk Cloud. That was easier said than done.

Figuring out that we need to use type: splunk_hec_logs in the Vector configuraton was quite straight forward. However figuring out the endpoint for the configuration turned out to be more of a challenge. After some time, I finally understood what my Splunk Cloud Platform instance ID is, and after encountering various 303 See Other and Unexpected status: 404 Not Found errors I finally got a working Vector configuration:

[..]  
    sinks:
      stdout:
        type: splunk_hec_logs
        inputs: [kubernetes_logs]
        endpoint: "https://prd-p-xxxxx.splunkcloud.com:8088"
        default_token: "${SPLUNK_HEC_TOKEN}"
        encoding:
          codec: json
        tls:
          verify_hostname: false
          ca_file: |
            -----BEGIN CERTIFICATE-----
            [... ADD CA CERTIFICATE HERE ....]
            -----END CERTIFICATE-----
        acknowledgements:
          enabled: false
          indexer_acknowledgements_enabled: false

The SPLUNK_HEC_TOKEN looks something like this: 12345678-abcd-efab-aaaa-cafecafecafe. It can be found by creating a new HTTP Event Collector in the Splunk UI (“Settings” -> “Data inputs” -> “HTTP Event Collector” -> “New Token”). In my case I am using an environment variable set on the Pod to store the HEC token in a Secret:

[..]
    env:
    - name: VECTOR_LOG
      value: info
    - name: SPLUNK_HEC_TOKEN
      valueFrom:
        secretKeyRef:
          key: token
          name: vector-splunk-hec-token
[..]

Vector will then use the environment variable for the configuration if you specify it as shown above. To debug all of this, I used the following curl command to manually send events to Splunk Cloud and to figure out the endpoint plus the necessary tokens:

curl -k https://prd-p-xxxxx.splunkcloud.com:8088/services/collector/event -H "Authorization: Splunk 12345678-abcd-efab-aaaa-cafecafecafe" -d '{"event": "hello world"}'

Maybe that saves someone from having to do a bit of trial-and-error to get this working.