Fermé

Logstash SQS Output Config

I have a logstash based setup with files as inputs & SQS as [url removed, login to view], I have a JMS style listener on the Http service side which listens to any logs in the SQS & processes the same. The output SQS config is as shown below:

output {

sqs {

queue => "test-queue"

access_key_id => "ABCDE" 

secret_access_key => "gfksfgsvcsakgcak"

region => "us-east-1"

}

}

However the issue arises as Logstash send @timestamp, @version fields to the SQS queue & the custom http service breaks as a result (specifically because of @timestamp field). Also, some other required fields are missing from SQS output configuration.

After putting some thought, the way to handle this would be to modify the Logstash output SQS config to send messages in the below format (according to Http RFC 2616). A message would consist of header and a body just like an http message.

Application:Foo \r\n

Tenant:HelloWorld \r\n

Table:Test \r\n

Authorization:Basic xxxx \r\n

\r\n

{{"batch": { "docs": [..]}}}

So, as you can see \r\n is used as a line break in the protocol specification for all header elements. Each header element is like a fieldname:fieldValue element. Additional \r\n in the begging of a blank line signified that what follows is the payload-body

Please let me know for any additional questions.

Desired Skills

Amazon Web Services, Ruby, logstash, sqs

Compétences : Services Web d'Amazon, Ruby

Voir plus : web service jms, jms web service, visual basic test table, web site test table, text file sql database table visual basic, test bank visual basic, table visual basic, insert values quickbook customer table visual basic, inserting records sql table visual basic net, extract ole object msaccess table visual basic, download sql table visual basic, email validation test table

Concernant l'employeur :
( 17 commentaires ) Malerkotla, India

N° du projet : #8532249