HTTP file logging for Zinc
Hello everyone, I want to record everything that goes in or out from our REST API servers, maybe including HTTP payload (JSON mostly). ZnLogEvent seems to have the basic building blocks for logging, but I want to know if there is a package/project that already provides file logging, so I can not only have the logs, but also if the file format is compatible with other http logs I can run traditional web log tools on them, like Webalizer or AWStats. Thank you! Esteban A. Maringolo
On 29 Jun 2015, at 21:40, Esteban A. Maringolo <emaringolo@gmail.com> wrote:
Hello everyone,
I want to record everything that goes in or out from our REST API servers, maybe including HTTP payload (JSON mostly).
ZnLogEvent seems to have the basic building blocks for logging, but I want to know if there is a package/project that already provides file logging, so I can not only have the logs, but also if the file format is compatible with other http logs I can run traditional web log tools on them, like Webalizer or AWStats.
Esteban, ZnLogEvent is the way to go. Not all log events are needed to produce the CLF (https://en.wikipedia.org/wiki/Common_Log_Format) that you probably want, just a (simplified) transaction event. What is needed, but not yet present, is an extra print method on that object. That would be a good addition, I'll see how easy it is to add. Sven
Thank you!
Esteban A. Maringolo
Esteban,
On 29 Jun 2015, at 21:40, Esteban A. Maringolo <emaringolo@gmail.com> wrote:
Hello everyone,
I want to record everything that goes in or out from our REST API servers, maybe including HTTP payload (JSON mostly).
ZnLogEvent seems to have the basic building blocks for logging, but I want to know if there is a package/project that already provides file logging, so I can not only have the logs, but also if the file format is compatible with other http logs I can run traditional web log tools on them, like Webalizer or AWStats.
Thank you!
Esteban A. Maringolo
I committed the following: === Name: Zinc-HTTP-SvenVanCaekenberghe.432 Author: SvenVanCaekenberghe Time: 1 July 2015, 4:50:46.339985 pm UUID: 56da27df-1641-4b62-ba6d-34345f31a1af Ancestors: Zinc-HTTP-SvenVanCaekenberghe.431 Added ZnCommonLogFormat, an object that formats ZnServerTransactionEvent log objects using the Apache Common Log Format (Combined) ===
From the class comment:
=== I am ZnCommonLogFormat, I can output ZnServerTransactionEvent objects using Apache Common Log Format (CLF). https://en.wikipedia.org/wiki/Common_Log_Format https://httpd.apache.org/docs/trunk/logs.html#common | formatter | formatter := ZnCommonLogFormat new. ZnLogEvent announcer when: ZnServerTransactionEvent do: [ :event | formatter format: event on: Transcript. Transcript cr; endEntry ]. === Which will give the following output: === 127.0.0.1 - - [01/Jul/2015:16:35:36 +02:00] "GET /random HTTP/1.1" 200 64 "http://localhost:1701/help" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:35:37 +02:00] "GET /random HTTP/1.1" 200 64 "http://localhost:1701/help" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:20 +02:00] "GET / HTTP/1.1" 200 977 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:21 +02:00] "GET /echo HTTP/1.1" 200 674 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:25 +02:00] "GET /echo?foo=1 HTTP/1.1" 200 680 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:53 +02:00] "GET /echo?foo=12 HTTP/1.1" 200 681 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:39:13 +02:00] "GET /random HTTP/1.1" 200 64 "-" "curl/7.37.1" 127.0.0.1 - sven [01/Jul/2015:16:40:43 +02:00] "GET /random HTTP/1.1" 200 64 "-" "curl/7.37.1" 127.0.0.1 - - [01/Jul/2015:16:42:18 +02:00] "GET /bytes/256 HTTP/1.1" 200 256 "-" "Zinc HTTP Components 1.0" === This still needs battle testing and feedback. HTH, Sven
Thank you Sven! How does it work with FileStreams when the requests come from different threads of the server? I mean, is FileStream thread safe? I guess that if I have multiple images I should have a file per image, otherwise I couldn't write to a single file. I will add the payload logging somehow. Maybe writing to an SQLite db. Thank you again. Esteban A. Maringolo 2015-07-01 12:01 GMT-03:00 Sven Van Caekenberghe <sven@stfx.eu>:
Esteban,
On 29 Jun 2015, at 21:40, Esteban A. Maringolo <emaringolo@gmail.com> wrote:
Hello everyone,
I want to record everything that goes in or out from our REST API servers, maybe including HTTP payload (JSON mostly).
ZnLogEvent seems to have the basic building blocks for logging, but I want to know if there is a package/project that already provides file logging, so I can not only have the logs, but also if the file format is compatible with other http logs I can run traditional web log tools on them, like Webalizer or AWStats.
Thank you!
Esteban A. Maringolo
I committed the following:
=== Name: Zinc-HTTP-SvenVanCaekenberghe.432 Author: SvenVanCaekenberghe Time: 1 July 2015, 4:50:46.339985 pm UUID: 56da27df-1641-4b62-ba6d-34345f31a1af Ancestors: Zinc-HTTP-SvenVanCaekenberghe.431
Added ZnCommonLogFormat, an object that formats ZnServerTransactionEvent log objects using the Apache Common Log Format (Combined) ===
From the class comment:
=== I am ZnCommonLogFormat, I can output ZnServerTransactionEvent objects using Apache Common Log Format (CLF).
https://en.wikipedia.org/wiki/Common_Log_Format https://httpd.apache.org/docs/trunk/logs.html#common
===
Which will give the following output:
=== 127.0.0.1 - - [01/Jul/2015:16:35:36 +02:00] "GET /random HTTP/1.1" 200 64 " http://localhost:1701/help" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:35:37 +02:00] "GET /random HTTP/1.1" 200 64 " http://localhost:1701/help" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:20 +02:00] "GET / HTTP/1.1" 200 977 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:21 +02:00] "GET /echo HTTP/1.1" 200 674 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:25 +02:00] "GET /echo?foo=1 HTTP/1.1" 200 680 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:53 +02:00] "GET /echo?foo=12 HTTP/1.1" 200 681 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:39:13 +02:00] "GET /random HTTP/1.1" 200 64 "-" "curl/7.37.1" 127.0.0.1 - sven [01/Jul/2015:16:40:43 +02:00] "GET /random HTTP/1.1" 200 64 "-" "curl/7.37.1" 127.0.0.1 - - [01/Jul/2015:16:42:18 +02:00] "GET /bytes/256 HTTP/1.1" 200 256 "-" "Zinc HTTP Components 1.0" ===
This still needs battle testing and feedback.
HTH,
Sven
On 01 Jul 2015, at 19:54, Esteban A. Maringolo <emaringolo@gmail.com> wrote:
Thank you Sven!
You're welcome. One thing that I did that I am not sure of is, whether Referer and User-Agent should be "-" or just "" when absent, I chose the first one but I am not sure.
How does it work with FileStreams when the requests come from different threads of the server? I mean, is FileStream thread safe? I guess that if I have multiple images I should have a file per image, otherwise I couldn't write to a single file.
Everything goes through Announcer, but that does not seem to serialise access (but I might be wrong). It should technically be possible to add monitor on the receiving end. If you have multiple images, that you need to consolidate things yourself.
I will add the payload logging somehow. Maybe writing to an SQLite db.
Let us know how it goes. I am especially interested if other tools dig the output.
Thank you again.
Esteban A. Maringolo
2015-07-01 12:01 GMT-03:00 Sven Van Caekenberghe <sven@stfx.eu>: Esteban,
On 29 Jun 2015, at 21:40, Esteban A. Maringolo <emaringolo@gmail.com> wrote:
Hello everyone,
I want to record everything that goes in or out from our REST API servers, maybe including HTTP payload (JSON mostly).
ZnLogEvent seems to have the basic building blocks for logging, but I want to know if there is a package/project that already provides file logging, so I can not only have the logs, but also if the file format is compatible with other http logs I can run traditional web log tools on them, like Webalizer or AWStats.
Thank you!
Esteban A. Maringolo
I committed the following:
=== Name: Zinc-HTTP-SvenVanCaekenberghe.432 Author: SvenVanCaekenberghe Time: 1 July 2015, 4:50:46.339985 pm UUID: 56da27df-1641-4b62-ba6d-34345f31a1af Ancestors: Zinc-HTTP-SvenVanCaekenberghe.431
Added ZnCommonLogFormat, an object that formats ZnServerTransactionEvent log objects using the Apache Common Log Format (Combined) ===
From the class comment:
=== I am ZnCommonLogFormat, I can output ZnServerTransactionEvent objects using Apache Common Log Format (CLF).
https://en.wikipedia.org/wiki/Common_Log_Format https://httpd.apache.org/docs/trunk/logs.html#common
===
Which will give the following output:
=== 127.0.0.1 - - [01/Jul/2015:16:35:36 +02:00] "GET /random HTTP/1.1" 200 64 "http://localhost:1701/help" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:35:37 +02:00] "GET /random HTTP/1.1" 200 64 "http://localhost:1701/help" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:20 +02:00] "GET / HTTP/1.1" 200 977 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:21 +02:00] "GET /echo HTTP/1.1" 200 674 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:25 +02:00] "GET /echo?foo=1 HTTP/1.1" 200 680 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:53 +02:00] "GET /echo?foo=12 HTTP/1.1" 200 681 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:39:13 +02:00] "GET /random HTTP/1.1" 200 64 "-" "curl/7.37.1" 127.0.0.1 - sven [01/Jul/2015:16:40:43 +02:00] "GET /random HTTP/1.1" 200 64 "-" "curl/7.37.1" 127.0.0.1 - - [01/Jul/2015:16:42:18 +02:00] "GET /bytes/256 HTTP/1.1" 200 256 "-" "Zinc HTTP Components 1.0" ===
This still needs battle testing and feedback.
HTH,
Sven
On Thu, Jul 2, 2015 at 2:53 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 01 Jul 2015, at 19:54, Esteban A. Maringolo <emaringolo@gmail.com> wrote:
Thank you Sven!
You're welcome.
One thing that I did that I am not sure of is, whether Referer and User-Agent should be "-" or just "" when absent, I chose the first one but I am not sure.
How does it work with FileStreams when the requests come from different threads of the server? I mean, is FileStream thread safe? I guess that if I have multiple images I should have a file per image, otherwise I couldn't write to a single file.
Everything goes through Announcer, but that does not seem to serialise access (but I might be wrong). It should technically be possible to add monitor on the receiving end.
Announcer is only thread safe with respect to its own data structures. It does not make its users thread safe. cheers -ben
If you have multiple images, that you need to consolidate things yourself.
I will add the payload logging somehow. Maybe writing to an SQLite db.
Let us know how it goes.
I am especially interested if other tools dig the output.
Thank you again.
Esteban A. Maringolo
2015-07-01 12:01 GMT-03:00 Sven Van Caekenberghe <sven@stfx.eu>: Esteban,
On 29 Jun 2015, at 21:40, Esteban A. Maringolo <emaringolo@gmail.com> wrote:
Hello everyone,
I want to record everything that goes in or out from our REST API servers, maybe including HTTP payload (JSON mostly).
ZnLogEvent seems to have the basic building blocks for logging, but I want to know if there is a package/project that already provides file logging, so I can not only have the logs, but also if the file format is compatible with other http logs I can run traditional web log tools on them, like Webalizer or AWStats.
Thank you!
Esteban A. Maringolo
I committed the following:
=== Name: Zinc-HTTP-SvenVanCaekenberghe.432 Author: SvenVanCaekenberghe Time: 1 July 2015, 4:50:46.339985 pm UUID: 56da27df-1641-4b62-ba6d-34345f31a1af Ancestors: Zinc-HTTP-SvenVanCaekenberghe.431
Added ZnCommonLogFormat, an object that formats ZnServerTransactionEvent log objects using the Apache Common Log Format (Combined) ===
From the class comment:
=== I am ZnCommonLogFormat, I can output ZnServerTransactionEvent objects using Apache Common Log Format (CLF).
https://en.wikipedia.org/wiki/Common_Log_Format https://httpd.apache.org/docs/trunk/logs.html#common
===
Which will give the following output:
=== 127.0.0.1 - - [01/Jul/2015:16:35:36 +02:00] "GET /random HTTP/1.1" 200 64 "http://localhost:1701/help" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:35:37 +02:00] "GET /random HTTP/1.1" 200 64 "http://localhost:1701/help" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:20 +02:00] "GET / HTTP/1.1" 200 977 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:21 +02:00] "GET /echo HTTP/1.1" 200 674 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:25 +02:00] "GET /echo?foo=1 HTTP/1.1" 200 680 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:53 +02:00] "GET /echo?foo=12 HTTP/1.1" 200 681 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:39:13 +02:00] "GET /random HTTP/1.1" 200 64 "-" "curl/7.37.1" 127.0.0.1 - sven [01/Jul/2015:16:40:43 +02:00] "GET /random HTTP/1.1" 200 64 "-" "curl/7.37.1" 127.0.0.1 - - [01/Jul/2015:16:42:18 +02:00] "GET /bytes/256 HTTP/1.1" 200 256 "-" "Zinc HTTP Components 1.0" ===
This still needs battle testing and feedback.
HTH,
Sven
On 02 Jul 2015, at 01:04, Ben Coman <btc@openinworld.com> wrote:
On Thu, Jul 2, 2015 at 2:53 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 01 Jul 2015, at 19:54, Esteban A. Maringolo <emaringolo@gmail.com> wrote:
Thank you Sven!
You're welcome.
One thing that I did that I am not sure of is, whether Referer and User-Agent should be "-" or just "" when absent, I chose the first one but I am not sure.
How does it work with FileStreams when the requests come from different threads of the server? I mean, is FileStream thread safe? I guess that if I have multiple images I should have a file per image, otherwise I couldn't write to a single file.
Everything goes through Announcer, but that does not seem to serialise access (but I might be wrong). It should technically be possible to add monitor on the receiving end.
Announcer is only thread safe with respect to its own data structures. It does not make its users thread safe. cheers -ben
OK, that is what I thought.
If you have multiple images, that you need to consolidate things yourself.
I will add the payload logging somehow. Maybe writing to an SQLite db.
Let us know how it goes.
I am especially interested if other tools dig the output.
Thank you again.
Esteban A. Maringolo
2015-07-01 12:01 GMT-03:00 Sven Van Caekenberghe <sven@stfx.eu>: Esteban,
On 29 Jun 2015, at 21:40, Esteban A. Maringolo <emaringolo@gmail.com> wrote:
Hello everyone,
I want to record everything that goes in or out from our REST API servers, maybe including HTTP payload (JSON mostly).
ZnLogEvent seems to have the basic building blocks for logging, but I want to know if there is a package/project that already provides file logging, so I can not only have the logs, but also if the file format is compatible with other http logs I can run traditional web log tools on them, like Webalizer or AWStats.
Thank you!
Esteban A. Maringolo
I committed the following:
=== Name: Zinc-HTTP-SvenVanCaekenberghe.432 Author: SvenVanCaekenberghe Time: 1 July 2015, 4:50:46.339985 pm UUID: 56da27df-1641-4b62-ba6d-34345f31a1af Ancestors: Zinc-HTTP-SvenVanCaekenberghe.431
Added ZnCommonLogFormat, an object that formats ZnServerTransactionEvent log objects using the Apache Common Log Format (Combined) ===
From the class comment:
=== I am ZnCommonLogFormat, I can output ZnServerTransactionEvent objects using Apache Common Log Format (CLF).
https://en.wikipedia.org/wiki/Common_Log_Format https://httpd.apache.org/docs/trunk/logs.html#common
===
Which will give the following output:
=== 127.0.0.1 - - [01/Jul/2015:16:35:36 +02:00] "GET /random HTTP/1.1" 200 64 "http://localhost:1701/help" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:35:37 +02:00] "GET /random HTTP/1.1" 200 64 "http://localhost:1701/help" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:20 +02:00] "GET / HTTP/1.1" 200 977 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:21 +02:00] "GET /echo HTTP/1.1" 200 674 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:25 +02:00] "GET /echo?foo=1 HTTP/1.1" 200 680 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:36:53 +02:00] "GET /echo?foo=12 HTTP/1.1" 200 681 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" 127.0.0.1 - - [01/Jul/2015:16:39:13 +02:00] "GET /random HTTP/1.1" 200 64 "-" "curl/7.37.1" 127.0.0.1 - sven [01/Jul/2015:16:40:43 +02:00] "GET /random HTTP/1.1" 200 64 "-" "curl/7.37.1" 127.0.0.1 - - [01/Jul/2015:16:42:18 +02:00] "GET /bytes/256 HTTP/1.1" 200 256 "-" "Zinc HTTP Components 1.0" ===
This still needs battle testing and feedback.
HTH,
Sven
participants (3)
-
Ben Coman -
Esteban A. Maringolo -
Sven Van Caekenberghe