April 2016 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Class allSubInstances size takes 23 seconds to run on Pharo 5 on OS X

by Eliot Miranda

Hi Bernhard, > On Apr 20, 2016, at 9:56 PM, Bernhard Pieber <bernhard(a)pieber.com> wrote: > > Hi Eliot, > > Right, but in the latest Squeak5.1-trunk with the latest CogSpur.r3663.app it takes only 3 milliseconds. I guess itâs just an optimized implementation, there. (Canât look right now.) Indeed ;-). In most Smalltalk implementations an object's header contains a direct reference (a pointer) to its class, but in Spur, an object's header contains a "class index" (as does the Dart VM). IIRC this is an approach first tried in a system at Xerox PARC, but what follows, I invented for 64-bit VW, to avoid having 64-bits of class reference in every object. The object header contains both a 22-bit classIndex, for up to 4 million classes, and a 22-bit identityHash field for 4 million hash values. The classIndex is used in all method caches, which has advantages for the GC, and means that when sending a message via an inline cache, the object's index is fetched, not its class. Classes are stored in a hidden two-level sparse array (array of arrays), the class table. If the object's class object is required (for the #class message or to do a full message lookup searching the inheritance hierarchy) the classIndex is used to index the class table. The classes known to the VM, Array, Message, Context, LargeNegativeInteger, LargePositiveInteger have known indices and occupy the first page of the class table. So the VM can allocate instances of these by simply writing a header containing the relevant classIndex. A conventional implementation would have to fetch the class from the specialObjectsArray and write it into the header (and in pre-Spur, there are 31 "compact class indices" of which a handful are used for a subset of the classes). But how does the VM find the classIndex of an arbitrary class? Searching the table would be very slow. So Spur arranges that a class's identityHash is also its index in the classTable. Classes have their own special identityHash primitive, see Behavior>>#identityHash. Whenever the VM tries to Instantiate a class, or whenever the class's identityHash primitive is run, if the class's identityHash is zero, the VM assigns an unused index in the class table as the identityHash and stores the class in the table. So a class's identityHash is its index in the class table. To Instantiate a class its identityHash is copied into the classIndex field of the new instance's header. To search for all instances of a class, the VM searches for objects whose classIndex is that class's identityHash. A consequence is that if a class doesn't yet have an identityHash (identityHash = 0) it can't have instances, and so there is no need to scan the heap. Note that you can't (easily) tell from the image. If you send identityHash to a class that will enter it into the table. There's a hidden hasIdentityHash primitive that can be used, and if you're curious I can post code to get at it. Note that this implies all the metaclasses are in the table because they have as their single instance a normal named class, but Metaclasses holds onto its instance via thisClass and so can also avoid traversing the heap. Some things to try: {Array. ByteString. Character. Context. LargeNegativeInteger. LargePositiveInteger. Message. SmallFloat64. BoxedFloat64. SmallInteger} collect: #identityHash and in a pre-Spur system Smalltalk compactClassesArray select: [:c| Smalltalk specialObjectsArray includes: c] thenCollect: [:c| {c. c indexIfCompact] Hope this explains what's going on under the covers. > Cheers, > Bernhard > >> Am 20.04.2016 um 17:43 schrieb Eliot Miranda <eliot.miranda(a)gmail.com>: >> >> Hi Bernhard, >> >> >>> On Apr 20, 2016, at 5:26 AM, Bernhard Pieber <bernhard(a)pieber.com> wrote: >>> >>> Dear Pharoers, >>> >>> I found something strange: >>> Time millisecondsToRun: [ Class allSubInstances size ]. â23617" >> >> Because it does an allInstances for Class and all its subclasses, and allInstances visits every object in the heap, so this does thousands of scans of the heap. >> >> >>> I did this on a new Pharo 5 image >>> curl get.pharo.org/alpha+vmLatest | bash >>> >>> Can somebody confirm this on their machine? What might be the reason? >>> >>> Cheers, >>> Bernhard > >

April 21, 2016

Re: [Pharo-dev] What the state of unifying Beacon with SystemLogger?

by Denis Kudriashov

2016-04-21 15:45 GMT+02:00 Tudor Girba <tudor(a)tudorgirba.com>: > > I think we not need special LoggerUI app. We can just inspect > SystemLogger instance to look at all recorded objects. And in perspective > of my suggestions it will mean that we just explore particular ObjectLog. > And we can extend GTInspector for this. > > > > Take a look at Beacon. The GTInspector extensions are provided out of the > box. Yes, I know. That's why I mention it. And I expect it will become much better in future. That's why I would not spend time to create special inspector for logs.

April 21, 2016

Re: [Pharo-dev] What the state of unifying Beacon with SystemLogger?

by Denis Kudriashov

2016-04-21 14:57 GMT+02:00 stepharo <stepharo(a)free.fr>: > > - name SystemLogger for collection based log. > > SystemLogger makes me think that it is singleton which is used by system > and I should not use it for my own application. > I would call it like ObjectLog or NativeLog (following my first sentence > about Log). > > It was the same as SystemAnnouncer. SystemAnnouncer is what we really use. We subscribe our tools on it. But SystemLogger is completely different case. Nobody is supposed to use it because users work with Log.

April 21, 2016

Re: [Pharo-dev] What the state of unifying Beacon with SystemLogger?

by Tudor Girba

> On Apr 20, 2016, at 9:05 PM, Ben Coman <btc(a)openInWorld.com> wrote: > > Sorry for not having a better reviewed understanding of both > frameworks, but I have some thoughts from perspective of one user... > > On Thu, Apr 21, 2016 at 12:05 AM, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: >> Hi. >> >> Thank's for answers. >> >> 2016-04-20 10:53 GMT+02:00 Norbert Hartl <norbert(a)hartl.name>: >>> >>> Agreed. As I said above. If you take to point of view from legacy logging >>> tool Beacon is not a big help. But the ideas in it are good and should be >>> integrated somewhere. If you are talking about legacy tooling it might ok to >>> use the terms logging as well, no? I personally like the work Beacon but in >>> my opinion it is wrong for this purpose. A beacon sends a uniform signal in >>> order to be discovered. The Beacon code sends signals to a beacon. And I can >>> register at the beacon in which signals I'm interested in. Feels wrong to >>> me. >> >> >> Your words are reasonable. Interesting what Tudor think about it? >> >> There are few things which I not like in SystemLogger (which not means that >> everything is good with Beacon): >> >> name Log to represent records of the underlying "physical log". >> >> IMHO Log means exactly end logging backend like text file, syslog or >> external database. Log is the place where we can explore (read) what was >> happen. > > You are considering Log as a noun, but it can also be used as a verb. > defn "13. to enter in a log; keep a record of; e.g. to log a day's events." [1] > > Stef said: "Remember that you want to have them short especially for > the main one. This is why in SystemLogger > we have Log instead of what it is LogObject" > from which I understood that Log was the client interface, to be used > like Transcript. I think this is a good idea. It would create the > LogEntry > > [1] http://www.dictionary.com/browse/log > >> And now these backends are represented by subclasses of Logger: >> StdoutStreamLogger, SysLogSender. I would call them StdoutLog, SysLog. >> But with such change we need to call log entry differently. It can be >> LogRecord or LogEntry. I not understand what was the problem with such >> names? (there was some explanation why Log was chosen for this) > > Considering a "log" being the final resting point of the logged > objects, that is the external disk text file or database, then within > Pharo we don't have logs, only processes that write to the external > logs, so Logger seems appropriate. This is except for an in memory > log, which for backend might need both MemoryLogger and > MemoryObjectLog. > > Log would create and send theLogEntry to SystemLogger > which forwards theLogEntry to other registered XXXLogger backend. > >> name SystemLogger for collection based log. >> >> SystemLogger makes me think that it is singleton which is used by system and >> I should not use it for my own application. > > I guess your application would make use SystemLogger via Log at the > front end, and registering backends with SystemLogger. > >> I would call it like ObjectLog or NativeLog (following my first sentence >> about Log). > > NativeLog doesn't seems right. ObjectLog might be suitable for an > in-memory backend, like above I mention MemoryObjectLog. > >> Also it is intercepted with SysLog name and I guess that's why it was called >> SysLogSender and not SysLogLogger. >> >> name LogDispatcher. > > LogDispatcher is okay, but I'm not sure of convention to be associate > the "doer" with the "target" rather than the "item", where > doer=Dispatcher; item=LogEntry; target=final-backend-log; A little > it sounds like its dispatching a whole log rather than one log entry. > Maybe should be LogEntryDispatcher(??) or... SystemLogger as the > marshalling point to forward to other XXXLoggers. > >> >> Here same problem as Log. What means physical logs dispatching? I know it is >> introduced around Log as log entry. But it is not intuitive for me (same as >> Log). > > I don't think Log should be a log entry. It should be the user > interface that creates and sends LogEntries via some XXXLogger. > >> In context of Log as real log (FileLog, SysLog) I would call this dispatcher >> as Logger with same responsibility to register new events in all registered >> logs. >> >> >> announcements LogAdded and LogRemoved in the core. >> >> Why any Logger should announce every log entry? >> I think we not need special LoggerUI app. We can just inspect SystemLogger >> instance to look at all recorded objects. And in perspective of my >> suggestions it will mean that we just explore particular ObjectLog. And we >> can extend GTInspector for this. > > GTInspector would need good live updating, filtering, clearing, > scrolling to keep cursor position as new entries arrive. It does already. Please take a look at Beacon :). Cheers, Doru > cheers -ben -- www.tudorgirba.com www.feenk.com "Obvious things are difficult to teach."

April 21, 2016

Re: [Pharo-dev] What the state of unifying Beacon with SystemLogger?

by Tudor Girba

Hi, > On Apr 20, 2016, at 6:05 PM, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > Hi. > > Thank's for answers. > > 2016-04-20 10:53 GMT+02:00 Norbert Hartl <norbert(a)hartl.name>: > Agreed. As I said above. If you take to point of view from legacy logging tool Beacon is not a big help. But the ideas in it are good and should be integrated somewhere. If you are talking about legacy tooling it might ok to use the terms logging as well, no? I personally like the work Beacon but in my opinion it is wrong for this purpose. A beacon sends a uniform signal in order to be discovered. The Beacon code sends signals to a beacon. And I can register at the beacon in which signals I'm interested in. Feels wrong to me. > > Your words are reasonable. Interesting what Tudor think about it? Please take a look at the reply I sent to Norbertâs email for explanations and further questions. > There are few things which I not like in SystemLogger (which not means that everything is good with Beacon): > â¢ name Log to represent records of the underlying "physical log". > IMHO Log means exactly end logging backend like text file, syslog or external database. Log is the place where we can explore (read) what was happen. > And now these backends are represented by subclasses of Logger: StdoutStreamLogger, SysLogSender. I would call them StdoutLog, SysLog. > But with such change we need to call log entry differently. It can be LogRecord or LogEntry. I not understand what was the problem with such names? (there was some explanation why Log was chosen for this) > â¢ name SystemLogger for collection based log. > SystemLogger makes me think that it is singleton which is used by system and I should not use it for my own application. > I would call it like ObjectLog or NativeLog (following my first sentence about Log). > Also it is intercepted with SysLog name and I guess that's why it was called SysLogSender and not SysLogLogger. > â¢ name LogDispatcher. > Here same problem as Log. What means physical logs dispatching? I know it is introduced around Log as log entry. But it is not intuitive for me (same as Log). > In context of Log as real log (FileLog, SysLog) I would call this dispatcher as Logger with same responsibility to register new events in all registered logs. > â¢ announcements LogAdded and LogRemoved in the core. > Why any Logger should announce every log entry? > I think we not need special LoggerUI app. We can just inspect SystemLogger instance to look at all recorded objects. And in perspective of my suggestions it will mean that we just explore particular ObjectLog. And we can extend GTInspector for this. > Take a look at Beacon. The GTInspector extensions are provided out of the box. Cheers, Tudor > > -- www.tudorgirba.com www.feenk.com "We are all great at making mistakes."

April 21, 2016

Re: [Pharo-dev] What the state of unifying Beacon with SystemLogger?

by Denis Kudriashov

2016-04-21 15:04 GMT+02:00 stepharo <stepharo(a)free.fr>: > Denis > > you see, we (norbert and me) did not take this names randonmly we really > thought about them. > Yes, I know. I just try to make fresh look on it :)

April 21, 2016

Re: [Pharo-dev] What the state of unifying Beacon with SystemLogger?

by Tudor Girba

Hi, > On Apr 20, 2016, at 10:53 AM, Norbert Hartl <norbert(a)hartl.name> wrote: > > Denis, > > >> Am 18.04.2016 um 18:28 schrieb Denis Kudriashov <dionisiydk(a)gmail.com>: >> >> Hello. >> >> Year ago Beacon and SystemLogger were announced. There was big discussion about them. And there were plans to provide single solution. What was done around that? >> > we had some talks but nothing happened afterwards. The whole discussion went somehow off track. The good part is that we had a conclusion that we want to log objects instead of strings. The bad part is that we forget to talk about the "logging" part of it. And the discussion and actions are not a good example for collaboration rather the opposite. > > The SystemLogger started as a project to build a small core about what is usually done in logging frameworks. Usually log messages are strings with a severity level like error, warn, etc attached. The log messages are dispatched to a central location. At that location there is a bunch of log message consumers that filter by severity level or other criteria. This filtering consumers are attached to sinks that store the messages in some format somewhere. The sinks implement formatting and adaptation to storage like files, syslog, database,â¦ > > We want to do better and we care about objects instead of strings. In case of writing to a file it will be a string anyway. So we can see that "logging objects" means that the distribution of a log through the system should happen as objects with rich interfaces. And it means that the conversion to a string should happen (if needed at all) at the latest possible moment. So in my opinion the logging framework should support: > > - having an easy way to emit any kind of log object. The most basic thing a log entity should have is a timestamp. Around it there could be some common use cases but most of it is application specific. > - multiple log consumers > - filtering of log entities. IMO this includes filtering of objects you log to store. In a typical scenario where you would have a severity level you only want to log certain messages. This can also include filtering for special types of objects. That would mean you filter the kind of objects the consumer can understand > - storage facilities. It is quite important to support files storage in many flavors, logging to syslog, logging to database etc. Without that you are just logging in memory and I'm not sure if a lot of people would agree that this is what is meant by using the word logging Good summary. When we discussed about adopting Beacon we also agreed that we will use the code or at least the experience of binding to concrete storage facilities that exist in SystemLogger. > As an example I can put one of my projects where I use SystemLogger. When sending a push notification I log/emit an object/event of type COMessageSent with sublcasses for apple, google, etc. It is emitted to a central location. In SystemLogger there a three loggers configured. One is in memory and counts the number of events per class. One takes the object, converts it into a short string representation like "apple push sent to XXXX" and sends it via syslog/UDP to a central logging server. Finally the third one takes the object, converts it to json and stores the json in a elasticsearch database. The database allows me to do all sorts of statistical analysis of the data over a long time frame such as number of push messages per month for one year. Which looks like this > > <Bildschirmfoto 2016-04-20 um 10.24.36.png> > > So if you take my point of view then Beacon is pretty useless. SystemLogger and Beacon both use a singleton to dispatch log entities. This is incorrect. Beacon does not require the singleton. We just have one for convenience, but you can use any announcer for the purpose. That is the beauty of it â¦ the fact that we do not have to reinvent anything because it is already provided by Announcements :). This was also a point that was made by Sven and is what I took into account when I designed Beacon. Please read the sections "Beyond global bindingsâ, and âLogging any announcements from any announcerâ from: http://www.humane-assessment.com/blog/beacon The singleton convenience is for replacing the places where Transcript is being currently used. > In SystemLogger has own implementation of registry for that loggers. Beacon uses Announcements for it. That's basically the difference. I like the idea of using Announcements and not invent yet another distribution mechanism. But with Announcements you can only filter by class and not by instance. > So it cannot do what system logger can do. Doru writes that he is working on that and it would be great. Although I have doubt this can be done performant I would be lucky to have it. Indeed, I think this is important. However, I also found that going extreme in creating signal types is quite powerful and enough for practical purposes. The cool thing about having fine-grained signal types is that you can also define easily dedicated inspector presentations. > On the other hand Beacon implements a handful of classes for some use cases but nothing more. So I see Svens point that you don't need it really because Announcements are there anyway and most of the stuff is application specific so I can just do that without having to integrate a "frameworkâ. Exactly :). These are convenience, and the cool thing is that people can just choose various levels without having to necessarily buy into the complete framework. We do not get often chances like this :) >> What exactly should be merged between this libraries? Was Beacon or SystemLogger planned to be end solution? >> > I'm not sure anymore. I can see some good things in Beacon that enable to register a logger for part of the execution. The drawback is that if you think about output facilities like log files there is no central place to attach it. And I surely don't want to have log configurations in the middle of my code. Beacon is not well thought in that direction, yet. I do not understand this part and I might be missing something. Could you go into details about where you see the log configurations problem in Beacon? > IMHO I'd like to have a SystemLogger with the distribution of objects like it is done in Beacon. Filtering is to be solved. This is hard to do with Announcements and to be honest I don't really like it how it is done in SystemLogger. I think that if we invest in the filtering part we can consolidate around Announcements. >> We definitely need legacy logging tool for Pharo. And I am going to work on this direction. >> Personally I prefer metaphor and names from Beacon. But both libraries implement similar idea. >> > Agreed. As I said above. If you take to point of view from legacy logging tool Beacon is not a big help. I do not understand what you mean by legacy logging. Could you explain? > But the ideas in it are good and should be integrated somewhere. If you are talking about legacy tooling it might ok to use the terms logging as well, no? I personally like the work Beacon but in my opinion it is wrong for this purpose. A beacon sends a uniform signal in order to be discovered. The Beacon code sends signals to a beacon. And I can register at the beacon in which signals I'm interested in. Feels wrong to me. Beacon is a central place which is both discoverable and propagates messages around to whomever is interested in consuming them. From this point of view, I think it works. Cheers, Doru > Norbert > -- www.tudorgirba.com www.feenk.com "Some battles are better lost than fought."

April 21, 2016

Re: [Pharo-dev] What the state of unifying Beacon with SystemLogger?

by stepharo

Denis you see, we (norbert and me) did not take this names randonmly we really thought about them. Stef

April 21, 2016

Re: [Pharo-dev] What the state of unifying Beacon with SystemLogger?

by stepharo

> You are considering Log as a noun, but it can also be used as a verb. > defn "13. to enter in a log; keep a record of; e.g. to log a day's events." [1] > > Stef said: "Remember that you want to have them short especially for > the main one. This is why in SystemLogger > we have Log instead of what it is LogObject" > from which I understood that Log was the client interface, to be used > like Transcript. Yes > I think this is a good idea. It would create the > LogEntry > > [1] http://www.dictionary.com/browse/log > >> And now these backends are represented by subclasses of Logger: >> StdoutStreamLogger, SysLogSender. I would call them StdoutLog, SysLog. >> But with such change we need to call log entry differently. It can be >> LogRecord or LogEntry. I not understand what was the problem with such >> names? (there was some explanation why Log was chosen for this) > Considering a "log" being the final resting point of the logged > objects, that is the external disk text file or database, then within > Pharo we don't have logs, only processes that write to the external > logs, so Logger seems appropriate. This is except for an in memory > log, which for backend might need both MemoryLogger and > MemoryObjectLog. > > Log would create and send theLogEntry to SystemLogger > which forwards theLogEntry to other registered XXXLogger backend. Yes this is exactly that Log -> Dispatcher -> Loggers > >> name SystemLogger for collection based log. >> >> SystemLogger makes me think that it is singleton which is used by system and >> I should not use it for my own application. > I guess your application would make use SystemLogger via Log at the > front end, and registering backends with SystemLogger. > >> I would call it like ObjectLog or NativeLog (following my first sentence >> about Log). > NativeLog doesn't seems right. ObjectLog might be suitable for an > in-memory backend, like above I mention MemoryObjectLog. > >> Also it is intercepted with SysLog name and I guess that's why it was called >> SysLogSender and not SysLogLogger. >> >> name LogDispatcher. > LogDispatcher is okay, but I'm not sure of convention to be associate > the "doer" with the "target" rather than the "item", where > doer=Dispatcher; item=LogEntry; target=final-backend-log; A little > it sounds like its dispatching a whole log rather than one log entry. > Maybe should be LogEntryDispatcher(??) or... SystemLogger as the > marshalling point to forward to other XXXLoggers. > >> Here same problem as Log. What means physical logs dispatching? I know it is >> introduced around Log as log entry. But it is not intuitive for me (same as >> Log). > I don't think Log should be a log entry. It should be the user > interface that creates and sends LogEntries via some XXXLogger. Here since we did not want to have two class Log for the creation API LogEntry/LogObject for representing the entry we choose to keep Log even if Log was not name LogEntry > >> In context of Log as real log (FileLog, SysLog) I would call this dispatcher >> as Logger with same responsibility to register new events in all registered >> logs. >> >> >> announcements LogAdded and LogRemoved in the core. >> >> Why any Logger should announce every log entry? >> I think we not need special LoggerUI app. We can just inspect SystemLogger >> instance to look at all recorded objects. And in perspective of my >> suggestions it will mean that we just explore particular ObjectLog. And we >> can extend GTInspector for this. > GTInspector would need good live updating, filtering, clearing, > scrolling to keep cursor position as new entries arrive. > > cheers -ben > >

April 21, 2016

Re: [Pharo-dev] What the state of unifying Beacon with SystemLogger?

by stepharo

Le 20/4/16 18:05, Denis Kudriashov a Ã©crit : > Hi. > > Thank's for answers. > > 2016-04-20 10:53 GMT+02:00 Norbert Hartl <norbert(a)hartl.name > <mailto:norbert@hartl.name>>: > > Agreed. As I said above. If you take to point of view from legacy > logging tool Beacon is not a big help. But the ideas in it are > good and should be integrated somewhere. If you are talking about > legacy tooling it might ok to use the terms logging as well, no? I > personally like the work Beacon but in my opinion it is wrong for > this purpose. A beacon sends a uniform signal in order to be > discovered. The Beacon code sends signals to a beacon. And I can > register at the beacon in which signals I'm interested in. Feels > wrong to me. > > > Your words are reasonable. Interesting what Tudor think about it? > > There are few things which I not like in SystemLogger (which not means > that everything is good with Beacon): > > * name Log to represent records of the underlying "physical log". > > IMHO Log means exactly end logging backend like text file, syslog or > external database. Log is the place where we can explore (read) what > was happen. > And now these backends are represented by subclasses of Logger: > StdoutStreamLogger, SysLogSender. I would call them StdoutLog, SysLog. > But with such change we need to call log entry differently. It can be > LogRecord or LogEntry. I not understand what was the problem with such > names? (there was some explanation why Log was chosen for this) We picked Log because we wanted to have it short. > > * name SystemLogger for collection based log. > > SystemLogger makes me think that it is singleton which is used by > system and I should not use it for my own application. > I would call it like ObjectLog or NativeLog (following my first > sentence about Log). It was the same as SystemAnnouncer. It could be name Logger. > Also it is intercepted with SysLog name and I guess that's why it was > called SysLogSender and not SysLogLogger. > > * name LogDispatcher. > > Here same problem as Log. What means physical logs dispatching? I know > it is introduced around Log as log entry. But it is not intuitive for > me (same as Log). > In context of Log as real log (FileLog, SysLog) I would call this > dispatcher as Logger with same responsibility to register new events > in all registered logs. LogDispatcher is coherent with Log. If you change Log to be LogEntry then LogDispatcher could be named LogEntryDispatcher. > > * announcements LogAdded and LogRemoved in the core. > > Why any Logger should announce every log entry? Because you can build tools to reflect changes. You do not want to use changed? We will remove changed from Pharo. > I think we not need special LoggerUI app. We can just inspect > SystemLogger instance to look at all recorded objects. So we do not have the same dream and need. Not all tools should fit in an inspector. > And in perspective of my suggestions it will mean that we just explore > particular ObjectLog. And we can extend GTInspector for this. > > > >

April 21, 2016